使用goldengate 迁移Oracle到postgresql

环境：

--源端：

IP：10.0.4.16

hostname：tencent

Oracle数据库版本：12.2.0.1.0

ogg for oracle版本：19.1.0.0.4

SID：orcl

--目标端：

IP：10.0.4.16

hostname：tencent

postgresql数据库版本：14.6

ogg for postgresql版本：21.3.0.0.0

db:ogg

2、安装和配置OGG软件（源端），以下操作都是在源端操作

2.1创建安装目录

mkdir -p /oracle/ogg

2.2设置oracle环境变量

export ORACLE_BASE=/oracle/app/oracle

export ORACLE_HOME=$ORACLE_BASE/product/12.2.0/db_1

export PATH=$ORACLE_HOME/bin:/oracle/ogg:$PATH

export OGGHOME=/oracle/ogg

export ORACLE_SID=orcl

export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$OGGHOME:$LD_LIBRARY_PATH

export TNS_ADMIN=$ORACLE_HOME/network/admin

2.3建立OGG表空间和用户

create tablespace oggtbs datafile '/oradata/orcl/oggtbs.dbf' size 200M autoextend on;

create user ogg identified by ogg default tablespace ogg;

grant connect, resource TO ogg;

grant select any dictionary, select any table TO ogg;

grant flashback any table TO ogg;

grant execute on dbms_flashback TO ogg;

grant ALTER ANY TABLE to ogg;

grant insert any table to ogg;

grant update any table to ogg;

grant delete any table to ogg;

grant create table,create sequence to ogg;

grant execute on utl_file to ogg;

grant dba to ogg;

alter system set enable_goldengate_replication=true;

2.4添加附加日志和force logging，并切换日志

如果源端不开启归档，需添加多组日志。

alter database force logging;

alter database add SUPPLEMENTAL log data;

SQL>select NAME,OPEN_MODE,FORCE_LOGGING,SUPPLEMENTAL_LOG_DATA_MIN from v$database;

NAME OPEN_MODE FORCE_LOGGING SUPPLEME

ORCL READ WRITE YES YES

2.5 创建OGG的管理工作目录

在配置OGG进程之前，首先需要创建OGG的管理目录，执行以下操作：

GGSCI (tencent) 3> create subdirs

Creating subdirectories under current directory /oracle/ogg

Parameter file /oracle/ogg/dirprm: created.

Report file /oracle/ogg/dirrpt: created.

Checkpoint file /oracle/ogg/dirchk: created.

Process status files /oracle/ogg/dirpcs: created.

SQL script files /oracle/ogg/dirsql: created.

Database definitions files /oracle/ogg/dirdef: created.

Extract data files /oracle/ogg/dirdat: created.

Temporary files /oracle/ogg/dirtmp: created.

Credential store files /oracle/ogg/dircrd: created.

Masterkey wallet files /oracle/ogg/dirwlt: created.

Dump files /oracle/ogg/dirdmp: created.

只有提交事务的record才会被捕获。抽取、投递、应用日志的单位是record，而不是trail文件。

2.6配置MGR参数文件，并启动mgr

mgr进程是这些进程的管理和守护进程，目标端的mgr进程还与源端进行通信

edit params mgr

PORT 7809

GGSCI (tencent) 7> start mgr

Manager started.

GGSCI (tencent) 8> info all

Program Status Group Lag at Chkpt Time Since Chkpt

MANAGER RUNNING

2.7添加extract进程抽取进程：负责抓取需要传输的数据（添加进程的时候为抽取点，而不是启动进程的时候）从21开始OGG需要配置tns #查看当前数据库语言 SQL> select userenv('language') from dual; USERENV('LANGUAGE')

AMERICAN_AMERICA.ZHS16GBK

--配置extract进程

#编辑参数

GGSCI (tencent as ogg@orcl) 21> edit params exta

extract exta

SETENV (NLS_LANG=AMERICAN_AMERICA.ZHS16GBK)

SETENV (ORACLE_HOME=/oracle/app/oracle/product/12.2.0/db_1)

SETENV (ORACLE_SID=orcl)

USERID ogg@orcl, PASSWORD ogg

DISCARDFILE ./dirrpt/exta.dsc,APPEND,MEGABYTES 1024

EXTTRAIL ./dirdat/ea

table hr.*;

#创建extract组

ADD EXTRACT exta, TRANLOG, BEGIN NOW

#创建本地 trail 文件

ADD EXTTRAIL ./dirdat/ea, EXTRACT exta, MEGABYTES 5

#启动extract进程

GGSCI (tencent as ogg@orcl) 22> start exta

GGSCI (tencent as ogg@orcl) 24> info all

Program Status Group Lag at Chkpt Time Since Chkpt

MANAGER RUNNING

EXTRACT RUNNING EXTA 00:01:45 00:00:03

2.8投递进程

投递进程：

EDIT PARAMS pumpa

#编辑参数

GGSCI (tencent as ogg@orcl) 26> edit param pumpa

EXTRACT pumpa

SETENV (NLS_LANG=AMERICAN_AMERICA.UTF8)

PASSTHRU

RMTHOST 10.0.4.16, MGRPORT 7810

RMTTRAIL ./dirdat/pa

TABLE hr.*;

#用add extract指定本地 trail 文件

ADD EXTRACT pumpa, EXTTRAILSOURCE ./dirdat/ea

#用add rmttrail指定远程 trail 文件

ADD RMTTRAIL ./dirdat/pa, EXTRACT pumpa, MEGABYTES 5

#启动Pump进程

START EXTRACT pumpa

GGSCI (tencent) 13> info all

Program Status Group Lag at Chkpt Time Since Chkpt

MANAGER RUNNING

EXTRACT RUNNING EXTA 00:00:00 00:00:00

EXTRACT RUNNING PUMPA 00:00:00 00:00:01

2.9生成define文件

在异构的同步复制中需要生成define文件。用于数据兼容不同数据库。

edit params defgen

#参数文件内容

defsfile ./dirdef/defgen.def

userid ogg, password ogg

TABLE hr.*;

#退出ggsci命令

GGSCI (tencent) 20> exit

#生成文件

[oracle@tencent ogg]$ defgen paramfile ./dirprm/defgen.prm

Oracle GoldenGate Table Definition Generator for Oracle

Version 19.1.0.0.4 OGGCORE_19.1.0.0.0_PLATFORMS_191017.1054

Linux, x64, 64bit (optimized), Oracle 12c on Oct 17 2019 13:20:56

Starting at 2023-07-25 23:54:08

Operating System Version:

Linux

Version #1 SMP Tue Jun 28 15:37:28 UTC 2022, Release 3.10.0-1160.71.1.el7.x86_64

Node: tencent

Machine: x86_64

soft limit hard limit

Address Space Size : unlimited unlimited

Heap Size : unlimited unlimited

File Size : unlimited unlimited

CPU Time : unlimited unlimited

Process id: 2283

** Running with the following parameters **

defsfile ./dirdef/defgen.def

userid ogg, password ***

TABLE hr.;Expanding wildcard table specification hr.:

Retrieving definition for HR.COUNTRIES.

Retrieving definition for HR.DEPARTMENTS.

Retrieving definition for HR.EMPLOYEES.

Retrieving definition for HR.EMP_DETAILS_VIEW.

2023-07-25 23:54:10 WARNING OGG-06439 No unique key is defined for table EMP_DETAILS_VIEW. All viable columns will be used to represent the key, but may not guarantee uniqueness. KEYCOLS may be used to define the key.

Retrieving definition for HR.JOBS.

Retrieving definition for HR.JOB_HISTORY.

Retrieving definition for HR.LOCATIONS.

Retrieving definition for HR.REGIONS.

Retrieving definition for HR.RSXY_QT_ZRRSBJN_TEMP.

2023-07-25 23:54:10 WARNING OGG-06439 No unique key is defined for table RSXY_QT_ZRRSBJN_TEMP. All viable columns will be used to represent the key, but may not guarantee uniqueness. KEYCOLS may be used to define the key.

Definitions generated for 9 tables in ./dirdef/defgen.def.

#生成文件后传送文件到目标端dirdef目录下

cp /oracle/ogg/dirdef/defgen.def /postgresql/ogg/dirdef

三、安装和配置OGG软件（目标端），以下操作都是在目标端操作

3.1 创建安装目录

mkdir -p /postgresql/ogg

3.2 配置环境变量

export PGHOME=/postgresql/pg14

export PGDATA=/postgresql/pgdata

export PATH=$PGHOME/bin:$OGGHOME:/usr/bin/isql:$PATH

export OGGHOME=/postgresql/ogg

export ODBCINI=$OGGHOME/odbc.ini

export LD_LIBRARY_PATH=$PGHOME/lib:/postgresql/ogg/lib:$LD_LIBRARY_PATH

3.3 创建ogg用户

[postgres@tencent dirdef]$ psql

psql (14.6)

Type "help" for help.

postgres=# create database ogg;

CREATE DATABASE

postgres=# create user ogg superuser password 'ogg';

CREATE ROLE

postgres=# \c ogg ogg

You are now connected to database "ogg" as user "ogg".

ogg=# \c

You are now connected to database "ogg" as user "ogg".

ogg=# create schema ogg;

CREATE SCHEMA

3.4 创建DSN

ogg for postgresql使用的是ODBC来连接数据库，因此需要指明ODBC dirver

--安装UnixODBC

yum install -y unixODBC.x86_64

--安装PostgresqlODBC

yum install -y postgresql-odbc.x86_64

--配置ODBC

cd /etc/ 复制odbcinst.ini到ogg的home目录下，查看odbcinst

(注意该文件不要改动)

[postgres@tencent ogg]$ cat odbcinst.ini

Example driver definitions

Driver from the postgresql-odbc package

Setup from the unixODBC package

[PostgreSQL]

Description = ODBC for PostgreSQL

Driver = /usr/lib/psqlodbcw.so

Setup = /usr/lib/libodbcpsqlS.so

Driver64 = /usr/lib64/psqlodbcw.so

Setup64 = /usr/lib64/libodbcpsqlS.so

FileUsage = 1

Driver from the mysql-connector-odbc package

Setup from the unixODBC package

[MySQL]

Description = ODBC for MySQL

Driver = /usr/lib/libmyodbc5.so

Setup = /usr/lib/libodbcmyS.so

Driver64 = /usr/lib64/libmyodbc5.so

Setup64 = /usr/lib64/libodbcmyS.so

FileUsage = 1

--配置odbc.ini，在该文件中填写

[postgres@tencent ogg]$ vi odbc.ini

[ODBC Data Sources]

gg_pg=DataDirect 14.6 PostgreSQL Wire Protocol #gg_pg数据库别名，与下面gg_pg相同

[ODBC]

IANAAppCodePage=106 #106等价于UTF8

InstallDir=/postgresql/ogg #ogg home目录

[gg_pg]

Driver=/postgresql/ogg/lib/GGpsql25.so #OGG安装目录下有内置ODBC驱动

Description=DataDirect 14.6 PostgreSQL Wire Protocol

Database=ogg

HostName=10.0.4.16

PortNumber=4519

--登录Postgresql数据库

修改profile文件，添加ODBCINI，指定ODBC DRIVER路径

[postgres@tencent ogg]$ vi ~/.bash_profile

export ODBCINI=$OGGHOME/odbc.ini

#添加odbcini访问路径，默认访问/etc目录下的odbc.ini

#此时isql访问数据库失败，可以忽略，如下

[postgres@tencent ogg]$ isql gg_pg

[ISQL]ERROR: Could not SQLConnect

#ggsci登录postgresql

[postgres@tencent ogg]$ ggsci

Oracle GoldenGate Command Interpreter for PostgreSQL

Version 21.3.0.0.0 OGGCORE_21.3.0.0.0_PLATFORMS_210728.1047

Oracle Linux 7, x64, 64bit (optimized), PostgreSQL on Aug 4 2021 20:27:55

Operating system character set identified as UTF-8.

GGSCI (tencent) 1> dblogin sourcedb gg_pg userid ogg,password ogg

2023-07-26 11:12:21 INFO OGG-03036 Database character set identified as UTF-8. Locale: en_US.utf8.

2023-07-26 11:12:21 INFO OGG-03037 Session character set identified as UTF-8.

Successfully logged into database.

3.5 创建ogg目录

GGSCI (tencent) 1> create subdirs

Creating subdirectories under current directory /postgresql/ogg

Parameter file /postgresql/ogg/dirprm: created.

Report file /postgresql/ogg/dirrpt: created.

Checkpoint file /postgresql/ogg/dirchk: created.

Process status files /postgresql/ogg/dirpcs: created.

SQL script files /postgresql/ogg/dirsql: created.

Database definitions files /postgresql/ogg/dirdef: created.

Extract data files /postgresql/ogg/dirdat: created.

Temporary files /postgresql/ogg/dirtmp: created.

Credential store files /postgresql/ogg/dircrd: created.

Master encryption key wallet files /postgresql/ogg/dirwlt: created.

Dump files /postgresql/ogg/dirdmp: created.

3.6 配置mgr组

edit params mgr

GGSCI (tencent) 2> view param mgr

port 7810

GGSCI (tencent) 3> info all

Program Status Group Lag at Chkpt Time Since Chkpt

MANAGER RUNNING

3.7 添加checkpoint表

为了保证源和目标端在传数据的时候不会重复或者少传，在目标端添加checkpoin表

GGSCI (tencent) 4> dblogin sourcedb gg_pg userid ogg,password ogg

2023-07-26 11:12:21 INFO OGG-03036 Database character set identified as UTF-8. Locale: en_US.utf8.

2023-07-26 11:12:21 INFO OGG-03037 Session character set identified as UTF-8.

Successfully logged into database.

GGSCI (tencent as ogg@gg_pg) 2> add checkpointtable ogg.checkpointtab

Successfully created checkpoint table ogg.checkpointtab.

3.8 配置replicate进程

#编辑参数文件

GGSCI (tencent) 6>edit params repa

REPLICAT repa

SOURCEDEFS ./dirdef/defgen.def

SETENV(ODBCINI=/postgresql/ogg/odbc.ini)

SETENV(NLS_LANG=AMERICAN_AMERICA.ZHS16GBK)

TARGETDB gg_pg,userid ogg,password ogg

DISCARDFILE ./ogg/dirrpt/repa.rpt, append

MAP hr.*, TARGET hr.*;

#添加 Replicat 进程

add replicat repa,exttrail ./dirdat/pa,begin now,checkpointtable ogg.checkpointtab

#启动 Replicat 进程

GGSCI (tencent as ogg@gg_pg) 8> start repa

#查看进程启动情况

GGSCI (tencent as ogg@gg_pg) 9> info all

Program Status Group Lag at Chkpt Time Since Chkpt

MANAGER RUNNING

REPLICAT RUNNING REPA 00:00:00 00:00:00

第四章数据验证

--创建表结构

1、源端oracle执行

SQL> create table t1 (id int);

Table created.

#添加附加日志

GGSCI (tencent) 12> dblogin userid ogg,password ogg

Successfully logged into database.

GGSCI (tencent as ogg@orcl) 13> add trandata hr.*

2、目标端postgresql执行

ogg=# create user hr with password 'hr';

CREATE ROLE

ogg=# grant all on database ogg to hr;

GRANT

ogg=# create schema hr;

CREATE SCHEMA

ogg=# alter schema hr owner to hr;

ALTER SCHEMA

ogg=> create table hr.t1 (id int);

CREATE TABLE

ogg=> \d

List of relations

Schema | Name | Type | Owner

--------+------+-------+-------

hr | t1 | table | hr

(1 row)

--源端oracle插入数据

SQL> insert into t1 values(1);

1 row created.

SQL> commit;

Commit complete.

#查看数据插入情况

GGSCI (tencent) 10> stats exta

Sending STATS request to EXTRACT EXTA ...

Start of Statistics at 2023-07-26 11:35:09.

Output to /oracle/ogg/dirdat/ea:

Extracting from HR.T1 to HR.T1:

*** Total statistics since 2023-07-26 11:35:07 ***

Total inserts 1.00

Total updates 0.00

Total deletes 0.00

Total upserts 0.00

Total discards 0.00

Total operations 1.00

*** Daily statistics since 2023-07-26 11:35:07 ***

Total inserts 1.00

Total updates 0.00

Total deletes 0.00

Total upserts 0.00

Total discards 0.00

Total operations 1.00

*** Hourly statistics since 2023-07-26 11:35:07 ***

Total inserts 1.00

Total updates 0.00

Total deletes 0.00

Total upserts 0.00

Total discards 0.00

Total operations 1.00

*** Latest statistics since 2023-07-26 11:35:07 ***

Total inserts 1.00

Total updates 0.00

Total deletes 0.00

Total upserts 0.00

Total discards 0.00

Total operations 1.00

End of Statistics.

#捕获进程正常，捕获一条数据

GGSCI (tencent) 11> stats pumpa

Sending STATS request to EXTRACT PUMPA ...

Start of Statistics at 2023-07-26 11:35:15.

Output to ./dirdat/pa:

Extracting from HR.T1 to HR.T1:

*** Total statistics since 2023-07-26 11:35:07 ***

Total inserts 1.00

Total updates 0.00

Total deletes 0.00

Total upserts 0.00

Total discards 0.00

Total operations 1.00

*** Daily statistics since 2023-07-26 11:35:07 ***

Total inserts 1.00

Total updates 0.00

Total deletes 0.00

Total upserts 0.00

Total discards 0.00

Total operations 1.00

*** Hourly statistics since 2023-07-26 11:35:07 ***

Total inserts 1.00

Total updates 0.00

Total deletes 0.00

Total upserts 0.00

Total discards 0.00

Total operations 1.00

*** Latest statistics since 2023-07-26 11:35:07 ***

Total inserts 1.00

Total updates 0.00

Total deletes 0.00

Total upserts 0.00

Total discards 0.00

Total operations 1.00

End of Statistics.

#传输进程正常，传输一条数据

--目标端postgresql查看数据 [postgres@tencent ogg]$ psql ogg hr psql (14.6) Type "help" for help. ogg=> \c You are now connected to database "ogg" as user "hr". ogg=> select * from t1; id

(1 rows)

GGSCI (tencent as ogg@gg_pg) 35> stats repa

Sending STATS request to Replicat group REPA ...

Start of statistics at 2023-07-26 11:46:17.

Replicating from HR.T1 to hr.t1:

*** Total statistics since 2023-07-26 11:46:09 ***

Total inserts 1.00

Total updates 0.00

Total deletes 0.00

Total upserts 0.00

Total discards 0.00

Total operations 1.00