ETL异构数据源Datax_Oracle同步MySQL(全量)

ETL异构数据源Datax_Oracle同步MySQL(全量)_04

文章目录

- - - - 一、Oracle同步Mysql
      - 1. 构建json
        2. 执行数据同步
        3. 查看同步数据
        4. 同步数据正确性和准确性
        5. 同步日志分析

一、Oracle同步Mysql

1. 构建json

vim oracle2mysql.json

{"job": {"setting": {"speed": {"channel": 3},"errorLimit": {"record": 0,"percentage": 0.02}},"content": [{"reader": {"name": "oraclereader","parameter": {"column": ["IDNO","COL1","COL2","COL3","DT","COL5","COL6","COL7","COL8","COL9","COL10"],"connection": [{"jdbcUrl": ["jdbc:oracle:thin:@192.xxx.xxx.xxx:1521:orcl"],"table": ["TEST.OTBS1"]}],"username": "username","password": "password"}},"writer": {"name": "mysqlwriter","parameter": {"column": ["IDNO","COL1","COL2","COL3","DT","COL5","COL6","COL7","COL8","COL9","COL10"],"connection": [{"jdbcUrl": "jdbc:mysql://127.0.0.1:3306/datax?autoReconnect=true&useUnicode=true&characterEncoding=utf8&zeroDateTimeBehavior=CONVERT_TO_NULL&useSSL=false&serverTimezone=CTT&nullCatalogMeansCurrent=true","table": ["otbs1"]}],"username": "root","password": "123456"}}}]}
}

2. 执行数据同步

python /app/datax/bin/datax.py /app/datax/job/oracle2mysql.json

2021-06-23 12:28:12.390 [job-0] INFO  StandAloneJobContainerCommunicator - Total 1048576 records, 69143488 bytes | Speed 1.65MB/s, 26214 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 26.038s |  All Task WaitReaderTime 8.483s | Percentage 100.00%
2021-06-23 12:28:12.402 [job-0] INFO  JobContainer - 
任务启动时刻                    : 2021-06-23 12:27:31
任务结束时刻                    : 2021-06-23 12:28:12
任务总计耗时                    :                 41s
任务平均流量                    :            1.65MB/s
记录写入速度                    :          26214rec/s
读出记录总数                    :             1048576
读写失败总数                    :                   0

在这里插入图片描述

3. 查看同步数据

登陆mysql
在这里插入图片描述

4. 同步数据正确性和准确性

数据量的准确性和数据的正确性
在这里插入图片描述

Oracle同步数据量：1048576
Mysql插入数据量：1048576

DT同步日期没问题
COL5数据也没有都是精度

5. 同步日志分析

2021-06-23 12:27:32.228 [job-0] INFO  JobContainer - jobContainer starts to do prepare ...
2021-06-23 12:27:32.229 [job-0] INFO  JobContainer - DataX Reader.Job [oraclereader] do prepare work .
2021-06-23 12:27:32.230 [job-0] INFO  JobContainer - DataX Writer.Job [mysqlwriter] do prepare work .
2021-06-23 12:27:32.232 [job-0] INFO  JobContainer - jobContainer starts to do split ...
2021-06-23 12:27:32.233 [job-0] INFO  JobContainer - Job set Channel-Number to 3 channels.
2021-06-23 12:27:32.242 [job-0] INFO  JobContainer - DataX Reader.Job [oraclereader] splits to [1] tasks.
2021-06-23 12:27:32.243 [job-0] INFO  JobContainer - DataX Writer.Job [mysqlwriter] splits to [1] tasks.
2021-06-23 12:27:32.276 [job-0] INFO  JobContainer - jobContainer starts to do schedule ...
2021-06-23 12:27:32.280 [job-0] INFO  JobContainer - Scheduler starts [1] taskGroups.
2021-06-23 12:27:32.283 [job-0] INFO  JobContainer - Running by standalone Mode.
2021-06-23 12:27:32.300 [taskGroup-0] INFO  TaskGroupContainer - taskGroupId=[0] start [1] channels for [1] tasks.
2021-06-23 12:27:32.314 [taskGroup-0] INFO  Channel - Channel set byte_speed_limit to -1, No bps activated.
2021-06-23 12:27:32.315 [taskGroup-0] INFO  Channel - Channel set record_speed_limit to -1, No tps activated.
2021-06-23 12:27:32.348 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started