ogg 全量初始化进程,源端表的列时间字段是空值到目标端报错

为提高效率,提问时请提供以下信息,问题描述清晰可优先响应。

  • 【TiDB 版本】:tidb 4.0
  • 【问题描述】: 使用ogg 同步oracle 数据同步tidb

源端ogg 版本,数据库11.2.0.4
Oracle GoldenGate Command Interpreter for Oracle
Version 12.3.0.1.4 OGGCORE_12.3.0.1.0_PLATFORMS_180415.0359_FBO
Linux, x64, 64bit (optimized), Oracle 12c on Apr 16 2018 00:53:30
Operating system character set identified as UTF-8.

目标端版本tidb

Oracle GoldenGate Command Interpreter for MySQL
Version 12.3.0.1.5 OGGCORE_12.3.0.1.0_PLATFORMS_180501.2124
Linux, x64, 64bit (optimized), MySQL Enterprise on May 2 2018 10:58:16
Operating system character set identified as UTF-8.

问题描述
在做测试中init 全量初始化一个测试表,源端表列有一个字段有一个 a列 date 类型,可以为空,实际存储也是空值
目标端表 a列 datetime 类型 可以为空

同步报错

目标端日志
2020-07-22T11:18:18.552+0800 INFO OGG-06510 Oracle GoldenGate Delivery for MySQL: Using the following key columns for target table test.hs_o_e: e_no.
2020-07-22T11:18:18.560+0800 WARNING OGG-01004 Oracle GoldenGate Delivery for MySQL: Aborted grouped transaction on test.hs_o_e, Database error 1366 ([SQL error 1366]Incorrect datetime value: ‘’ for column ‘return_date’ at row 1
).
2020-07-22T11:18:18.563+0800 ERROR OGG-02245 Oracle GoldenGate Delivery for MySQL: Repositioning was attempted as REPERROR action, but is not supported by RMTTASK.
2020-07-22T11:18:23.565+0800 ERROR OGG-01668 Oracle GoldenGate Delivery for MySQL: PROCESS ABENDING.

若提问为性能优化、故障排查类问题,请下载脚本运行。终端输出的打印结果,请务必全选并复制粘贴上传。

  1. 看报错, ogg 将空值转换为 ‘’ 了?
  2. 这个时间是否有意义,是否可以在 ogg 中配置,如果时间为空改为当前时间,这样能否避免?

这个问题看来只能修改 sql_mode ,同步数据为 0000-00-00 00:00:00

好的,如果有这种格式,取消严格限制比价好吧

我看tidb 其实是可以对时间列存放NULL值得,为什么ogg 同步过来就不能转换为null 值了,这个可以实现嘛
mysql> select *from t1;
±-----±---------+
| id | job_time |
±-----±---------+
| 2 | NULL |
±-----±---------+
1 row in set (0.00 sec)

  1. 请问,ogg 在插入 tidb 时的 sql 是什么? 可以先试下能否直接使用命令行插入。

我测试了一下,在源端插入 create table test.t1(id int,namevarchar(10),d datetime);
insert into test.t1(id,name) values(3,‘b’)

源端
SQL> select *From test.t1;

                                 ID NAME       D

                                  3 b          

目标端

mysql> select *from t1;
±-----±-----±--------------------+
| id | name | d |
±-----±-----±--------------------+
| 3 | b | 0000-00-00 00:00:00 |
±-----±-----±--------------------+
2 rows in set (0.01 sec)

但是在目标端tidb 库直接
insert into t1 (id,name) values(2,‘a’);

mysql> select *from t1;
±-----±-----±--------------------+
| id | name | d |
±-----±-----±--------------------+
| 2 | a | NULL |
| 3 | b | 0000-00-00 00:00:00 |
±-----±-----±--------------------+
2 rows in set (0.01 sec)

当前tidb测试
mysql> select @@sql_mode;
±----------------------------------------------------------------------------------------------------------------------+
| @@sql_mode |
±----------------------------------------------------------------------------------------------------------------------+
| ONLY_FULL_GROUP_BY,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION |
±----------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

create table test.t1(id int,name varchar(10),d datetime not null default 0);
insert into test.t1(id,name) values(3,‘b’);
select * from t1;

此为 oracle 与 tidb 的实现不同,tidb 可以通过设置 default 0 来满足此需求,业务上可能需要通过增加 is null 的判断。

内部环境复现了这个问题

  • Oracle 版本:Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production

  • OGG 版本:Version 12.3.0.1.5 OGGCORE_12.3.0.1.0_PLATFORMS_180501.2124

  • 下游 TiDB 版本: v4.0.2

  • 在上游 Oracle 创建表

SQL> desc lzy.t1
 Name                                      Null?    Type
 ----------------------------------------- -------- ----------------------------
 ID                                                 NUMBER(38)
 NAME                                               VARCHAR2(10)
 D                                                  DATE
  • 在上游 datetime 字段上插入 null 值
SQL> insert into lzy.t1 values(2,'b',null);

1 row created.

SQL> commit;

Commit complete.

SQL> select * from lzy.t1;

        ID NAME       D
---------- ---------- ------------
         1 a          23-JUL-20
         2 b

SQL> select * from lzy.t1 where D is null;

        ID NAME       D
---------- ---------- ------------
         2 b

  • 查看 OGG 同步进程退出

GGSCI (node4235) 1> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING
REPLICAT    ABENDED     REP1        00:00:00      00:02:09
  • 查看 OGG 日志
2020-07-23T22:20:53.176+0800  WARNING OGG-01004  Oracle GoldenGate Delivery for MySQL:  Aborted grouped transaction on lzy.t1, Database error 1366 ([SQL error 1366]Incorrect datetime value: '' for column '
d' at row 1
                              ).
2020-07-23T22:20:53.176+0800  ERROR   OGG-02245  Oracle GoldenGate Delivery for MySQL:  Repositioning was attempted as REPERROR action, but is not supported by RMTTASK.
2020-07-23T22:20:58.183+0800  ERROR   OGG-01668  Oracle GoldenGate Delivery for MySQL:  PROCESS ABENDING.
  • 下游 TiDB 查询没有数据同步过来
MySQL [lzy]> select * from t1;
+------+------+---------------------+
| id   | name | d                   |
+------+------+---------------------+
|    1 | a    | 2020-07-23 23:29:15 |
+------+------+---------------------+
1 row in set (0.00 sec)

  • 下游 TiDB 日志
[2020/07/24 09:56:20.883 +08:00] [ERROR] [conn.go:728] ["command dispatched failed"] [conn=17] [connInfo="id:17, addr:172.16.4.235:46518 status:1, collation:latin1_swedish_ci, user:oggadmin"] [command=Execute] [status="inTxn:1, autocommit:0"] [sql="INSERT INTO `lzy`.`t1` (`id`,`name`,`d`) VALUES (?,?,?) [arguments: (2, b, )]"] [txn_mode=PESSIMISTIC] [err="[table:1366]Incorrect datetime value: '' for column 'd' at row 1\
github.com/pingcap/errors.AddStack\
\t/home/jenkins/agent/workspace/tidb_v4.0.2/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20190809092503-95897b64e011/errors.go:174\
github.com/pingcap/parser/terror.(*Error).GenWithStackByArgs\
\t/home/jenkins/agent/workspace/tidb_v4.0.2/go/pkg/mod/github.com/pingcap/parser@v0.0.0-20200623164729-3a18f1e5dceb/terror/terror.go:243\
github.com/pingcap/tidb/executor.(*InsertValues).handleErr\
\t/home/jenkins/agent/workspace/tidb_v4.0.2/go/src/github.com/pingcap/tidb/executor/insert_common.go:289\
github.com/pingcap/tidb/executor.(*InsertValues).fastEvalRow\
\t/home/jenkins/agent/workspace/tidb_v4.0.2/go/src/github.com/pingcap/tidb/executor/insert_common.go:353\
github.com/pingcap/tidb/executor.insertRows\
\t/home/jenkins/agent/workspace/tidb_v4.0.2/go/src/github.com/pingcap/tidb/executor/insert_common.go:223\
github.com/pingcap/tidb/executor.(*InsertExec).Next\
\t/home/jenkins/agent/workspace/tidb_v4.0.2/go/src/github.com/pingcap/tidb/executor/insert.go:262\
github.com/pingcap/tidb/executor.Next\
\t/home/jenkins/agent/workspace/tidb_v4.0.2/go/src/github.com/pingcap/tidb/executor/executor.go:248\
github.com/pingcap/tidb/executor.(*ExecStmt).handleNoDelayExecutor\
\t/home/jenkins/agent/workspace/tidb_v4.0.2/go/src/github.com/pingcap/tidb/executor/adapter.go:506\
github.com/pingcap/tidb/executor.(*ExecStmt).handlePessimisticDML\
\t/home/jenkins/agent/workspace/tidb_v4.0.2/go/src/github.com/pingcap/tidb/executor/adapter.go:525\
github.com/pingcap/tidb/executor.(*ExecStmt).handleNoDelay\
\t/home/jenkins/agent/workspace/tidb_v4.0.2/go/src/github.com/pingcap/tidb/executor/adapter.go:386\
github.com/pingcap/tidb/executor.(*ExecStmt).Exec\
\t/home/jenkins/agent/workspace/tidb_v4.0.2/go/src/github.com/pingcap/tidb/executor/adapter.go:356\
github.com/pingcap/tidb/session.runStmt\
\t/home/jenkins/agent/workspace/tidb_v4.0.2/go/src/github.com/pingcap/tidb/session/tidb.go:276\
github.com/pingcap/tidb/session.(*session).CommonExec\
\t/home/jenkins/agent/workspace/tidb_v4.0.2/go/src/github.com/pingcap/tidb/session/session.go:1211\
github.com/pingcap/tidb/session.(*session).ExecutePreparedStmt\
\t/home/jenkins/agent/workspace/tidb_v4.0.2/go/src/github.com/pingcap/tidb/session/session.go:1328\
github.com/pingcap/tidb/server.(*TiDBStatement).Execute\
\t/home/jenkins/agent/workspace/tidb_v4.0.2/go/src/github.com/pingcap/tidb/server/driver_tidb.go:76\
github.com/pingcap/tidb/server.(*clientConn).handleStmtExecute\
\t/home/jenkins/agent/workspace/tidb_v4.0.2/go/src/github.com/pingcap/tidb/server/conn_stmt.go:184\
github.com/pingcap/tidb/server.(*clientConn).dispatch\
\t/home/jenkins/agent/workspace/tidb_v4.0.2/go/src/github.com/pingcap/tidb/server/conn.go:912\
github.com/pingcap/tidb/server.(*clientConn).Run\
\t/home/jenkins/agent/workspace/tidb_v4.0.2/go/src/github.com/pingcap/tidb/server/conn.go:713\
github.com/pingcap/tidb/server.(*Server).onConn\
\t/home/jenkins/agent/workspace/tidb_v4.0.2/go/src/github.com/pingcap/tidb/server/server.go:415\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357\
INSERT INTO `lzy`.`t1` (`id`,`name`,`d`) VALUES (?,?,?) [arguments: (2, b, )]"]

从测试结果看上有 date 字段插入 null 值,会被 OGG 以 ‘’ 插入到 TiDB,在严格模式下 ,TiDB 的 datetime 字段不能插入 ‘’
建议

  • 可以选择将上游的 null 值数据,修改为某个符合数据类型要求的时间
  • 或者将下游的 sql_mode 修改去掉严格模式

此话题已在最后回复的 1 分钟后被自动关闭。不再允许新回复。