DM迁移时PD当中这个配置key-type起到什么作用？对迁移有什么影响，有具体说明，如何进行设置么？

[Whm412]

为提高效率，提问时请提供以下信息，问题描述清晰可优先响应。

• 【TiDB 版本】：TiDB 4.0-rc
• 【问题描述】：原来由于TiDB当中有很多的empty region，但还不合并，所以将这个值改为了tnx，但今天发现出错了，具体原还还不太清楚。后来改成raw也不行。那个task还是出错。
  所以需要知道这个参数要改成什么合适，对整体TiDB有什么影响，是不是可以随意更改？（因为已经更改了，不知道有什么影响）
  

“msg”: “[code=10006:class=database:scope=not-set:level=high] execute statement failed: REPLACE INTO cop_m1_cm.sale_car_pay_check_inout_day_record (id,create_user,create_time,modified_time,modified_user,company_id,vehicle_id,business_id,business_type,in_out_time,in_out_type,car_moved,record_day) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?);: Error 1105: unexpected resolve err: abort:"Txn(Mvcc(Committed { commit_ts: TimeStamp(417050726311657477) }))” , lock: key: {tableID=1582093, handle=1266761347419541515}, primary: {tableID=1330526, handle=1266825489017475073}, txnStartTS: 417050726272335879, lockForUpdateTS:0, ttl: 3093, type: Del\ngithub.com/pingcap/dm/pkg/terror.(*Error).Delegate\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\ngithub.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:182\ngithub.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore.func2\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:260\ngithub.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71\ngithub.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:203\ngithub.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:255\ngithub.com/pingcap/dm/syncer.(*DBConn).executeSQL\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:282\ngithub.com/pingcap/dm/syncer.(*Syncer).sync.func3\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:979\ngithub.com/pingcap/dm/syncer.(*Syncer).sync\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1011\ngithub.com/pingcap/dm/syncer.(*Syncer).Run.func2\ \t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1092\ runtime.goexit\ \t/usr/local/go/src/runtime/asm_amd64.s:1357"

若提问为性能优化、故障排查类问题，请下载脚本运行。终端输出的打印结果，请务必全选并复制粘贴上传。

[yilong (yi888long)]

1. empty region 可以参考以下两个帖子

[FAQ] DM 导入变慢 empty-region-count 很多

[问题澄清] DM 导入变慢 [原因分析] 查看 empty-region-count 很多 [解决方案] 开启 region merge 后，速度恢复到和初始导入一致 [参考案例]

[FAQ] 解决已经开启 region merge， empty-region-count 仍然很多问题

【问题澄清】 检查监控，发现 [overview - pd - region health[empty-region-count]] 空 region 达到了 559K，但是已经开启了 region merge，空 region 也没有明显的下降，这是为何？ [image] [image] 【知识点引入】 参数解释： pd 参数解释： < v3.1 使用 namespace-cl…

2. 报错是修改 key-type 后出现的报错吗？

3. query-status 的完整查询结果，麻烦发送下。 DM-worker 的日志 和 tidb 的日志 报错时间端段的日志也反馈下，多谢。

[Whm412]

@rongyilong-PingCAP

DM-worker 的日志 和 tid

1、empty region的问题已经处理完了，也是按照上面处理的

2、key-type修改完很长时间后出现的。但上面贴子的异常里面有txn后，报了上面的异常，请仔细看里面的异常信息，包含了Txn的字样。

3、这个问题已经有快两天时间了，今天早发现后，又将key-type改为了raw，重启了dm集群。具体worker日志和pd日志可能不全，先传上去看一下吧

pd.log (4.2 MB) dm-worker.log (4.8 MB)

[Whm412]

使用query-status还能查出来数据，但使用qurey-error，就会显示出很多很多的sql语句,只是将最下面的截取了一下

[yilong (yi888long)]

1. 查看 PD 日志 ，有报错日志 ，请检查当前的 集群是否正常，pd-ctl 中反馈 member ，store ，health 结果

[2020/06/01 09:47:56.006 +08:00] [ERROR] [grpclog.go:75] [“transport: Got too many pings from the client, closing the connection.”] [2020/06/01 09:47:56.006 +08:00] [ERROR] [grpclog.go:75] [“transport: loopyWriter.run returning. Err: transport: Connection closing”] [2020/06/01 09:47:56.006 +08:00] [WARN] [grpclog.go:60] [“transport: http2Server.HandleStreams failed to read frame: read tcp 10.7.110.116:2379->10.7.110.113:51642: use of closed network connection”]

2. dm-worker 中 有断开 日志

[code=10003:class=database:scope=downstream:level=high] database driver: invalid connection

3. 请问当前同步正常吗？ 能否反馈 query-status 完整结果，多谢。

[Whm412]

@rongyilong-PingCAP

member ，store ，health

现在还有异常信息，正在想着，是不是要重新做这个task呢

1、member信息： member
{
“header”: {
“cluster_id”: 6812808469366657611
},
“members”: [
{
“name”: “pd_kaikaitidb3”,
“member_id”: 10885746490409258171,
“peer_urls”: [
“http://10.7.110.114:2380”
],
“client_urls”: [
“http://10.7.110.114:2379”
],
“deploy_path”: “/data1/deploy/bin”,
“binary_version”: “v4.0.0-rc”
},
{
“name”: “pd_kaikaitidb5”,
“member_id”: 12884478277380313438,
“peer_urls”: [
“http://10.7.110.116:2380”
],
“client_urls”: [
“http://10.7.110.116:2379”
],
“deploy_path”: “/data1/deploy/bin”,
“binary_version”: “v4.0.0-rc”
},
{
“name”: “pd_kaikaitidb4”,
“member_id”: 14427905117949164962,
“peer_urls”: [
“http://10.7.110.115:2380”
],
“client_urls”: [
“http://10.7.110.115:2379”
],
“deploy_path”: “/data1/deploy/bin”,
“binary_version”: “v4.0.0-rc”
}
],
“leader”: {
“name”: “pd_kaikaitidb4”,
“member_id”: 14427905117949164962,
“peer_urls”: [
“http://10.7.110.115:2380”
],
“client_urls”: [
“http://10.7.110.115:2379”
]
},
“etcd_leader”: {
“name”: “pd_kaikaitidb4”,
“member_id”: 14427905117949164962,
“peer_urls”: [
“http://10.7.110.115:2380”
],
“client_urls”: [
“http://10.7.110.115:2379”
],
“deploy_path”: “/data1/deploy/bin”,
“binary_version”: “v4.0.0-rc”
}
}


store信息：
{
“count”: 3,
“stores”: [
{
“store”: {
“id”: 1,
“address”: “10.7.110.117:20160”,
“version”: “4.0.0-rc”,
“status_address”: “10.7.110.117:20180”,
“git_hash”: “f45d0c963df3ee4b1011caf5eb146cacd1fbbad8”,
“start_timestamp”: 1590648746,
“last_heartbeat”: 1590982474971669124,
“state_name”: “Up”
},
“status”: {
“capacity”: “787.3GiB”,
“available”: “676.5GiB”,
“used_size”: “24.73GiB”,
“leader_count”: 666,
“leader_weight”: 1,
“leader_score”: 666,
“leader_size”: 60756,
“region_count”: 1338,
“region_weight”: 1,
“region_score”: 120422,
“region_size”: 120422,
“start_ts”: “2020-05-28T14:52:26+08:00”,
“last_heartbeat_ts”: “2020-06-01T11:34:34.971669124+08:00”,
“uptime”: “92h42m8.971669124s”
}
},
{
“store”: {
“id”: 4,
“address”: “10.7.110.119:20160”,
“version”: “4.0.0-rc”,
“status_address”: “10.7.110.119:20180”,
“git_hash”: “f45d0c963df3ee4b1011caf5eb146cacd1fbbad8”,
“start_timestamp”: 1590648725,
“last_heartbeat”: 1590982477896753357,
“state_name”: “Up”
},
“status”: {
“capacity”: “787.3GiB”,
“available”: “677.7GiB”,
“used_size”: “25.45GiB”,
“leader_count”: 672,
“leader_weight”: 1,
“leader_score”: 672,
“leader_size”: 60595,
“region_count”: 1349,
“region_weight”: 1,
“region_score”: 120607,
“region_size”: 120607,
“start_ts”: “2020-05-28T14:52:05+08:00”,
“last_heartbeat_ts”: “2020-06-01T11:34:37.896753357+08:00”,
“uptime”: “92h42m32.896753357s”
}
},
{
“store”: {
“id”: 5,
“address”: “10.7.110.118:20160”,
“version”: “4.0.0-rc”,
“status_address”: “10.7.110.118:20180”,
“git_hash”: “f45d0c963df3ee4b1011caf5eb146cacd1fbbad8”,
“start_timestamp”: 1590648733,
“last_heartbeat”: 1590982483878507308,
“state_name”: “Up”
},
“status”: {
“capacity”: “787.3GiB”,
“available”: “683.8GiB”,
“used_size”: “25.31GiB”,
“leader_count”: 673,
“leader_weight”: 1,
“leader_score”: 673,
“leader_size”: 59741,
“region_count”: 1335,
“region_weight”: 1,
“region_score”: 121155,
“region_size”: 121155,
“start_ts”: “2020-05-28T14:52:13+08:00”,
“last_heartbeat_ts”: “2020-06-01T11:34:43.878507308+08:00”,
“uptime”: “92h42m30.878507308s”
}
}
]
}

health信息：
[
{
“name”: “pd_kaikaitidb3”,
“member_id”: 10885746490409258171,
“client_urls”: [
“http://10.7.110.114:2379”
],
“health”: true
},
{
“name”: “pd_kaikaitidb5”,
“member_id”: 12884478277380313438,
“client_urls”: [
“http://10.7.110.116:2379”
],
“health”: true
},
{
“name”: “pd_kaikaitidb4”,
“member_id”: 14427905117949164962,
“client_urls”: [
“http://10.7.110.115:2379”
],
“health”: true
}
]

3、query-status完成信息：
query-status cop_m1_all_cm
{
“result”: true,
“msg”: “”,
“workers”: [
{
“result”: true,
“worker”: “10.7.110.98:8262”,
“msg”: “”,
“subTaskStatus”: [
{
“name”: “cop_m1_all_cm”,
“stage”: “Paused”,
“unit”: “Sync”,
“result”: {
“isCanceled”: false,
“errors”: [
{
“Type”: “ExecSQL”,
“msg”: “[code=10006:class=database:scope=not-set:level=high] execute statement failed: REPLACE INTO cop_m1_cm.cm_ops_maintain_dashboard (id,company_id,statistic_type,title,content,param,update_time,sub_statistic_type) VALUES (?,?,?,?,?,?,?,?);: Error 1105: unexpected resolve err: abort:"Txn(Mvcc(Committed { commit_ts: TimeStamp(417066463405801519) }))" , lock: key: {tableID=1990491, handle=1263350373318930433}, primary: {tableID=1330526, handle=1266845872634466306}, txnStartTS: 417066463379587105, lockForUpdateTS:0, ttl: 3074, type: Del
github.com/pingcap/dm/pkg/terror.(*Error).Delegate
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:182
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore.func2
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:260
github.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71
github.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:203
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:255
github.com/pingcap/dm/syncer.(*DBConn).executeSQL
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:282
github.com/pingcap/dm/syncer.(*Syncer).sync.func3
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:979
github.com/pingcap/dm/syncer.(*Syncer).sync
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1011
github.com/pingcap/dm/syncer.(*Syncer).Run.func2
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1092
runtime.goexit
\t/usr/local/go/src/runtime/asm_amd64.s:1357”,
“error”: null
},
{
“Type”: “ExecSQL”,
“msg”: “[code=10006:class=database:scope=not-set:level=high] execute statement failed: begin: sql: connection is already closed
github.com/pingcap/dm/pkg/terror.(*Error).Delegate
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:145
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore.func2
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:260
github.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71
github.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:203
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:255
github.com/pingcap/dm/syncer.(*DBConn).executeSQL
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:282
github.com/pingcap/dm/syncer.(*Syncer).sync.func3
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:979
github.com/pingcap/dm/syncer.(*Syncer).sync
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1011
github.com/pingcap/dm/syncer.(*Syncer).Run.func2
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1092
runtime.goexit
\t/usr/local/go/src/runtime/asm_amd64.s:1357”,
“error”: null
},
{
“Type”: “ExecSQL”,
“msg”: “[code=10006:class=database:scope=not-set:level=high] execute statement failed: begin: driver: bad connection
github.com/pingcap/dm/pkg/terror.(*Error).Delegate
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:145
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore.func2
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:260
github.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71
github.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:203
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:255
github.com/pingcap/dm/syncer.(*DBConn).executeSQL
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:282
github.com/pingcap/dm/syncer.(*Syncer).sync.func3
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:979
github.com/pingcap/dm/syncer.(*Syncer).sync
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1011
github.com/pingcap/dm/syncer.(*Syncer).Run.func2
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1092
runtime.goexit
\t/usr/local/go/src/runtime/asm_amd64.s:1357”,
“error”: null
},
{
“Type”: “ExecSQL”,
“msg”: “[code=10006:class=database:scope=not-set:level=high] execute statement failed: begin: sql: connection is already closed
github.com/pingcap/dm/pkg/terror.(*Error).Delegate
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:145
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore.func2
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:260
github.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71
github.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:203
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:255
github.com/pingcap/dm/syncer.(*DBConn).executeSQL
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:282
github.com/pingcap/dm/syncer.(*Syncer).sync.func3
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:979
github.com/pingcap/dm/syncer.(*Syncer).sync
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1011
github.com/pingcap/dm/syncer.(*Syncer).Run.func2
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1092
runtime.goexit
\t/usr/local/go/src/runtime/asm_amd64.s:1357”,
“error”: null
},
{
“Type”: “ExecSQL”,
“msg”: “[code=10006:class=database:scope=not-set:level=high] execute statement failed: begin: sql: connection is already closed
github.com/pingcap/dm/pkg/terror.(*Error).Delegate
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:145
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore.func2
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:260
github.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71
github.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:203
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:255
github.com/pingcap/dm/syncer.(*DBConn).executeSQL
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:282
github.com/pingcap/dm/syncer.(*Syncer).sync.func3
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:979
github.com/pingcap/dm/syncer.(*Syncer).sync
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1011
github.com/pingcap/dm/syncer.(*Syncer).Run.func2
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1092
runtime.goexit
\t/usr/local/go/src/runtime/asm_amd64.s:1357”,
“error”: null
},
{
“Type”: “ExecSQL”,
“msg”: “[code=10006:class=database:scope=not-set:level=high] execute statement failed: begin: sql: connection is already closed
github.com/pingcap/dm/pkg/terror.(*Error).Delegate
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:145
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore.func2
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:260
github.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71
github.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:203
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:255
github.com/pingcap/dm/syncer.(*DBConn).executeSQL
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:282
github.com/pingcap/dm/syncer.(*Syncer).sync.func3
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:979
github.com/pingcap/dm/syncer.(*Syncer).sync
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1011
github.com/pingcap/dm/syncer.(*Syncer).Run.func2
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1092
runtime.goexit
\t/usr/local/go/src/runtime/asm_amd64.s:1357”,
“error”: null
},
{
“Type”: “ExecSQL”,
“msg”: “[code=10006:class=database:scope=not-set:level=high] execute statement failed: begin: sql: connection is already closed
github.com/pingcap/dm/pkg/terror.(*Error).Delegate
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:145
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore.func2
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:260
github.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71
github.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:203
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:255
github.com/pingcap/dm/syncer.(*DBConn).executeSQL
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:282
github.com/pingcap/dm/syncer.(*Syncer).sync.func3
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:979
github.com/pingcap/dm/syncer.(*Syncer).sync
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1011
github.com/pingcap/dm/syncer.(*Syncer).Run.func2
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1092
runtime.goexit
\t/usr/local/go/src/runtime/asm_amd64.s:1357”,
“error”: null
},
{
“Type”: “ExecSQL”,
“msg”: “[code=10006:class=database:scope=not-set:level=high] execute statement failed: begin: sql: connection is already closed
github.com/pingcap/dm/pkg/terror.(*Error).Delegate
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:145
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore.func2
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:260
github.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71
github.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:203
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:255
github.com/pingcap/dm/syncer.(*DBConn).executeSQL
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:282
github.com/pingcap/dm/syncer.(*Syncer).sync.func3
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:979
github.com/pingcap/dm/syncer.(*Syncer).sync
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1011
github.com/pingcap/dm/syncer.(*Syncer).Run.func2
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1092
runtime.goexit
\t/usr/local/go/src/runtime/asm_amd64.s:1357”,
“error”: null
},
{
“Type”: “ExecSQL”,
“msg”: “[code=10006:class=database:scope=not-set:level=high] execute statement failed: begin: sql: connection is already closed
github.com/pingcap/dm/pkg/terror.(*Error).Delegate
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:145
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore.func2
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:260
github.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71
github.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:203
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:255
github.com/pingcap/dm/syncer.(*DBConn).executeSQL
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:282
github.com/pingcap/dm/syncer.(*Syncer).sync.func3
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:979
github.com/pingcap/dm/syncer.(*Syncer).sync
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1011
github.com/pingcap/dm/syncer.(*Syncer).Run.func2
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1092
runtime.goexit
\t/usr/local/go/src/runtime/asm_amd64.s:1357”,
“error”: null
},
{
“Type”: “ExecSQL”,
“msg”: “[code=10006:class=database:scope=not-set:level=high] execute statement failed: begin: sql: connection is already closed
github.com/pingcap/dm/pkg/terror.(*Error).Delegate
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:145
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore.func2
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:260
github.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71
github.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:203
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:255
github.com/pingcap/dm/syncer.(*DBConn).executeSQL
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:282
github.com/pingcap/dm/syncer.(*Syncer).sync.func3
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:979
github.com/pingcap/dm/syncer.(*Syncer).sync
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1011
github.com/pingcap/dm/syncer.(*Syncer).Run.func2
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1092
runtime.goexit
\t/usr/local/go/src/runtime/asm_amd64.s:1357”,
“error”: null
},
{
“Type”: “ExecSQL”,
“msg”: “[code=10006:class=database:scope=not-set:level=high] execute statement failed: begin: sql: connection is already closed
github.com/pingcap/dm/pkg/terror.(*Error).Delegate
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:145
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore.func2
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:260
github.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71
github.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:203
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:255
github.com/pingcap/dm/syncer.(*DBConn).executeSQL
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:282
github.com/pingcap/dm/syncer.(*Syncer).sync.func3
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:979
github.com/pingcap/dm/syncer.(*Syncer).sync
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1011
github.com/pingcap/dm/syncer.(*Syncer).Run.func2
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1092
runtime.goexit
\t/usr/local/go/src/runtime/asm_amd64.s:1357”,
“error”: null
},
{
“Type”: “ExecSQL”,
“msg”: “[code=10006:class=database:scope=not-set:level=high] execute statement failed: begin: sql: connection is already closed
github.com/pingcap/dm/pkg/terror.(*Error).Delegate
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:145
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore.func2
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:260
github.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71
github.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:203
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:255
github.com/pingcap/dm/syncer.(*DBConn).executeSQL
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:282
github.com/pingcap/dm/syncer.(*Syncer).sync.func3
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:979
github.com/pingcap/dm/syncer.(*Syncer).sync
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1011
github.com/pingcap/dm/syncer.(*Syncer).Run.func2
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1092
runtime.goexit
\t/usr/local/go/src/runtime/asm_amd64.s:1357”,
“error”: null
}
],
“detail”: null
},
“unresolvedDDLLockID”: “”,
“sync”: {
“totalEvents”: “2296562”,
“totalTps”: “0”,
“recentTps”: “0”,
“masterBinlog”: “(mysqlslave-bin.000204, 68674683)”,
“masterBinlogGtid”: “”,
“syncerBinlog”: “(mysqlslave-bin|000001.000189, 782029641)”,
“syncerBinlogGtid”: “”,
“blockingDDLs”: [
],
“unresolvedGroups”: [
],
“synced”: false
}
}
],
“relayStatus”: {
“masterBinlog”: “(mysqlslave-bin.000204, 68674683)”,
“masterBinlogGtid”: “”,
“relaySubDir”: “965117b9-e61f-11e8-ba1e-0242ac110002.000001”,
“relayBinlog”: “(mysqlslave-bin.000204, 68674683)”,
“relayBinlogGtid”: “”,
“relayCatchUpMaster”: true,
“stage”: “Running”,
“result”: null
},
“sourceID”: “cop-m1”
}
]
}

[imtbkcat-PingCAP (Lingyu Song)]

你好，可以提供一下 TiDB 日志，方便我们进一步调查吗。

[Whm412]

你好，只取了最近的日志，日志太大了。
tidb.log (4.2 MB)

[Whm412]

你好，还需要提供什么？业务在使用，如果需要时间长，这边先将这个task重新部署一下。

[imtbkcat-PingCAP (Lingyu Song)]

你好，还需要提供下和 tidb 日志同一时间段的pd和tikv日志。

[yilong (yi888long)]

1. 麻烦再查一下 curl http://{TiDBIP}:10080/mvcc/key/{db}/{table}/1266846852033810434 的结果

table id= 1330526， 您可以查一下 select * from INFORMATION_SCHEMA.tables where tidb_table_id=1330526; 确认表名和库名

[Whm412]

@rongyilong-PingCAP

db

日志暂时没去查，收集起来有些麻烦了，这个查出来了。
http://10.7.110.103:10080/mvcc/key/cop_m1_cm/home_page_remind_statistics/1266846852033810434

[Whm412]

针对这个问题，这边由于业务要求，只能是重新部署了一task，现象没办法重现了。重新部署的方法：
1、原task1，tidb中库名tidb1，现在新建一个task2，向tidb2中完成同步
2、task2完成后，将task1和task2都停止掉，这一步一定要做
3、准备好脚本，可用查询语句生成。将所有tidb2中的表都转称到tidb1当中，如果tidb1当中存在，就drop掉。drop table if exists tidb1.table1;RENAME TABLE tidb2.table1 to tidb1.table1;执行完成后，tidb2中的表就都没有了
4、将两张checkpoint表都由task2转到task1当中，也可以使用脚本实现，原来的数据也没用了
5、将dm-worker当中的两个文件夹名称互换一下
6、重启task1(task2就不需要了，可以将相关信息都删除掉）
按照上面的方法，可以将正在使用的同步库，完成一次影响最小的迁移，是一个表一个表迁移的。
可能还有其他的办法，但没有试，这个是成功的

[yilong (yi888long)]

感谢反馈，上面的问题，我们再尽力研究一下，如果有更新会尽快回复，多谢。

[yilong (yi888long)]

您好，能麻烦收集下 tikv 出错时间的日志吗？ 可能还是需要这个日志查看下，辛苦反馈下，多谢。

[Whm412]

你好，这个收集不到了，被另外一个同事给删除掉了，现在日志占用的空间有些多，有时候会随手将过期的日志给删除掉。

[yilong (yi888long)]

好的，感谢您的反馈，如果再遇到这个问题，麻烦再继续跟下帖子，麻烦了，多谢。