dm同步失败

为提高效率,提问时请提供以下信息,问题描述清晰可优先响应。
* 【TiDB 版本】:V3.0.12

* 【问题描述】:
? query-status mysql-replica-boe
{
    "result": true,
    "msg": "",
    "workers": [
        {
            "result": true,
            "worker": "10.1.0.95:8262",
            "msg": "",
            "subTaskStatus": [
                {
                    "name": "mysql-replica-boe",
                    "stage": "Paused",
                    "unit": "Sync",
                    "result": {
                        "isCanceled": false,
                        "errors": [
                            {
                                "Type": "ExecSQL",
                                "msg": "[code=10006:class=database:scope=not-set:level=high] execute statement failed: commit: Error 9007: Write conflict, txnStartTS=416065315202924546, conflictStartTS=416065315111174146, conflictCommitTS=416065315347103754, key={tableID=27158, indexID=1, indexValues={99485, }} primary={tableID=27158, indexID=1, indexValues={97296, }} [try again later]\ngithub.com/pingcap/dm/pkg/terror.(*Error).Delegate\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\ngithub.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:188\ngithub.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore.func3\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:268\ngithub.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71\ngithub.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:204\ngithub.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:263\ngithub.com/pingcap/dm/syncer.(*DBConn).executeSQL\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:290\ngithub.com/pingcap/dm/syncer.(*Syncer).sync.func3\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:980\ngithub.com/pingcap/dm/syncer.(*Syncer).sync\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1022\ngithub.com/pingcap/dm/syncer.(*Syncer).Run.func2\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1093\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357",
                                "error": null
                            },
                            {
                                "Type": "ExecSQL",
                                "msg": "[code=10006:class=database:scope=not-set:level=high] execute statement failed: begin: sql: connection is already closed\ngithub.com/pingcap/dm/pkg/terror.(*Error).Delegate\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\ngithub.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:146\ngithub.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore.func3\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:268\ngithub.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71\ngithub.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:204\ngithub.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:263\ngithub.com/pingcap/dm/syncer.(*DBConn).executeSQL\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:290\ngithub.com/pingcap/dm/syncer.(*Syncer).sync.func3\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:980\ngithub.com/pingcap/dm/syncer.(*Syncer).sync\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1022\ngithub.com/pingcap/dm/syncer.(*Syncer).Run.func2\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1093\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357",
                                "error": null
                            },
                            {
                                "Type": "ExecSQL",
                                "msg": "[code=10006:class=database:scope=not-set:level=high] execute statement failed: begin: sql: connection is already closed\ngithub.com/pingcap/dm/pkg/terror.(*Error).Delegate\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\ngithub.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:146\ngithub.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore.func3\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:268\ngithub.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71\ngithub.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:204\ngithub.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:263\ngithub.com/pingcap/dm/syncer.(*DBConn).executeSQL\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:290\ngithub.com/pingcap/dm/syncer.(*Syncer).sync.func3\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:980\ngithub.com/pingcap/dm/syncer.(*Syncer).sync\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1022\ngithub.com/pingcap/dm/syncer.(*Syncer).Run.func2\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1093\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357",
                                "error": null
                            },
                            {
                                "Type": "ExecSQL",
                                "msg": "[code=10006:class=database:scope=not-set:level=high] execute statement failed: begin: driver: bad connection\ngithub.com/pingcap/dm/pkg/terror.(*Error).Delegate\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\ngithub.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:146\ngithub.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore.func3\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:268\ngithub.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71\ngithub.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:204\ngithub.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:263\ngithub.com/pingcap/dm/syncer.(*DBConn).executeSQL\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:290\ngithub.com/pingcap/dm/syncer.(*Syncer).sync.func3\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:980\ngithub.com/pingcap/dm/syncer.(*Syncer).sync\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1022\ngithub.com/pingcap/dm/syncer.(*Syncer).Run.func2\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1093\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357",
                                "error": null
                            },
                            {
                                "Type": "ExecSQL",
                                "msg": "[code=10006:class=database:scope=not-set:level=high] execute statement failed: begin: driver: bad connection\ngithub.com/pingcap/dm/pkg/terror.(*Error).Delegate\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\ngithub.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:146\ngithub.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore.func3\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:268\ngithub.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71\ngithub.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:204\ngithub.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:263\ngithub.com/pingcap/dm/syncer.(*DBConn).executeSQL\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:290\ngithub.com/pingcap/dm/syncer.(*Syncer).sync.func3\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:980\ngithub.com/pingcap/dm/syncer.(*Syncer).sync\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1022\ngithub.com/pingcap/dm/syncer.(*Syncer).Run.func2\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1093\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357",
                                "error": null
                            },
                            {
                                "Type": "ExecSQL",
                                "msg": "[code=10006:class=database:scope=not-set:level=high] execute statement failed: begin: driver: bad connection\ngithub.com/pingcap/dm/pkg/terror.(*Error).Delegate\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\ngithub.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:146\ngithub.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore.func3\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:268\ngithub.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71\ngithub.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:204\ngithub.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:263\ngithub.com/pingcap/dm/syncer.(*DBConn).executeSQL\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:290\ngithub.com/pingcap/dm/syncer.(*Syncer).sync.func3\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:980\ngithub.com/pingcap/dm/syncer.(*Syncer).sync\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1022\ngithub.com/pingcap/dm/syncer.(*Syncer).Run.func2\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1093\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357",
                                "error": null
                            },
                            {
                                "Type": "ExecSQL",
                                "msg": "[code=10006:class=database:scope=not-set:level=high] execute statement failed: begin: driver: bad connection\ngithub.com/pingcap/dm/pkg/terror.(*Error).Delegate\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\ngithub.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:146\ngithub.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore.func3\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:268\ngithub.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71\ngithub.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:204\ngithub.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:263\ngithub.com/pingcap/dm/syncer.(*DBConn).executeSQL\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:290\ngithub.com/pingcap/dm/syncer.(*Syncer).sync.func3\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:980\ngithub.com/pingcap/dm/syncer.(*Syncer).sync\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1022\ngithub.com/pingcap/dm/syncer.(*Syncer).Run.func2\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1093\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357",
                                "error": null
                            },
                            {
                                "Type": "ExecSQL",
                                "msg": "[code=10006:class=database:scope=not-set:level=high] execute statement failed: begin: driver: bad connection\ngithub.com/pingcap/dm/pkg/terror.(*Error).Delegate\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\ngithub.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:146\ngithub.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore.func3\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:268\ngithub.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71\ngithub.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:204\ngithub.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:263\ngithub.com/pingcap/dm/syncer.(*DBConn).executeSQL\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:290\ngithub.com/pingcap/dm/syncer.(*Syncer).sync.func3\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:980\ngithub.com/pingcap/dm/syncer.(*Syncer).sync\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1022\ngithub.com/pingcap/dm/syncer.(*Syncer).Run.func2\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1093\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357",
                                "error": null
                            },
                            {
                                "Type": "ExecSQL",
                                "msg": "[code=10006:class=database:scope=not-set:level=high] execute statement failed: begin: driver: bad connection\ngithub.com/pingcap/dm/pkg/terror.(*Error).Delegate\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\ngithub.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:146\ngithub.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore.func3\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:268\ngithub.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71\ngithub.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:204\ngithub.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:263\ngithub.com/pingcap/dm/syncer.(*DBConn).executeSQL\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:290\ngithub.com/pingcap/dm/syncer.(*Syncer).sync.func3\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:980\ngithub.com/pingcap/dm/syncer.(*Syncer).sync\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1022\ngithub.com/pingcap/dm/syncer.(*Syncer).Run.func2\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1093\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357",
                                "error": null
                            },
                            {
                                "Type": "ExecSQL",
                                "msg": "[code=10006:class=database:scope=not-set:level=high] execute statement failed: begin: driver: bad connection\ngithub.com/pingcap/dm/pkg/terror.(*Error).Delegate\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\ngithub.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:146\ngithub.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore.func3\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:268\ngithub.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71\ngithub.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:204\ngithub.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:263\ngithub.com/pingcap/dm/syncer.(*DBConn).executeSQL\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:290\ngithub.com/pingcap/dm/syncer.(*Syncer).sync.func3\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:980\ngithub.com/pingcap/dm/syncer.(*Syncer).sync\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1022\ngithub.com/pingcap/dm/syncer.(*Syncer).Run.func2\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1093\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357",
                                "error": null
                            },
                            {
                                "Type": "ExecSQL",
                                "msg": "[code=10006:class=database:scope=not-set:level=high] execute statement failed: begin: sql: connection is already closed\ngithub.com/pingcap/dm/pkg/terror.(*Error).Delegate\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\ngithub.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:146\ngithub.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore.func3\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:268\ngithub.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71\ngithub.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:204\ngithub.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:263\ngithub.com/pingcap/dm/syncer.(*DBConn).executeSQL\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:290\ngithub.com/pingcap/dm/syncer.(*Syncer).sync.func3\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:980\ngithub.com/pingcap/dm/syncer.(*Syncer).sync\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1022\ngithub.com/pingcap/dm/syncer.(*Syncer).Run.func2\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1093\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357",
                                "error": null
                            },
                            {
                                "Type": "ExecSQL",
                                "msg": "[code=10006:class=database:scope=not-set:level=high] execute statement failed: begin: driver: bad connection\ngithub.com/pingcap/dm/pkg/terror.(*Error).Delegate\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\ngithub.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:146\ngithub.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore.func3\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:268\ngithub.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71\ngithub.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:204\ngithub.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:263\ngithub.com/pingcap/dm/syncer.(*DBConn).executeSQL\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:290\ngithub.com/pingcap/dm/syncer.(*Syncer).sync.func3\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:980\ngithub.com/pingcap/dm/syncer.(*Syncer).sync\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1022\ngithub.com/pingcap/dm/syncer.(*Syncer).Run.func2\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1093\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357",
                                "error": null
                            },
                            {
                                "Type": "ExecSQL",
                                "msg": "[code=10006:class=database:scope=not-set:level=high] execute statement failed: begin: driver: bad connection\ngithub.com/pingcap/dm/pkg/terror.(*Error).Delegate\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\ngithub.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:146\ngithub.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore.func3\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:268\ngithub.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71\ngithub.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:204\ngithub.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:263\ngithub.com/pingcap/dm/syncer.(*DBConn).executeSQL\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:290\ngithub.com/pingcap/dm/syncer.(*Syncer).sync.func3\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:980\ngithub.com/pingcap/dm/syncer.(*Syncer).sync\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1022\ngithub.com/pingcap/dm/syncer.(*Syncer).Run.func2\n\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1093\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357",
                                "error": null
                            }
                        ],
                        "detail": null
                    },
                    "unresolvedDDLLockID": "",
                    "sync": {
                        "totalEvents": "55167",
                        "totalTps": "0",
                        "recentTps": "0",
                        "masterBinlog": "(mysql-bin.007106, 621819364)",
                        "masterBinlogGtid": "1961de6e-db49-11e9-9eb7-ac1f6b41525a:1-2639441422,22863285-e03a-11e9-a452-ac1f6b403494:1-8454425,704aa4cd-eaab-11e9-a391-ac1f6b41525a:1-930974510",
                        "syncerBinlog": "(mysql-bin|000001.006841, 934506002)",
                        "syncerBinlogGtid": "",
                        "blockingDDLs": [
                        ],
                        "unresolvedGroups": [
                        ],
                        "synced": false
                    }
                }
            ],
            "relayStatus": {
                "masterBinlog": "(mysql-bin.007106, 621819364)",
                "masterBinlogGtid": "1961de6e-db49-11e9-9eb7-ac1f6b41525a:1-2639441422,22863285-e03a-11e9-a452-ac1f6b403494:1-8454425,704aa4cd-eaab-11e9-a391-ac1f6b41525a:1-930974510",
                "relaySubDir": "22863285-e03a-11e9-a452-ac1f6b403494.000001",
                "relayBinlog": "(mysql-bin.007106, 621819364)",
                "relayBinlogGtid": "704aa4cd-eaab-11e9-a391-ac1f6b41525a:1-907941017,1961de6e-db49-11e9-9eb7-ac1f6b41525a:2035655839-2639441422,22863285-e03a-11e9-a452-ac1f6b403494:1-8454425",
                "relayCatchUpMaster": true,
                "stage": "Running",
                "result": null
            },
            "sourceID": "mysql-replica-boe"
        }
    ]
}
若提问为**性能优化、故障排查**类问题,请下载<a href="/uploads/short-url/uGisshjxFnxx1KgpFOYbfeZjsc6" download="info_gathering.py">脚本</a>运行。终端输出的打印结果,请**务必全选**并复制粘贴上传。

1.请问DM版本是什么?

  1. task文件 和 dm-worker日志麻烦上传下

  2. Write conflict 是有写冲突,看下这个表是否有主键,数据冲突了? connection is already closed ,这个问题这么频繁,在DM服务器和tidb服务器之间有防火墙吗? tidb有使用HAproxy之类的负载均衡吗?

dm-worker.log (1.3 MB) task_advanced.yaml (3.3 KB)

版本是1.0.4-1

3、没有弄防火墙的

dm-worker.rar (144.6 KB)

您好: 1. 先来检查这个报错:

  2. MySQL [INFORMATION_SCHEMA]> select table_name from tables where tidb_table_id=27158, 找到这个表
  
 3. 再从TIDB_INDEXES 表里看下索引id=1的是哪个索引
 4. show create table 表名,看下这个索引是不是唯一的,当时是不是已经有这个值138670,导致导入失败

请尝试重新resume-task看看能否恢复,多谢

MySQL [information_schema]> select table_name from tables where tidb_table_id=27158 ;

±----------------+ | table_name | ±----------------+ | if_ally_science | ±----------------+ 1 row in set (0.89 sec)

[information_schema]> show create table infinity_boe.if_ally_science\G *************************** 1. row *************************** Table: if_ally_science Create Table: CREATE TABLEif_ally_science(idbigint(20) NOT NULL AUTO_INCREMENT,datedate NOT NULL DEFAULT ‘0000-00-00’,kidint(5) NOT NULL DEFAULT ‘0’,aidint(11) NOT NULL DEFAULT ‘0’,scienceint(5) DEFAULT ‘0’, PRIMARY KEY (id), UNIQUE KEYu_k(aid), KEYidx_date(date) ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin AUTO_INCREMENT=120430

MySQL [infinity_boe]> select * from if_ally_science where id =138670\G Empty set (0.01 sec)

MySQL [infinity_boe]>

resume-task.txt (27.5 KB)

您好: 请尝试stop-task,再start-task多谢

试过了,也不行

stop-task,再start-task 后是一样出错吗

同步的表除了dm 有其它服务在写入吗

没有,只有dm,新的集群环境

stop,start之后的报错是否一样? 能否再上传一个query status的结果和dm-worker日志,多谢

TASK [import grafana dashboards - run import script] ****************************************************************************
fatal: [grafana -> localhost]: FAILED! => {"changed": true, "cmd": "python import_grafana_dashboards.py", "delta": "0:00:00.375524", "end": "2020-04-19 02:02:43.986551", "msg": "non-zero return code", "rc": 1, "start": "2020-04-19 02:02:43.611027", "stderr": "Traceback (most recent call last):\n  File \"import_grafana_dashboards.py\", line 95, in <module>\n    raise RuntimeError\nRuntimeError", "stderr_lines": ["Traceback (most recent call last):", "  File \"import_grafana_dashboards.py\", line 95, in <module>", "    raise RuntimeError", "RuntimeError"], "stdout": "[load] from <dm_instances.json>:dm_worker_instances\n[import] <Dm-Cluster-DM-worker-instances> to [Dm-Cluster]\t............. ERROR:  Unauthorized", "stdout_lines": ["[load] from <dm_instances.json>:dm_worker_instances", "[import] <Dm-Cluster-DM-worker-instances> to [Dm-Cluster]\t............. ERROR:  Unauthorized"]}
        to retry, use: --limit @/home/tidb/dm-ansible/retry_files/start.retry

帮忙看一下昨天tidb和dm 集群关闭查找问题,今天重启DM 报这个错误

您好:

    1. 请问DM有修改配置吗?
    2. 请查看dm_instances.json,inventory.ini是否修改了哪些配置信息,格式不对,多谢

就改了grafana_admin_password ,现在改回来了,也这样

dm-worker.log (2.1 MB) query_status.txt (297.6 KB)

你好,

根据 ERROR: Unauthorized 报错,

  1. 尝试查看该目录或者文件是否为部署管理用户权限。
  2. 是否使用部署管理用户操作
  3. 请在 inventory.ini 中配置正确 grafana 账户密码,正确性可通过 web 登录验证下

这个呢?这个是同步的问题

这个问题可以了,现在就是同步失败的问题,帮忙查一下原因,日志文件在

dm-worker.log (2.1 MB) query_status.txt (297.6 KB)