dm同步失败

Bella_chen · 2020 年4 月 17 日 22:47

为提高效率，提问时请提供以下信息，问题描述清晰可优先响应。
* 【TiDB 版本】：V3.0.12

* 【问题描述】：
? query-status mysql-replica-boe
{
    "result": true,
    "msg": "",
    "workers": [
        {
            "result": true,
            "worker": "10.1.0.95:8262",
            "msg": "",
            "subTaskStatus": [
                {
                    "name": "mysql-replica-boe",
                    "stage": "Paused",
                    "unit": "Sync",
                    "result": {
                        "isCanceled": false,
                        "errors": [
                            {
                                "Type": "ExecSQL",
                                "msg": "[code=10006:class=database:scope=not-set:level=high] execute statement failed: commit: Error 9007: Write conflict, txnStartTS=416065315202924546, conflictStartTS=416065315111174146, conflictCommitTS=416065315347103754, key={tableID=27158, indexID=1, indexValues={99485, }} primary={tableID=27158, indexID=1, indexValues={97296, }} [try again later]\
github.com/pingcap/dm/pkg/terror.(*Error).Delegate\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:188\
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore.func3\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:268\
github.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:204\
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:263\
github.com/pingcap/dm/syncer.(*DBConn).executeSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:290\
github.com/pingcap/dm/syncer.(*Syncer).sync.func3\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:980\
github.com/pingcap/dm/syncer.(*Syncer).sync\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1022\
github.com/pingcap/dm/syncer.(*Syncer).Run.func2\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1093\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357",
                                "error": null
                            },
                            {
                                "Type": "ExecSQL",
                                "msg": "[code=10006:class=database:scope=not-set:level=high] execute statement failed: begin: sql: connection is already closed\
github.com/pingcap/dm/pkg/terror.(*Error).Delegate\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:146\
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore.func3\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:268\
github.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:204\
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:263\
github.com/pingcap/dm/syncer.(*DBConn).executeSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:290\
github.com/pingcap/dm/syncer.(*Syncer).sync.func3\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:980\
github.com/pingcap/dm/syncer.(*Syncer).sync\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1022\
github.com/pingcap/dm/syncer.(*Syncer).Run.func2\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1093\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357",
                                "error": null
                            },
                            {
                                "Type": "ExecSQL",
                                "msg": "[code=10006:class=database:scope=not-set:level=high] execute statement failed: begin: sql: connection is already closed\
github.com/pingcap/dm/pkg/terror.(*Error).Delegate\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:146\
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore.func3\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:268\
github.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:204\
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:263\
github.com/pingcap/dm/syncer.(*DBConn).executeSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:290\
github.com/pingcap/dm/syncer.(*Syncer).sync.func3\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:980\
github.com/pingcap/dm/syncer.(*Syncer).sync\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1022\
github.com/pingcap/dm/syncer.(*Syncer).Run.func2\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1093\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357",
                                "error": null
                            },
                            {
                                "Type": "ExecSQL",
                                "msg": "[code=10006:class=database:scope=not-set:level=high] execute statement failed: begin: driver: bad connection\
github.com/pingcap/dm/pkg/terror.(*Error).Delegate\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:146\
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore.func3\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:268\
github.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:204\
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:263\
github.com/pingcap/dm/syncer.(*DBConn).executeSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:290\
github.com/pingcap/dm/syncer.(*Syncer).sync.func3\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:980\
github.com/pingcap/dm/syncer.(*Syncer).sync\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1022\
github.com/pingcap/dm/syncer.(*Syncer).Run.func2\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1093\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357",
                                "error": null
                            },
                            {
                                "Type": "ExecSQL",
                                "msg": "[code=10006:class=database:scope=not-set:level=high] execute statement failed: begin: driver: bad connection\
github.com/pingcap/dm/pkg/terror.(*Error).Delegate\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:146\
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore.func3\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:268\
github.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:204\
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:263\
github.com/pingcap/dm/syncer.(*DBConn).executeSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:290\
github.com/pingcap/dm/syncer.(*Syncer).sync.func3\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:980\
github.com/pingcap/dm/syncer.(*Syncer).sync\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1022\
github.com/pingcap/dm/syncer.(*Syncer).Run.func2\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1093\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357",
                                "error": null
                            },
                            {
                                "Type": "ExecSQL",
                                "msg": "[code=10006:class=database:scope=not-set:level=high] execute statement failed: begin: driver: bad connection\
github.com/pingcap/dm/pkg/terror.(*Error).Delegate\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:146\
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore.func3\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:268\
github.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:204\
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:263\
github.com/pingcap/dm/syncer.(*DBConn).executeSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:290\
github.com/pingcap/dm/syncer.(*Syncer).sync.func3\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:980\
github.com/pingcap/dm/syncer.(*Syncer).sync\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1022\
github.com/pingcap/dm/syncer.(*Syncer).Run.func2\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1093\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357",
                                "error": null
                            },
                            {
                                "Type": "ExecSQL",
                                "msg": "[code=10006:class=database:scope=not-set:level=high] execute statement failed: begin: driver: bad connection\
github.com/pingcap/dm/pkg/terror.(*Error).Delegate\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:146\
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore.func3\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:268\
github.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:204\
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:263\
github.com/pingcap/dm/syncer.(*DBConn).executeSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:290\
github.com/pingcap/dm/syncer.(*Syncer).sync.func3\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:980\
github.com/pingcap/dm/syncer.(*Syncer).sync\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1022\
github.com/pingcap/dm/syncer.(*Syncer).Run.func2\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1093\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357",
                                "error": null
                            },
                            {
                                "Type": "ExecSQL",
                                "msg": "[code=10006:class=database:scope=not-set:level=high] execute statement failed: begin: driver: bad connection\
github.com/pingcap/dm/pkg/terror.(*Error).Delegate\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:146\
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore.func3\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:268\
github.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:204\
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:263\
github.com/pingcap/dm/syncer.(*DBConn).executeSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:290\
github.com/pingcap/dm/syncer.(*Syncer).sync.func3\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:980\
github.com/pingcap/dm/syncer.(*Syncer).sync\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1022\
github.com/pingcap/dm/syncer.(*Syncer).Run.func2\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1093\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357",
                                "error": null
                            },
                            {
                                "Type": "ExecSQL",
                                "msg": "[code=10006:class=database:scope=not-set:level=high] execute statement failed: begin: driver: bad connection\
github.com/pingcap/dm/pkg/terror.(*Error).Delegate\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:146\
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore.func3\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:268\
github.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:204\
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:263\
github.com/pingcap/dm/syncer.(*DBConn).executeSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:290\
github.com/pingcap/dm/syncer.(*Syncer).sync.func3\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:980\
github.com/pingcap/dm/syncer.(*Syncer).sync\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1022\
github.com/pingcap/dm/syncer.(*Syncer).Run.func2\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1093\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357",
                                "error": null
                            },
                            {
                                "Type": "ExecSQL",
                                "msg": "[code=10006:class=database:scope=not-set:level=high] execute statement failed: begin: driver: bad connection\
github.com/pingcap/dm/pkg/terror.(*Error).Delegate\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:146\
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore.func3\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:268\
github.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:204\
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:263\
github.com/pingcap/dm/syncer.(*DBConn).executeSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:290\
github.com/pingcap/dm/syncer.(*Syncer).sync.func3\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:980\
github.com/pingcap/dm/syncer.(*Syncer).sync\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1022\
github.com/pingcap/dm/syncer.(*Syncer).Run.func2\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1093\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357",
                                "error": null
                            },
                            {
                                "Type": "ExecSQL",
                                "msg": "[code=10006:class=database:scope=not-set:level=high] execute statement failed: begin: sql: connection is already closed\
github.com/pingcap/dm/pkg/terror.(*Error).Delegate\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:146\
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore.func3\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:268\
github.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:204\
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:263\
github.com/pingcap/dm/syncer.(*DBConn).executeSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:290\
github.com/pingcap/dm/syncer.(*Syncer).sync.func3\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:980\
github.com/pingcap/dm/syncer.(*Syncer).sync\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1022\
github.com/pingcap/dm/syncer.(*Syncer).Run.func2\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1093\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357",
                                "error": null
                            },
                            {
                                "Type": "ExecSQL",
                                "msg": "[code=10006:class=database:scope=not-set:level=high] execute statement failed: begin: driver: bad connection\
github.com/pingcap/dm/pkg/terror.(*Error).Delegate\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:146\
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore.func3\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:268\
github.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:204\
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:263\
github.com/pingcap/dm/syncer.(*DBConn).executeSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:290\
github.com/pingcap/dm/syncer.(*Syncer).sync.func3\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:980\
github.com/pingcap/dm/syncer.(*Syncer).sync\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1022\
github.com/pingcap/dm/syncer.(*Syncer).Run.func2\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1093\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357",
                                "error": null
                            },
                            {
                                "Type": "ExecSQL",
                                "msg": "[code=10006:class=database:scope=not-set:level=high] execute statement failed: begin: driver: bad connection\
github.com/pingcap/dm/pkg/terror.(*Error).Delegate\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:267\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ExecuteSQLWithIgnoreError\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:146\
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore.func3\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:268\
github.com/pingcap/dm/pkg/retry.(*FiniteRetryStrategy).Apply\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/retry/strategy.go:71\
github.com/pingcap/dm/pkg/conn.(*BaseConn).ApplyRetryStrategy\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/conn/baseconn.go:204\
github.com/pingcap/dm/syncer.(*DBConn).executeSQLWithIgnore\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:263\
github.com/pingcap/dm/syncer.(*DBConn).executeSQL\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:290\
github.com/pingcap/dm/syncer.(*Syncer).sync.func3\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:980\
github.com/pingcap/dm/syncer.(*Syncer).sync\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1022\
github.com/pingcap/dm/syncer.(*Syncer).Run.func2\
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1093\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357",
                                "error": null
                            }
                        ],
                        "detail": null
                    },
                    "unresolvedDDLLockID": "",
                    "sync": {
                        "totalEvents": "55167",
                        "totalTps": "0",
                        "recentTps": "0",
                        "masterBinlog": "(mysql-bin.007106, 621819364)",
                        "masterBinlogGtid": "1961de6e-db49-11e9-9eb7-ac1f6b41525a:1-2639441422,22863285-e03a-11e9-a452-ac1f6b403494:1-8454425,704aa4cd-eaab-11e9-a391-ac1f6b41525a:1-930974510",
                        "syncerBinlog": "(mysql-bin|000001.006841, 934506002)",
                        "syncerBinlogGtid": "",
                        "blockingDDLs": [
                        ],
                        "unresolvedGroups": [
                        ],
                        "synced": false
                    }
                }
            ],
            "relayStatus": {
                "masterBinlog": "(mysql-bin.007106, 621819364)",
                "masterBinlogGtid": "1961de6e-db49-11e9-9eb7-ac1f6b41525a:1-2639441422,22863285-e03a-11e9-a452-ac1f6b403494:1-8454425,704aa4cd-eaab-11e9-a391-ac1f6b41525a:1-930974510",
                "relaySubDir": "22863285-e03a-11e9-a452-ac1f6b403494.000001",
                "relayBinlog": "(mysql-bin.007106, 621819364)",
                "relayBinlogGtid": "704aa4cd-eaab-11e9-a391-ac1f6b41525a:1-907941017,1961de6e-db49-11e9-9eb7-ac1f6b41525a:2035655839-2639441422,22863285-e03a-11e9-a452-ac1f6b403494:1-8454425",
                "relayCatchUpMaster": true,
                "stage": "Running",
                "result": null
            },
            "sourceID": "mysql-replica-boe"
        }
    ]
}
若提问为**性能优化、故障排查**类问题，请下载<a href="/uploads/short-url/uGisshjxFnxx1KgpFOYbfeZjsc6" download="info_gathering.py">脚本</a>运行。终端输出的打印结果，请**务必全选**并复制粘贴上传。

yilong · 2020 年4 月 18 日 01:50

1.请问DM版本是什么？

task文件和 dm-worker日志麻烦上传下
Write conflict 是有写冲突，看下这个表是否有主键，数据冲突了？ connection is already closed ，这个问题这么频繁，在DM服务器和tidb服务器之间有防火墙吗？ tidb有使用HAproxy之类的负载均衡吗？

Bella_chen · 2020 年4 月 18 日 02:36

dm-worker.log (1.3 MB)
task_advanced.yaml (3.3 KB)

版本是1.0.4-1

3、没有弄防火墙的

Bella_chen · 2020 年4 月 18 日 03:11

dm-worker.rar (144.6 KB)

yilong · 2020 年4 月 18 日 04:10

您好：
1. 先来检查这个报错：

  2. MySQL [INFORMATION_SCHEMA]> select table_name from tables where tidb_table_id=27158， 找到这个表
  
 3. 再从TIDB_INDEXES 表里看下索引id=1的是哪个索引
 4. show create table 表名，看下这个索引是不是唯一的，当时是不是已经有这个值138670，导致导入失败

yilong · 2020 年4 月 18 日 04:36

请尝试重新resume-task看看能否恢复，多谢

Bella_chen · 2020 年4 月 18 日 05:03

MySQL [information_schema]> select table_name from tables where tidb_table_id=27158 ;

±----------------+ | table_name | ±----------------+ | if_ally_science | ±----------------+ 1 row in set (0.89 sec)

[information_schema]> show create table infinity_boe.if_ally_science\G *************************** 1. row *************************** Table: if_ally_science Create Table: CREATE TABLEif_ally_science(idbigint(20) NOT NULL AUTO_INCREMENT,datedate NOT NULL DEFAULT ‘0000-00-00’,kidint(5) NOT NULL DEFAULT ‘0’,aidint(11) NOT NULL DEFAULT ‘0’,scienceint(5) DEFAULT ‘0’, PRIMARY KEY (id), UNIQUE KEYu_k(aid), KEYidx_date(date) ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin AUTO_INCREMENT=120430

Bella_chen · 2020 年4 月 18 日 05:08

MySQL [infinity_boe]> select * from if_ally_science where id =138670\G
Empty set (0.01 sec)

MySQL [infinity_boe]>

resume-task.txt (27.5 KB)

yilong · 2020 年4 月 18 日 05:48

您好：请尝试stop-task，再start-task多谢

Bella_chen · 2020 年4 月 18 日 06:31

试过了，也不行

Jiahao · 2020 年4 月 18 日 07:06

stop-task，再start-task 后是一样出错吗

同步的表除了dm 有其它服务在写入吗

Bella_chen · 2020 年4 月 18 日 07:48

没有，只有dm，新的集群环境

yilong · 2020 年4 月 18 日 08:33

stop，start之后的报错是否一样？能否再上传一个query status的结果和dm-worker日志，多谢

Bella_chen · 2020 年4 月 19 日 02:06

TASK [import grafana dashboards - run import script] ****************************************************************************
fatal: [grafana -> localhost]: FAILED! => {"changed": true, "cmd": "python import_grafana_dashboards.py", "delta": "0:00:00.375524", "end": "2020-04-19 02:02:43.986551", "msg": "non-zero return code", "rc": 1, "start": "2020-04-19 02:02:43.611027", "stderr": "Traceback (most recent call last):\
  File \"import_grafana_dashboards.py\", line 95, in <module>\
    raise RuntimeError\
RuntimeError", "stderr_lines": ["Traceback (most recent call last):", "  File \"import_grafana_dashboards.py\", line 95, in <module>", "    raise RuntimeError", "RuntimeError"], "stdout": "[load] from <dm_instances.json>:dm_worker_instances\
[import] <Dm-Cluster-DM-worker-instances> to [Dm-Cluster]\t............. ERROR:  Unauthorized", "stdout_lines": ["[load] from <dm_instances.json>:dm_worker_instances", "[import] <Dm-Cluster-DM-worker-instances> to [Dm-Cluster]\t............. ERROR:  Unauthorized"]}
        to retry, use: --limit @/home/tidb/dm-ansible/retry_files/start.retry

帮忙看一下昨天tidb和dm 集群关闭查找问题，今天重启DM 报这个错误

yilong · 2020 年4 月 20 日 02:35

您好：

    1. 请问DM有修改配置吗？
    2. 请查看dm_instances.json，inventory.ini是否修改了哪些配置信息，格式不对，多谢

Bella_chen · 2020 年4 月 20 日 02:43

就改了grafana_admin_password ，现在改回来了，也这样

Bella_chen · 2020 年4 月 20 日 02:53

dm-worker.log (2.1 MB)
query_status.txt (297.6 KB)

来了老弟 · 2020 年4 月 20 日 04:13

你好，

根据 ERROR: Unauthorized 报错，

尝试查看该目录或者文件是否为部署管理用户权限。
是否使用部署管理用户操作
请在 inventory.ini 中配置正确 grafana 账户密码，正确性可通过 web 登录验证下

Bella_chen · 2020 年4 月 20 日 06:04

这个呢？这个是同步的问题

Bella_chen · 2020 年4 月 20 日 06:09

这个问题可以了，现在就是同步失败的问题，帮忙查一下原因，日志文件在

dm-worker.log (2.1 MB) query_status.txt (297.6 KB)