dm同步syncerBinlog停在固定位置

🪐 TiDB 技术问题 备份&数据迁移
1

DM 版本：
Release Version: v1.0.5
Git Commit Hash: a8e9f53f91e29756b09a22cdc37a6a6efcdfe55b
Git Branch: release-1.0
UTC Build Time: 2020-04-27 06:56:31
Go Version: go version go1.13 linux/amd64
没有报错

query-status hang-135

“masterBinlog”: “(mysql-bin.001178, 455292799)”,
这个位置是不断更新的
而，下面不更新
“syncerBinlog”: “(mysql-bin|000001.001175, 818221505)”,

使用下面命令跳过也不生效（无法跳过）：
pause-task hang-135
sql-skip --worker=10.3.5.27:8262 --binlog-pos=mysql-bin|000001.001175:818221505 hang-135
resume-task hang-135

其他：
dm-worker.log

[2020/09/24 10:33:12.836 +08:00] [WARN] [syncer.go:632] [“something wrong with rollback global checkpoint”] [task=shihang-135] [unit=“binlog replication”] [“previous position”=“(mysql-bin|000001.001175, 818221678)”] [“current position”=“(mysql-bin|000001.001175, 818221505)”]
[2020/09/24 10:38:18.840 +08:00] [WARN] [syncer.go:868] [“error detected when executing SQL job, skip flush checkpoint”] [task=shihang-135] [unit=“binlog replication”] [checkpoint=“(mysql-bin|000001.001175, 818221678)(flushed (mysql-bin|000001.001175, 818221505))”]
[2020/09/24 10:38:18.840 +08:00] [WARN] [syncer.go:868] [“error detected when executing SQL job, skip flush checkpoint”] [task=shihang-135] [unit=“binlog replication”] [checkpoint=“(mysql-bin|000001.001175, 818221678)(flushed (mysql-bin|000001.001175, 818221505))”]
[2020/09/24 10:38:18.841 +08:00] [WARN] [reader.go:137] [“parse relay finished”] [task=shihang-135] [unit=“binlog replication”] [component=“binlog reader”] [error=“context canceled”]
[2020/09/24 10:38:18.848 +08:00] [WARN] [syncer.go:632] [“something wrong with rollback global checkpoint”] [task=shihang-135] [unit=“binlog replication”] [“previous position”=“(mysql-bin|000001.001175, 818221678)”] [“current position”=“(mysql-bin|000001.001175, 818221505)”]
[2020/09/24 10:43:22.822 +08:00] [WARN] [syncer.go:868] [“error detected when executing SQL job, skip flush checkpoint”] [task=shihang-135] [unit=“binlog replication”] [checkpoint=“(mysql-bin|000001.001175, 818221678)(flushed (mysql-bin|000001.001175, 818221505))”]
[2020/09/24 10:43:22.822 +08:00] [WARN] [syncer.go:868] [“error detected when executing SQL job, skip flush checkpoint”] [task=shihang-135] [unit=“binlog replication”] [checkpoint=“(mysql-bin|000001.001175, 818221678)(flushed (mysql-bin|000001.001175, 818221505))”]
[2020/09/24 10:43:22.823 +08:00] [WARN] [reader.go:137] [“parse relay finished”] [task=shihang-135] [unit=“binlog replication”] [component=“binlog reader”] [error=“context canceled”]
[2020/09/24 10:43:22.831 +08:00] [WARN] [syncer.go:632] [“something wrong with rollback global checkpoint”] [task=shihang-135] [unit=“binlog replication”] [“previous position”=“(mysql-bin|000001.001175, 818221678)”] [“current position”=“(mysql-bin|000001.001175, 818221505)”]
[2020/09/24 10:48:27.777 +08:00] [WARN] [syncer.go:868] [“error detected when executing SQL job, skip flush checkpoint”] [task=shihang-135] [unit=“binlog replication”] [checkpoint=“(mysql-bin|000001.001175, 818221678)(flushed (mysql-bin|000001.001175, 818221505))”]
[2020/09/24 10:48:27.777 +08:00] [WARN] [syncer.go:868] [“error detected when executing SQL job, skip flush checkpoint”] [task=shihang-135] [unit=“binlog replication”] [checkpoint=“(mysql-bin|000001.001175, 818221678)(flushed (mysql-bin|000001.001175, 818221505))”]
[2020/09/24 10:48:27.778 +08:00] [WARN] [reader.go:137] [“parse relay finished”] [task=shihang-135] [unit=“binlog replication”] [component=“binlog reader”] [error=“context canceled”]
[2020/09/24 10:48:27.786 +08:00] [WARN] [syncer.go:632] [“something wrong with rollback global checkpoint”] [task=shihang-135] [unit=“binlog replication”] [“previous position”=“(mysql-bin|000001.001175, 818221678)”] [“current position”=“(mysql-bin|000001.001175, 818221505)”]
DM_META: 这一行取库名 表名：

image
image1428×60 25.9 KB

2

麻烦确认一下 mysql-bin|000001.001175, 818221505 的 binlog file 文件状态是否正常可读

3

可读：
日志提示：
something wrong with rollback global checkpoint
binlog位置处的信息：

at 818221474

#200924 4:21:56 server id 27 end_log_pos 818221505 CRC32 0xfc05d885 Xid = 2803584894
COMMIT/!/;

at 818221505

#200924 4:21:52 server id 135 end_log_pos 818221570 CRC32 0xe9100c3a GTID last_committed=1782551 sequence_number=1782552 rbr_only=no
SET @@SESSION.GTID_NEXT= ‘117f84fa-8576-11e7-89a4-f48e38bf13fc:11926651988’/!/;

at 818221570

#200924 4:21:52 server id 135 end_log_pos 818221678 CRC32 0x3b2e3609 Query thread_id=426231831 exec_time=4 error_code=0

4

麻烦确认一下下游同步的表数据是否还在增加 ？还是一直保持不变了。

5

已删除重新部署，想以增量方式同步但是提示：

ansible-playbook deploy.yml --tags=dm-worker -l dm-worker8
ansible-playbook start.yml --tags=dm-worker -l dm-worker8

start-task conf/task135.yaml
{
“result”: false,
“msg”: “[code=38028:class=dm-master:scope=internal:level=high] mysql-replica-08 relevant worker not found
github.com/pingcap/dm/pkg/terror.(*Error).Generatef
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/pkg/terror/terror.go:237
github.com/pingcap/dm/dm/master.(*Server).generateSubTask
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/dm/master/server.go:1861
github.com/pingcap/dm/dm/master.(*Server).StartTask
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/dm/master/server.go:230
github.com/pingcap/dm/dm/pb._Master_StartTask_Handler
\t/home/jenkins/agent/workspace/build_dm_master/go/src/github.com/pingcap/dm/dm/pb/dmmaster.pb.go:2355
google.golang.org/grpc.(*Server).processUnaryRPC
\t/go/pkg/mod/google.golang.org/grpc@v1.25.1/server.go:1007
google.golang.org/grpc.(*Server).handleStream
\t/go/pkg/mod/google.golang.org/grpc@v1.25.1/server.go:1287
google.golang.org/grpc.(*Server).serveStreams.func1.1
\t/go/pkg/mod/google.golang.org/grpc@v1.25.1/server.go:722
runtime.goexit
\t/usr/local/go/src/runtime/asm_amd64.s:1357”,
“workers”: [
]
}

task：
name: shang-135 # global unique
task-mode: incremental # full/incremental/all
enable-heartbeat: true # 是否开启 heartbeat 功能
online-ddl-scheme: “pt”
remove-meta: false
case-sensitive : true

target-database:
host: “10.3.6.24”
port: 4000
user: “root”
password: “NSSUAkBLZzTic=”

mysql-instances: # one or more source database, config more source database for sharding merge

source-id: "mysql-replica-08" # unique in all instances, used as id when save checkpoints, configs, etc.
meta:
  binlog-name: mysql-bin.001175
  binlog-pos: 818221570
  #binlog-gtid: "117f84fa-8576-11e7-89a4-f48e38bf13fc:1-11926651988,5d597c0e-ac74-11ea-8b6a-f8f21e99eeb1:1-83444493"

black-white-list:  "instance"
filter-rules: ["filter-rule-1"]

。。。。。。。。。。。。。。。

dm-worker8 ansible_host=10.3.5.x source_id=“mysql-replica-08” server_id=227 relay_binlog_name=“mysql-bin.001175” mysql_host=10.3.5.2x mysql_user=root mysql_password=‘VYgduPJaRw==’ mysql_port=3306

重新部署在源来的基础上修改：
dm-worker8 mysql-replica-08 的名称

启动出现上面的提示。

6

麻烦确认一下这个对应的上游的源配置是否可以正常访问，日志看应该是这个上游源不可访问。