Drainer 同步异常

[关山月]

系统版本：TiDB 3.0.5
问题描述：TiDB2.1.8同步数据到3.0.5版本，drarner配置三个，凌晨过后drainer CPU 一直100%，drainer端口一直无法启动，进程还在
问题：
2019/11/13 10:41:40 server.go:296: [info] register success, this drainer’s node id is bi-tidb-pd01:8308
2019/11/13 10:43:42 server.go:347: [info] start to server request on http://172.16.96.192:8308
2019/11/13 10:43:42 merge.go:208: [info] merger add source bi-tidb-pd02:8250
2019/11/13 10:43:42 merge.go:208: [info] merger add source bi-tidb-pd03:8250
2019/11/13 10:43:42 merge.go:208: [info] merger add source bi-tidb-pd01:8250
2019/11/13 10:43:42 pump.go:116: [info] [pump bi-tidb-pd02:8250] create pull binlogs client
2019/11/13 10:43:42 pump.go:184: [info] [pump bi-tidb-pd02:8250] grpc compression enabled
2019/11/13 10:43:42 pump.go:116: [info] [pump bi-tidb-pd01:8250] create pull binlogs client
2019/11/13 10:43:42 pump.go:184: [info] [pump bi-tidb-pd01:8250] grpc compression enabled
2019/11/13 10:43:42 pump.go:116: [info] [pump bi-tidb-pd03:8250] create pull binlogs client
2019/11/13 10:43:42 pump.go:184: [info] [pump bi-tidb-pd03:8250] grpc compression enabled
2019/11/13 10:44:01 version.go:20: [info] Release Version: v2.1.8
2019/11/13 10:44:01 version.go:21: [info] Git Commit Hash: 53474a5918ae77400495695d41d6b700243b6baa
2019/11/13 10:44:01 version.go:22: [info] Build TS: 2019-04-12 07:46:15
2019/11/13 10:44:01 version.go:23: [info] Go Version: go1.12
2019/11/13 10:44:01 version.go:24: [info] Go OS/Arch: linuxamd64
2019/11/13 10:44:01 main.go:30: [info] use config: {
“log-level”: “info”,
“addr”: “http://172.16.96.192:8308”,
“data-dir”: “/data1/drainer/weimaiads_01/data.drainer”,
“detect-interval”: 10,
“pd-urls”: “http://172.16.96.192:2379,http://172.16.96.193:2379,http://172.16.96.194:2379”,
“log-file”: “/data1/drainer/weimaiads_01/log/drainer.log”,
“log-rotate”: “”,
“initial-commit-ts”: 412497689283133446,
“sycner”: {
“sql-mode”: null,
“ignore-schemas”: “INFORMATION_SCHEMA,PERFORMANCE_SCHEMA,mysql”,
“ignore-table”: null,
“txn-batch”: 1000,
“worker-count”: 16,
“to”: {
“host”: “172.16.96.197”,
“user”: “root”,
“password”: “*****”,
“port”: 4000,
“checkpoint”: {
“schema”: “”
},
“dir”: “”,
“compression”: “”,
“time-limit”: “”,
“size-limit”: “”,
“zookeeper-addrs”: “”,
“kafka-addrs”: “”,
“kafka-version”: “”,
“kafka-max-messages”: 0,
“topic-name”: “”
},
“replicate-do-table”: null,
“replicate-do-db”: [
“weimaiads”
],
“db-type”: “mysql”,
“disable-dispatch”: false,
“safe-mode”: false,
“disable-detect”: false
},
“security”: {
“ssl-ca”: “”,
“ssl-cert”: “”,
“ssl-key”: “”
},
“synced-check-time”: 5,
“compressor”: “”,
“EtcdTimeout”: 5000000000,
“MetricsAddr”: “”,
“MetricsInterval”: 15
}
2019/11/13 10:44:01 server.go:102: [info] clusterID of drainer server is 6716487223662948010
2019/11/13 10:44:01 checkpoint.go:49: [info] initialize mysql type checkpoint binlog commitTS = 412505402660618241 with config &{Db:0xc00012ea80 Schema:tidb_binlog Table:checkpoint ClusterID:6716487223662948010 InitialCommitTS:412497689283133446 CheckPointFile:/data1/drainer/weimaiads_01/data.drainer/savepoint}
2019/11/13 10:44:01 server.go:296: [info] register success, this drainer’s node id is bi-tidb-pd01:8308

[yilong (yi888long)]

您好：
1. 当前这三个drainer的进程还是100%？ 整体cpu使用率是多少？
2. 请查看下当前pump和drainer的状态，参考以下文章，反馈下信息，多谢
https://pingcap.com/docs-cn/stable/reference/tidb-binlog/maintain/

[关山月]

drainer进程 CPU100%，图片在上面；目前pump、drainer的状态都是online

[关山月]

@关山月

commitTS = 412505402660618241

感觉是卡在 commitTS = 412505402660618241 这个地方过不了了

[yilong (yi888long)]

方便反馈下当前drainer的信息吗？ 我看下drainer都在什么状态和位点

show drainer status;

show pump status;

[关山月]

[yilong (yi888long)]

好的，请稍等，已经再查看，会尽快回复，多谢

[yilong (yi888long)]

您好： 请将完整的日志上传，需要查看完整日志，另外，同步启动了3个drainer吗？

[YilinZhao-PingCAP (leoppro-PingCAP)]

麻烦执行

curl http://drainerIP:port/debug/pprof/goroutine?debug=3

curl http://drainerIP:port/debug/pprof/profile

上传输出

[关山月]

目前8306、8307、8308端口不存在，drainer后台进程在，上面的命令执行不了

[yilong (yi888long)]

您好： 1. 请发送完整日志 2. netstat -ntlp | grep 8306 netstat -ntlp | grep 8307 netstat -ntlp | grep 8308 3. curl 无法执行，报什么错，把截图发一下.

[关山月]

无法上传附件，一直报我提出问题是的内容

[YilinZhao-PingCAP (leoppro-PingCAP)]

这是drainer 刚启动时候的日志，麻烦截图结尾部分的日志

[关山月]

一直在重复这个过程，无报错

[YilinZhao-PingCAP (leoppro-PingCAP)]

应该是 panic 了 看一下标准输出

(可通过 ps 查看进程标准输出重定向的文件)

[关山月]

[关山月]

感觉是端口假死，一直起不来

[yilong (yi888long)]

您好：
请再反馈下这些信息。 stderr log中的信息。

[YilinZhao-PingCAP (leoppro-PingCAP)]

刚刚确认2.1.8版本的ansible没有重定向标准错误输出。

如果使用的 systemd 管理方式，可以尝试看下 journalctl

如果不是 systemd 管理方式 麻烦修改 ansible 启动脚本：

pingcap/tidb-ansible/blob/v2.1.8/roles/drainer/templates/run_drainer_binary.sh.j2

中最后一行

exec bin/drainer 
    --addr="{{ my_ip }}:{{ drainer_port }}" 
    --pd-urls="{{ all_pd | join(',') }}" 
    --data-dir="{{ drainer_data_dir }}" 
    --log-file="{{ drainer_log_dir }}/{{ drainer_log_filename }}" 
    --config=conf/drainer.toml 
    --initial-commit-ts="{{ initial_commit_ts }}"

改为

exec bin/drainer 
    --addr="{{ my_ip }}:{{ drainer_port }}" 
    --pd-urls="{{ all_pd | join(',') }}" 
    --data-dir="{{ drainer_data_dir }}" 
    --log-file="{{ drainer_log_dir }}/{{ drainer_log_filename }}" 
    --config=conf/drainer.toml 
    --initial-commit-ts="{{ initial_commit_ts }}" 2>> "{{ drainer_log_dir }}/drainer_stderr.log"

启动 drainer 并在 log 目录中查看 drainer_stderr.log 文件

[关山月]

好的，稍等