用tiup启动刚安装的ticdc启动不起来,一堆报错?

其中一台ticdc日志日下

[2022/01/14 17:59:10.674 +08:00] [INFO] [helper.go:51] ["init log"] [file=/data/cdc/8300/log/cdc.log] [level=info]
[2022/01/14 17:59:10.675 +08:00] [INFO] [version.go:47] ["Welcome to Change Data Capture (CDC)"] [release-version=v5.3.0] [git-hash=20626babf21fc381d4364646c40dd84598533d66] [git-branch=heads/refs/tags/v5.3.0] [utc-build-time="2021-11-22 10:37:02"] [go-version="go version go1.16.4 linux/amd64"] [failpoint-build=false]
[2022/01/14 17:59:10.675 +08:00] [INFO] [server.go:67] ["creating CDC server"] [pd-addrs="[http://10.59.111.225:2379,http://10.59.111.226:2379,http://10.59.111.227:2379]"] [config="{\"addr\":\"0.0.0.0:8300\",\"advertise-addr\":\"10.59.110.17:8300\",\"log-file\":\"/data/cdc/8300/log/cdc.log\",\"log-level\":\"info\",\"log\":{\"file\":{\"max-size\":300,\"max-days\":0,\"max-backups\":0}},\"data-dir\":\"/data/cdc/8300/store\",\"gc-ttl\":86400,\"tz\":\"System\",\"capture-session-ttl\":10,\"owner-flush-interval\":200000000,\"processor-flush-interval\":100000000,\"sorter\":{\"num-concurrent-worker\":4,\"chunk-size-limit\":134217728,\"max-memory-percentage\":30,\"max-memory-consumption\":17179869184,\"num-workerpool-goroutine\":16,\"sort-dir\":\"/tmp/sorter\"},\"security\":{\"ca-path\":\"\",\"cert-path\":\"\",\"key-path\":\"\",\"cert-allowed-cn\":null},\"per-table-memory-quota\":10485760,\"kv-client\":{\"worker-concurrent\":8,\"worker-pool-size\":0,\"region-scan-limit\":40}}"]
[2022/01/14 17:59:10.675 +08:00] [INFO] [client.go:355] ["[pd] create pd client with endpoints"] [pd-address="[http://10.59.111.225:2379,http://10.59.111.226:2379,http://10.59.111.227:2379]"]
[2022/01/14 17:59:10.680 +08:00] [INFO] [base_client.go:349] ["[pd] switch leader"] [new-leader=http://10.59.111.227:2379] [old-leader=]
[2022/01/14 17:59:10.680 +08:00] [INFO] [base_client.go:104] ["[pd] init cluster id"] [cluster-id=6825873712670396671]
[2022/01/14 17:59:10.681 +08:00] [INFO] [client.go:651] ["[pd] tso dispatcher created"] [dc-location=global]
[2022/01/14 17:59:10.682 +08:00] [INFO] [server.go:274] ["/data/cdc/8300/store is set as data-dir (99GB available), sort-dir=/data/cdc/8300/store/tmp/sorter. It is recommended that the disk for data-dir at least have 500GB available space"]
[2022/01/14 17:59:10.686 +08:00] [INFO] [store.go:74] ["new store"] [path="tikv://10.59.111.225:2379,10.59.111.226:2379,10.59.111.227:2379?disableGC=true"]
[2022/01/14 17:59:10.686 +08:00] [INFO] [client.go:355] ["[pd] create pd client with endpoints"] [pd-address="[10.59.111.225:2379,10.59.111.226:2379,10.59.111.227:2379]"]
[2022/01/14 17:59:10.689 +08:00] [INFO] [base_client.go:349] ["[pd] switch leader"] [new-leader=http://10.59.111.227:2379] [old-leader=]
[2022/01/14 17:59:10.689 +08:00] [INFO] [base_client.go:104] ["[pd] init cluster id"] [cluster-id=6825873712670396671]
[2022/01/14 17:59:10.689 +08:00] [INFO] [client.go:651] ["[pd] tso dispatcher created"] [dc-location=global]
[2022/01/14 17:59:10.693 +08:00] [INFO] [store.go:80] ["new store with retry success"]
[2022/01/14 17:59:10.694 +08:00] [INFO] [http_status.go:80] ["http server is running"] [addr=0.0.0.0:8300]
[2022/01/14 17:59:10.697 +08:00] [INFO] [capture.go:107] ["init capture"] [capture-id=e4cab8e5-f144-4a49-a4fa-26b9f899e92e] [capture-addr=10.59.110.17:8300]
[2022/01/14 17:59:10.710 +08:00] [INFO] [reactor_state.go:74] ["remote capture online"] [capture-id=e4cab8e5-f144-4a49-a4fa-26b9f899e92e] [info="{\"id\":\"e4cab8e5-f144-4a49-a4fa-26b9f899e92e\",\"address\":\"10.59.110.17:8300\",\"version\":\"v5.3.0\"}"]
[2022/01/14 17:59:10.713 +08:00] [INFO] [capture.go:189] ["the processor routine has exited"] [error="json: cannot unmarshal number into Go struct field TaskStatus.tables of type model.TableReplicaInfo"] [errorVerbose="json: cannot unmarshal number into Go struct field TaskStatus.tables of type model.TableReplicaInfo\
github.com/pingcap/errors.AddStack\
\tgithub.com/pingcap/errors@v0.11.5-0.20210513014640-40f9a1999b3b/errors.go:174\
github.com/pingcap/errors.Trace\
\tgithub.com/pingcap/errors@v0.11.5-0.20210513014640-40f9a1999b3b/juju_adaptor.go:15\
github.com/pingcap/ticdc/pkg/orchestrator.(*ChangefeedReactorState).UpdateCDCKey\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/reactor_state.go:209\
github.com/pingcap/ticdc/pkg/orchestrator.(*GlobalReactorState).Update\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/reactor_state.go:89\
github.com/pingcap/ticdc/pkg/orchestrator.(*EtcdWorker).syncRawState\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/etcd_worker.go:266\
github.com/pingcap/ticdc/pkg/orchestrator.(*EtcdWorker).Run\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/etcd_worker.go:100\
github.com/pingcap/ticdc/cdc/capture.(*Capture).runEtcdWorker\
\tgithub.com/pingcap/ticdc/cdc/capture/capture.go:273\
github.com/pingcap/ticdc/cdc/capture.(*Capture).run.func3\
\tgithub.com/pingcap/ticdc/cdc/capture/capture.go:188\
runtime.goexit\
\truntime/asm_amd64.s:1371"]
[2022/01/14 17:59:10.716 +08:00] [INFO] [capture.go:178] ["the owner routine has exited"] []
[2022/01/14 17:59:10.719 +08:00] [INFO] [capture.go:145] ["the capture routine has exited"]
[2022/01/14 17:59:10.719 +08:00] [ERROR] [client.go:750] ["[pd] fetch pending tso requests error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]context canceled: context canceled"]
[2022/01/14 17:59:10.719 +08:00] [INFO] [client.go:669] ["[pd] exit tso dispatcher"] [dc-location=global]
[2022/01/14 17:59:10.719 +08:00] [ERROR] [server.go:130] ["run server"] [error="json: cannot unmarshal number into Go struct field TaskStatus.tables of type model.TableReplicaInfo\
github.com/pingcap/errors.AddStack\
\tgithub.com/pingcap/errors@v0.11.5-0.20210513014640-40f9a1999b3b/errors.go:174\
github.com/pingcap/errors.Trace\
\tgithub.com/pingcap/errors@v0.11.5-0.20210513014640-40f9a1999b3b/juju_adaptor.go:15\
github.com/pingcap/ticdc/pkg/orchestrator.(*ChangefeedReactorState).UpdateCDCKey\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/reactor_state.go:209\
github.com/pingcap/ticdc/pkg/orchestrator.(*GlobalReactorState).Update\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/reactor_state.go:89\
github.com/pingcap/ticdc/pkg/orchestrator.(*EtcdWorker).syncRawState\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/etcd_worker.go:266\
github.com/pingcap/ticdc/pkg/orchestrator.(*EtcdWorker).Run\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/etcd_worker.go:100\
github.com/pingcap/ticdc/cdc/capture.(*Capture).runEtcdWorker\
\tgithub.com/pingcap/ticdc/cdc/capture/capture.go:273\
github.com/pingcap/ticdc/cdc/capture.(*Capture).run.func3\
\tgithub.com/pingcap/ticdc/cdc/capture/capture.go:188\
runtime.goexit\
\truntime/asm_amd64.s:1371\
processor exited with error"]
[2022/01/14 17:59:10.719 +08:00] [ERROR] [client.go:750] ["[pd] fetch pending tso requests error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]context canceled: context canceled"]
[2022/01/14 17:59:10.719 +08:00] [INFO] [client.go:669] ["[pd] exit tso dispatcher"] [dc-location=global

看起来是跟PD的网络是不通的,可以考虑是不是网络问题

网络都是通的,telnet也没问题

这个是另外一个ticdc日志:

[2022/01/14 18:12:56.917 +08:00] [INFO] [helper.go:51] ["init log"] [file=/data/cdc/8300/log/cdc.log] [level=info]
[2022/01/14 18:12:56.917 +08:00] [INFO] [version.go:47] ["Welcome to Change Data Capture (CDC)"] [release-version=v5.3.0] [git-hash=20626babf21fc381d4364646c40dd84598533d66] [git-branch=heads/refs/tags/v5.3.0] [utc-build-time="2021-11-22 10:37:02"] [go-version="go version go1.16.4 linux/amd64"] [failpoint-build=false]
[2022/01/14 18:12:56.917 +08:00] [INFO] [server.go:67] ["creating CDC server"] [pd-addrs="[http://10.59.111.225:2379,http://10.59.111.226:2379,http://10.59.111.227:2379]"] [config="{\"addr\":\"0.0.0.0:8300\",\"advertise-addr\":\"10.59.110.17:8300\",\"log-file\":\"/data/cdc/8300/log/cdc.log\",\"log-level\":\"info\",\"log\":{\"file\":{\"max-size\":300,\"max-days\":0,\"max-backups\":0}},\"data-dir\":\"/data/cdc/8300/store\",\"gc-ttl\":86400,\"tz\":\"System\",\"capture-session-ttl\":10,\"owner-flush-interval\":200000000,\"processor-flush-interval\":100000000,\"sorter\":{\"num-concurrent-worker\":4,\"chunk-size-limit\":134217728,\"max-memory-percentage\":30,\"max-memory-consumption\":17179869184,\"num-workerpool-goroutine\":16,\"sort-dir\":\"/tmp/sorter\"},\"security\":{\"ca-path\":\"\",\"cert-path\":\"\",\"key-path\":\"\",\"cert-allowed-cn\":null},\"per-table-memory-quota\":10485760,\"kv-client\":{\"worker-concurrent\":8,\"worker-pool-size\":0,\"region-scan-limit\":40}}"]
[2022/01/14 18:12:56.917 +08:00] [INFO] [client.go:355] ["[pd] create pd client with endpoints"] [pd-address="[http://10.59.111.225:2379,http://10.59.111.226:2379,http://10.59.111.227:2379]"]
[2022/01/14 18:12:56.923 +08:00] [INFO] [base_client.go:349] ["[pd] switch leader"] [new-leader=http://10.59.111.227:2379] [old-leader=]
[2022/01/14 18:12:56.923 +08:00] [INFO] [base_client.go:104] ["[pd] init cluster id"] [cluster-id=6825873712670396671]
[2022/01/14 18:12:56.923 +08:00] [INFO] [client.go:651] ["[pd] tso dispatcher created"] [dc-location=global]
[2022/01/14 18:12:56.925 +08:00] [INFO] [server.go:274] ["/data/cdc/8300/store is set as data-dir (99GB available), sort-dir=/data/cdc/8300/store/tmp/sorter. It is recommended that the disk for data-dir at least have 500GB available space"]
[2022/01/14 18:12:56.929 +08:00] [INFO] [store.go:74] ["new store"] [path="tikv://10.59.111.225:2379,10.59.111.226:2379,10.59.111.227:2379?disableGC=true"]
[2022/01/14 18:12:56.929 +08:00] [INFO] [client.go:355] ["[pd] create pd client with endpoints"] [pd-address="[10.59.111.225:2379,10.59.111.226:2379,10.59.111.227:2379]"]
[2022/01/14 18:12:56.932 +08:00] [INFO] [base_client.go:349] ["[pd] switch leader"] [new-leader=http://10.59.111.227:2379] [old-leader=]
[2022/01/14 18:12:56.932 +08:00] [INFO] [base_client.go:104] ["[pd] init cluster id"] [cluster-id=6825873712670396671]
[2022/01/14 18:12:56.932 +08:00] [INFO] [client.go:651] ["[pd] tso dispatcher created"] [dc-location=global]
[2022/01/14 18:12:56.934 +08:00] [INFO] [store.go:80] ["new store with retry success"]
[2022/01/14 18:12:56.935 +08:00] [INFO] [http_status.go:80] ["http server is running"] [addr=0.0.0.0:8300]
[2022/01/14 18:12:56.938 +08:00] [INFO] [capture.go:107] ["init capture"] [capture-id=bdf6dd00-37c3-4a72-9b56-4c85f1a2345f] [capture-addr=10.59.110.17:8300]
[2022/01/14 18:12:56.946 +08:00] [INFO] [capture.go:249] ["campaign owner successfully"] [capture-id=bdf6dd00-37c3-4a72-9b56-4c85f1a2345f]
[2022/01/14 18:12:56.946 +08:00] [INFO] [reactor_state.go:74] ["remote capture online"] [capture-id=bdf6dd00-37c3-4a72-9b56-4c85f1a2345f] [info="{\"id\":\"bdf6dd00-37c3-4a72-9b56-4c85f1a2345f\",\"address\":\"10.59.110.17:8300\",\"version\":\"v5.3.0\"}"]
[2022/01/14 18:12:56.949 +08:00] [INFO] [capture.go:189] ["the processor routine has exited"] [error="json: cannot unmarshal number into Go struct field TaskStatus.tables of type model.TableReplicaInfo"] [errorVerbose="json: cannot unmarshal number into Go struct field TaskStatus.tables of type model.TableReplicaInfo\
github.com/pingcap/errors.AddStack\
\tgithub.com/pingcap/errors@v0.11.5-0.20210513014640-40f9a1999b3b/errors.go:174\
github.com/pingcap/errors.Trace\
\tgithub.com/pingcap/errors@v0.11.5-0.20210513014640-40f9a1999b3b/juju_adaptor.go:15\
github.com/pingcap/ticdc/pkg/orchestrator.(*ChangefeedReactorState).UpdateCDCKey\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/reactor_state.go:209\
github.com/pingcap/ticdc/pkg/orchestrator.(*GlobalReactorState).Update\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/reactor_state.go:89\
github.com/pingcap/ticdc/pkg/orchestrator.(*EtcdWorker).syncRawState\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/etcd_worker.go:266\
github.com/pingcap/ticdc/pkg/orchestrator.(*EtcdWorker).Run\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/etcd_worker.go:100\
github.com/pingcap/ticdc/cdc/capture.(*Capture).runEtcdWorker\
\tgithub.com/pingcap/ticdc/cdc/capture/capture.go:273\
github.com/pingcap/ticdc/cdc/capture.(*Capture).run.func3\
\tgithub.com/pingcap/ticdc/cdc/capture/capture.go:188\
runtime.goexit\
\truntime/asm_amd64.s:1371"]
[2022/01/14 18:12:56.949 +08:00] [INFO] [capture.go:254] ["run owner exited"] [error="[CDC:ErrPDEtcdAPIError]context canceled: context canceled"] [errorVerbose="[CDC:ErrPDEtcdAPIError]context canceled: context canceled\
github.com/pingcap/errors.AddStack\
\tgithub.com/pingcap/errors@v0.11.5-0.20210513014640-40f9a1999b3b/errors.go:174\
github.com/pingcap/errors.(*Error).GenWithStackByCause\
\tgithub.com/pingcap/errors@v0.11.5-0.20210513014640-40f9a1999b3b/normalize.go:302\
github.com/pingcap/ticdc/pkg/errors.WrapError\
\tgithub.com/pingcap/ticdc/pkg/errors/helper.go:30\
github.com/pingcap/ticdc/cdc/capture.(*Capture).runEtcdWorker\
\tgithub.com/pingcap/ticdc/cdc/capture/capture.go:287\
github.com/pingcap/ticdc/cdc/capture.(*Capture).campaignOwner\
\tgithub.com/pingcap/ticdc/cdc/capture/capture.go:252\
github.com/pingcap/ticdc/cdc/capture.(*Capture).run.func2\
\tgithub.com/pingcap/ticdc/cdc/capture/capture.go:177\
runtime.goexit\
\truntime/asm_amd64.s:1371"]
[2022/01/14 18:12:56.949 +08:00] [INFO] [capture.go:178] ["the owner routine has exited"] [error="resign owner failed, capture: bdf6dd00-37c3-4a72-9b56-4c85f1a2345f: [CDC:ErrCaptureResignOwner]context canceled: context canceled"] [errorVerbose="[CDC:ErrCaptureResignOwner]context canceled: context canceled\
github.com/pingcap/errors.AddStack\
\tgithub.com/pingcap/errors@v0.11.5-0.20210513014640-40f9a1999b3b/errors.go:174\
github.com/pingcap/errors.(*Error).GenWithStackByCause\
\tgithub.com/pingcap/errors@v0.11.5-0.20210513014640-40f9a1999b3b/normalize.go:302\
github.com/pingcap/ticdc/pkg/errors.WrapError\
\tgithub.com/pingcap/ticdc/pkg/errors/helper.go:30\
github.com/pingcap/ticdc/cdc/capture.(*Capture).resign\
\tgithub.com/pingcap/ticdc/cdc/capture/capture.go:327\
github.com/pingcap/ticdc/cdc/capture.(*Capture).campaignOwner\
\tgithub.com/pingcap/ticdc/cdc/capture/capture.go:256\
github.com/pingcap/ticdc/cdc/capture.(*Capture).run.func2\
\tgithub.com/pingcap/ticdc/cdc/capture/capture.go:177\
runtime.goexit\
\truntime/asm_amd64.s:1371\
resign owner failed, capture: bdf6dd00-37c3-4a72-9b56-4c85f1a2345f"]
[2022/01/14 18:12:56.954 +08:00] [INFO] [capture.go:142] ["capture recovered"] [capture-id=bdf6dd00-37c3-4a72-9b56-4c85f1a2345f]
[2022/01/14 18:12:56.965 +08:00] [INFO] [capture.go:107] ["init capture"] [capture-id=a578b4ac-2bbc-49e5-85b5-d7571498934e] [capture-addr=10.59.110.17:8300]
[2022/01/14 18:12:56.974 +08:00] [INFO] [reactor_state.go:74] ["remote capture online"] [capture-id=a578b4ac-2bbc-49e5-85b5-d7571498934e] [info="{\"id\":\"a578b4ac-2bbc-49e5-85b5-d7571498934e\",\"address\":\"10.59.110.17:8300\",\"version\":\"v5.3.0\"}"]
[2022/01/14 18:12:56.975 +08:00] [INFO] [capture.go:249] ["campaign owner successfully"] [capture-id=a578b4ac-2bbc-49e5-85b5-d7571498934e]
[2022/01/14 18:12:56.976 +08:00] [INFO] [capture.go:189] ["the processor routine has exited"] [error="json: cannot unmarshal number into Go struct field TaskStatus.tables of type model.TableReplicaInfo"] [errorVerbose="json: cannot unmarshal number into Go struct field TaskStatus.tables of type model.TableReplicaInfo\
github.com/pingcap/errors.AddStack\
\tgithub.com/pingcap/errors@v0.11.5-0.20210513014640-40f9a1999b3b/errors.go:174\
github.com/pingcap/errors.Trace\
\tgithub.com/pingcap/errors@v0.11.5-0.20210513014640-40f9a1999b3b/juju_adaptor.go:15\
github.com/pingcap/ticdc/pkg/orchestrator.(*ChangefeedReactorState).UpdateCDCKey\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/reactor_state.go:209\
github.com/pingcap/ticdc/pkg/orchestrator.(*GlobalReactorState).Update\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/reactor_state.go:89\
github.com/pingcap/ticdc/pkg/orchestrator.(*EtcdWorker).syncRawState\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/etcd_worker.go:266\
github.com/pingcap/ticdc/pkg/orchestrator.(*EtcdWorker).Run\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/etcd_worker.go:100\
github.com/pingcap/ticdc/cdc/capture.(*Capture).runEtcdWorker\
\tgithub.com/pingcap/ticdc/cdc/capture/capture.go:273\
github.com/pingcap/ticdc/cdc/capture.(*Capture).run.func3\
\tgithub.com/pingcap/ticdc/cdc/capture/capture.go:188\
runtime.goexit\
\truntime/asm_amd64.s:1371"]
[2022/01/14 18:12:56.977 +08:00] [INFO] [capture.go:254] ["run owner exited"] [error="[CDC:ErrPDEtcdAPIError]context canceled: context canceled"] [errorVerbose="[CDC:ErrPDEtcdAPIError]context canceled: context canceled\
github.com/pingcap/errors.AddStack\
\tgithub.com/pingcap/errors@v0.11.5-0.20210513014640-40f9a1999b3b/errors.go:174\
github.com/pingcap/errors.(*Error).GenWithStackByCause\
\tgithub.com/pingcap/errors@v0.11.5-0.20210513014640-40f9a1999b3b/normalize.go:302\
github.com/pingcap/ticdc/pkg/errors.WrapError\
\tgithub.com/pingcap/ticdc/pkg/errors/helper.go:30\
github.com/pingcap/ticdc/cdc/capture.(*Capture).runEtcdWorker\
\tgithub.com/pingcap/ticdc/cdc/capture/capture.go:287\
github.com/pingcap/ticdc/cdc/capture.(*Capture).campaignOwner\
\tgithub.com/pingcap/ticdc/cdc/capture/capture.go:252\
github.com/pingcap/ticdc/cdc/capture.(*Capture).run.func2\
\tgithub.com/pingcap/ticdc/cdc/capture/capture.go:177\
runtime.goexit\
\truntime/asm_amd64.s:1371"]
[2022/01/14 18:12:56.977 +08:00] [INFO] [capture.go:178] ["the owner routine has exited"] [error="resign owner failed, capture: a578b4ac-2bbc-49e5-85b5-d7571498934e: [CDC:ErrCaptureResignOwner]context canceled: context canceled"] [errorVerbose="[CDC:ErrCaptureResignOwner]context canceled: context canceled\
github.com/pingcap/errors.AddStack\
\tgithub.com/pingcap/errors@v0.11.5-0.20210513014640-40f9a1999b3b/errors.go:174\
github.com/pingcap/errors.(*Error).GenWithStackByCause\
\tgithub.com/pingcap/errors@v0.11.5-0.20210513014640-40f9a1999b3b/normalize.go:302\
github.com/pingcap/ticdc/pkg/errors.WrapError\
\tgithub.com/pingcap/ticdc/pkg/errors/helper.go:30\
github.com/pingcap/ticdc/cdc/capture.(*Capture).resign\
\tgithub.com/pingcap/ticdc/cdc/capture/capture.go:327\
github.com/pingcap/ticdc/cdc/capture.(*Capture).campaignOwner\
\tgithub.com/pingcap/ticdc/cdc/capture/capture.go:256\
github.com/pingcap/ticdc/cdc/capture.(*Capture).run.func2\
\tgithub.com/pingcap/ticdc/cdc/capture/capture.go:177\
runtime.goexit\
\truntime/asm_amd64.s:1371\
resign owner failed, capture: a578b4ac-2bbc-49e5-85b5-d7571498934e"]
[2022/01/14 18:12:56.984 +08:00] [INFO] [capture.go:142] ["capture recovered"] [capture-id=a578b4ac-2bbc-49e5-85b5-d7571498934e]
[2022/01/14 18:13:16.941 +08:00] [INFO] [capture.go:107] ["init capture"] [capture-id=1d2b4cfe-a039-418c-9d01-5b6984b7b71c] [capture-addr=10.59.110.17:8300]
[2022/01/14 18:13:16.949 +08:00] [INFO] [reactor_state.go:74] ["remote capture online"] [capture-id=1d2b4cfe-a039-418c-9d01-5b6984b7b71c] [info="{\"id\":\"1d2b4cfe-a039-418c-9d01-5b6984b7b71c\",\"address\":\"10.59.110.17:8300\",\"version\":\"v5.3.0\"}"]
[2022/01/14 18:13:16.949 +08:00] [INFO] [capture.go:249] ["campaign owner successfully"] [capture-id=1d2b4cfe-a039-418c-9d01-5b6984b7b71c]
[2022/01/14 18:13:16.952 +08:00] [INFO] [capture.go:189] ["the processor routine has exited"] [error="json: cannot unmarshal number into Go struct field TaskStatus.tables of type model.TableReplicaInfo"] [errorVerbose="json: cannot unmarshal number into Go struct field TaskStatus.tables of type model.TableReplicaInfo\
github.com/pingcap/errors.AddStack\
\tgithub.com/pingcap/errors@v0.11.5-0.20210513014640-40f9a1999b3b/errors.go:174\
github.com/pingcap/errors.Trace\
\tgithub.com/pingcap/errors@v0.11.5-0.20210513014640-40f9a1999b3b/juju_adaptor.go:15\
github.com/pingcap/ticdc/pkg/orchestrator.(*ChangefeedReactorState).UpdateCDCKey\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/reactor_state.go:209\
github.com/pingcap/ticdc/pkg/orchestrator.(*GlobalReactorState).Update\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/reactor_state.go:89\
github.com/pingcap/ticdc/pkg/orchestrator.(*EtcdWorker).syncRawState\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/etcd_worker.go:266\
github.com/pingcap/ticdc/pkg/orchestrator.(*EtcdWorker).Run\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/etcd_worker.go:100\
github.com/pingcap/ticdc/cdc/capture.(*Capture).runEtcdWorker\
\tgithub.com/pingcap/ticdc/cdc/capture/capture.go:273\
github.com/pingcap/ticdc/cdc/capture.(*Capture).run.func3\
\tgithub.com/pingcap/ticdc/cdc/capture/capture.go:188\
runtime.goexit\
\truntime/asm_amd64.s:1371"]
[2022/01/14 18:13:16.952 +08:00] [INFO] [capture.go:254] ["run owner exited"] [error="[CDC:ErrPDEtcdAPIError]context canceled: context canceled"] [errorVerbose="[CDC:ErrPDEtcdAPIError]context canceled: context canceled\
github.com/pingcap/errors.AddStack\
\tgithub.com/pingcap/errors@v0.11.5-0.20210513014640-40f9a1999b3b/errors.go:174\
github.com/pingcap/errors.(*Error).GenWithStackByCause\
\tgithub.com/pingcap/errors@v0.11.5-0.20210513014640-40f9a1999b3b/normalize.go:302\
github.com/pingcap/ticdc/pkg/errors.WrapError\
\tgithub.com/pingcap/ticdc/pkg/errors/helper.go:30\
github.com/pingcap/ticdc/cdc/capture.(*Capture).runEtcdWorker\
\tgithub.com/pingcap/ticdc/cdc/capture/capture.go:287\
github.com/pingcap/ticdc/cdc/capture.(*Capture).campaignOwner\
\tgithub.com/pingcap/ticdc/cdc/capture/capture.go:252\
github.com/pingcap/ticdc/cdc/capture.(*Capture).run.func2\
\tgithub.com/pingcap/ticdc/cdc/capture/capture.go:177\
runtime.goexit\
\truntime/asm_amd64.s:1371"]
[2022/01/14 18:13:16.952 +08:00] [INFO] [capture.go:178] ["the owner routine has exited"] [error="resign owner failed, capture: 1d2b4cfe-a039-418c-9d01-5b6984b7b71c: [CDC:ErrCaptureResignOwner]context canceled: context canceled"] [errorVerbose="[CDC:ErrCaptureResignOwner]context canceled: context canceled\
github.com/pingcap/errors.AddStack\
\tgithub.com/pingcap/errors@v0.11.5-0.20210513014640-40f9a1999b3b/errors.go:174\
github.com/pingcap/errors.(*Error).GenWithStackByCause\
\tgithub.com/pingcap/errors@v0.11.5-0.20210513014640-40f9a1999b3b/normalize.go:302\
github.com/pingcap/ticdc/pkg/errors.WrapError\
\tgithub.com/pingcap/ticdc/pkg/errors/helper.go:30\
github.com/pingcap/ticdc/cdc/capture.(*Capture).resign\
\tgithub.com/pingcap/ticdc/cdc/capture/capture.go:327\
github.com/pingcap/ticdc/cdc/capture.(*Capture).campaignOwner\
\tgithub.com/pingcap/ticdc/cdc/capture/capture.go:256\
github.com/pingcap/ticdc/cdc/capture.(*Capture).run.func2\
\tgithub.com/pingcap/ticdc/cdc/capture/capture.go:177\
runtime.goexit\
\truntime/asm_amd64.s:1371\
resign owner failed, capture: 1d2b4cfe-a039-418c-9d01-5b6984b7b71c"]
[2022/01/14 18:13:16.956 +08:00] [INFO] [capture.go:142] ["capture recovered"] [capture-id=1d2b4cfe-a039-418c-9d01-5b6984b7b71c]
[2022/01/14 18:13:36.943 +08:00] [INFO] [capture.go:107] ["init capture"] [capture-id=d3ffa653-7ede-494b-984b-2b98f0da8bce] [capture-addr=10.59.110.17:8300]
[2022/01/14 18:13:36.951 +08:00] [INFO] [reactor_state.go:74] ["remote capture online"] [capture-id=d3ffa653-7ede-494b-984b-2b98f0da8bce] [info="{\"id\":\"d3ffa653-7ede-494b-984b-2b98f0da8bce\",\"address\":\"10.59.110.17:8300\",\"version\":\"v5.3.0\"}"]
[2022/01/14 18:13:36.953 +08:00] [INFO] [capture.go:189] ["the processor routine has exited"] [error="json: cannot unmarshal number into Go struct field TaskStatus.tables of type model.TableReplicaInfo"] [errorVerbose="json: cannot unmarshal number into Go struct field TaskStatus.tables of type model.TableReplicaInfo\
github.com/pingcap/errors.AddStack\
\tgithub.com/pingcap/errors@v0.11.5-0.20210513014640-40f9a1999b3b/errors.go:174\
github.com/pingcap/errors.Trace\
\tgithub.com/pingcap/errors@v0.11.5-0.20210513014640-40f9a1999b3b/juju_adaptor.go:15\
github.com/pingcap/ticdc/pkg/orchestrator.(*ChangefeedReactorState).UpdateCDCKey\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/reactor_state.go:209\
github.com/pingcap/ticdc/pkg/orchestrator.(*GlobalReactorState).Update\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/reactor_state.go:89\
github.com/pingcap/ticdc/pkg/orchestrator.(*EtcdWorker).syncRawState\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/etcd_worker.go:266\
github.com/pingcap/ticdc/pkg/orchestrator.(*EtcdWorker).Run\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/etcd_worker.go:100\
github.com/pingcap/ticdc/cdc/capture.(*Capture).runEtcdWorker\
\tgithub.com/pingcap/ticdc/cdc/capture/capture.go:273\
github.com/pingcap/ticdc/cdc/capture.(*Capture).run.func3\
\tgithub.com/pingcap/ticdc/cdc/capture/capture.go:188\
runtime.goexit\
\truntime/asm_amd64.s:1371"]
[2022/01/14 18:13:36.957 +08:00] [INFO] [capture.go:178] ["the owner routine has exited"] []
[2022/01/14 18:13:36.959 +08:00] [INFO] [capture.go:145] ["the capture routine has exited"]
[2022/01/14 18:13:36.960 +08:00] [ERROR] [client.go:750] ["[pd] fetch pending tso requests error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]context canceled: context canceled"]
[2022/01/14 18:13:36.960 +08:00] [INFO] [client.go:669] ["[pd] exit tso dispatcher"] [dc-location=global]
[2022/01/14 18:13:36.960 +08:00] [ERROR] [server.go:130] ["run server"] [error="json: cannot unmarshal number into Go struct field TaskStatus.tables of type model.TableReplicaInfo\
github.com/pingcap/errors.AddStack\
\tgithub.com/pingcap/errors@v0.11.5-0.20210513014640-40f9a1999b3b/errors.go:174\
github.com/pingcap/errors.Trace\
\tgithub.com/pingcap/errors@v0.11.5-0.20210513014640-40f9a1999b3b/juju_adaptor.go:15\
github.com/pingcap/ticdc/pkg/orchestrator.(*ChangefeedReactorState).UpdateCDCKey\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/reactor_state.go:209\
github.com/pingcap/ticdc/pkg/orchestrator.(*GlobalReactorState).Update\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/reactor_state.go:89\
github.com/pingcap/ticdc/pkg/orchestrator.(*EtcdWorker).syncRawState\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/etcd_worker.go:266\
github.com/pingcap/ticdc/pkg/orchestrator.(*EtcdWorker).Run\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/etcd_worker.go:100\
github.com/pingcap/ticdc/cdc/capture.(*Capture).runEtcdWorker\
\tgithub.com/pingcap/ticdc/cdc/capture/capture.go:273\
github.com/pingcap/ticdc/cdc/capture.(*Capture).run.func3\
\tgithub.com/pingcap/ticdc/cdc/capture/capture.go:188\
runtime.goexit\
\truntime/asm_amd64.s:1371\
processor exited with error"]
[2022/01/14 18:13:36.960 +08:00] [ERROR] [client.go:750] ["[pd] fetch pending tso requests error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]context canceled: context canceled"]
[2022/01/14 18:13:36.960 +08:00] [INFO] [client.go:669] ["[pd] exit tso dispatcher"] [dc-location=global]

这个是第三个ticdc日志:

[2022/01/14 18:14:33.011 +08:00] [INFO] [helper.go:51] ["init log"] [file=/data/cdc/8300/log/cdc.log] [level=info]
[2022/01/14 18:14:33.011 +08:00] [INFO] [version.go:47] ["Welcome to Change Data Capture (CDC)"] [release-version=v5.3.0] [git-hash=20626babf21fc381d4364646c40dd84598533d66] [git-branch=heads/refs/tags/v5.3.0] [utc-build-time="2021-11-22 10:37:02"] [go-version="go version go1.16.4 linux/amd64"] [failpoint-build=false]
[2022/01/14 18:14:33.011 +08:00] [INFO] [server.go:67] ["creating CDC server"] [pd-addrs="[http://10.59.111.225:2379,http://10.59.111.226:2379,http://10.59.111.227:2379]"] [config="{\"addr\":\"0.0.0.0:8300\",\"advertise-addr\":\"10.59.110.93:8300\",\"log-file\":\"/data/cdc/8300/log/cdc.log\",\"log-level\":\"info\",\"log\":{\"file\":{\"max-size\":300,\"max-days\":0,\"max-backups\":0}},\"data-dir\":\"/data/cdc/8300/store\",\"gc-ttl\":86400,\"tz\":\"System\",\"capture-session-ttl\":10,\"owner-flush-interval\":200000000,\"processor-flush-interval\":100000000,\"sorter\":{\"num-concurrent-worker\":4,\"chunk-size-limit\":134217728,\"max-memory-percentage\":30,\"max-memory-consumption\":17179869184,\"num-workerpool-goroutine\":16,\"sort-dir\":\"/tmp/sorter\"},\"security\":{\"ca-path\":\"\",\"cert-path\":\"\",\"key-path\":\"\",\"cert-allowed-cn\":null},\"per-table-memory-quota\":10485760,\"kv-client\":{\"worker-concurrent\":8,\"worker-pool-size\":0,\"region-scan-limit\":40}}"]
[2022/01/14 18:14:33.011 +08:00] [INFO] [client.go:355] ["[pd] create pd client with endpoints"] [pd-address="[http://10.59.111.225:2379,http://10.59.111.226:2379,http://10.59.111.227:2379]"]
[2022/01/14 18:14:33.017 +08:00] [INFO] [base_client.go:349] ["[pd] switch leader"] [new-leader=http://10.59.111.227:2379] [old-leader=]
[2022/01/14 18:14:33.017 +08:00] [INFO] [base_client.go:104] ["[pd] init cluster id"] [cluster-id=6825873712670396671]
[2022/01/14 18:14:33.017 +08:00] [INFO] [client.go:651] ["[pd] tso dispatcher created"] [dc-location=global]
[2022/01/14 18:14:33.019 +08:00] [INFO] [server.go:274] ["/data/cdc/8300/store is set as data-dir (99GB available), sort-dir=/data/cdc/8300/store/tmp/sorter. It is recommended that the disk for data-dir at least have 500GB available space"]
[2022/01/14 18:14:33.023 +08:00] [INFO] [store.go:74] ["new store"] [path="tikv://10.59.111.225:2379,10.59.111.226:2379,10.59.111.227:2379?disableGC=true"]
[2022/01/14 18:14:33.023 +08:00] [INFO] [client.go:355] ["[pd] create pd client with endpoints"] [pd-address="[10.59.111.225:2379,10.59.111.226:2379,10.59.111.227:2379]"]
[2022/01/14 18:14:33.026 +08:00] [INFO] [base_client.go:349] ["[pd] switch leader"] [new-leader=http://10.59.111.227:2379] [old-leader=]
[2022/01/14 18:14:33.026 +08:00] [INFO] [base_client.go:104] ["[pd] init cluster id"] [cluster-id=6825873712670396671]
[2022/01/14 18:14:33.026 +08:00] [INFO] [client.go:651] ["[pd] tso dispatcher created"] [dc-location=global]
[2022/01/14 18:14:33.029 +08:00] [INFO] [store.go:80] ["new store with retry success"]
[2022/01/14 18:14:33.029 +08:00] [INFO] [http_status.go:80] ["http server is running"] [addr=0.0.0.0:8300]
[2022/01/14 18:14:33.032 +08:00] [INFO] [capture.go:107] ["init capture"] [capture-id=e805ea9a-ada3-4a84-b90c-c830230ecbfd] [capture-addr=10.59.110.93:8300]
[2022/01/14 18:14:33.045 +08:00] [INFO] [reactor_state.go:74] ["remote capture online"] [capture-id=e805ea9a-ada3-4a84-b90c-c830230ecbfd] [info="{\"id\":\"e805ea9a-ada3-4a84-b90c-c830230ecbfd\",\"address\":\"10.59.110.93:8300\",\"version\":\"v5.3.0\"}"]
[2022/01/14 18:14:33.047 +08:00] [INFO] [capture.go:189] ["the processor routine has exited"] [error="json: cannot unmarshal number into Go struct field TaskStatus.tables of type model.TableReplicaInfo"] [errorVerbose="json: cannot unmarshal number into Go struct field TaskStatus.tables of type model.TableReplicaInfo\
github.com/pingcap/errors.AddStack\
\tgithub.com/pingcap/errors@v0.11.5-0.20210513014640-40f9a1999b3b/errors.go:174\
github.com/pingcap/errors.Trace\
\tgithub.com/pingcap/errors@v0.11.5-0.20210513014640-40f9a1999b3b/juju_adaptor.go:15\
github.com/pingcap/ticdc/pkg/orchestrator.(*ChangefeedReactorState).UpdateCDCKey\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/reactor_state.go:209\
github.com/pingcap/ticdc/pkg/orchestrator.(*GlobalReactorState).Update\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/reactor_state.go:89\
github.com/pingcap/ticdc/pkg/orchestrator.(*EtcdWorker).syncRawState\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/etcd_worker.go:266\
github.com/pingcap/ticdc/pkg/orchestrator.(*EtcdWorker).Run\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/etcd_worker.go:100\
github.com/pingcap/ticdc/cdc/capture.(*Capture).runEtcdWorker\
\tgithub.com/pingcap/ticdc/cdc/capture/capture.go:273\
github.com/pingcap/ticdc/cdc/capture.(*Capture).run.func3\
\tgithub.com/pingcap/ticdc/cdc/capture/capture.go:188\
runtime.goexit\
\truntime/asm_amd64.s:1371"]
[2022/01/14 18:14:33.050 +08:00] [INFO] [capture.go:178] ["the owner routine has exited"] []
[2022/01/14 18:14:33.053 +08:00] [INFO] [capture.go:145] ["the capture routine has exited"]
[2022/01/14 18:14:33.053 +08:00] [ERROR] [client.go:750] ["[pd] fetch pending tso requests error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]context canceled: context canceled"]
[2022/01/14 18:14:33.053 +08:00] [INFO] [client.go:669] ["[pd] exit tso dispatcher"] [dc-location=global]
[2022/01/14 18:14:33.054 +08:00] [ERROR] [server.go:130] ["run server"] [error="json: cannot unmarshal number into Go struct field TaskStatus.tables of type model.TableReplicaInfo\
github.com/pingcap/errors.AddStack\
\tgithub.com/pingcap/errors@v0.11.5-0.20210513014640-40f9a1999b3b/errors.go:174\
github.com/pingcap/errors.Trace\
\tgithub.com/pingcap/errors@v0.11.5-0.20210513014640-40f9a1999b3b/juju_adaptor.go:15\
github.com/pingcap/ticdc/pkg/orchestrator.(*ChangefeedReactorState).UpdateCDCKey\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/reactor_state.go:209\
github.com/pingcap/ticdc/pkg/orchestrator.(*GlobalReactorState).Update\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/reactor_state.go:89\
github.com/pingcap/ticdc/pkg/orchestrator.(*EtcdWorker).syncRawState\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/etcd_worker.go:266\
github.com/pingcap/ticdc/pkg/orchestrator.(*EtcdWorker).Run\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/etcd_worker.go:100\
github.com/pingcap/ticdc/cdc/capture.(*Capture).runEtcdWorker\
\tgithub.com/pingcap/ticdc/cdc/capture/capture.go:273\
github.com/pingcap/ticdc/cdc/capture.(*Capture).run.func3\
\tgithub.com/pingcap/ticdc/cdc/capture/capture.go:188\
runtime.goexit\
\truntime/asm_amd64.s:1371\
processor exited with error"]
[2022/01/14 18:14:33.054 +08:00] [ERROR] [client.go:750] ["[pd] fetch pending tso requests error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]context canceled: context canceled"]
[2022/01/14 18:14:33.054 +08:00] [INFO] [client.go:669] ["[pd] exit tso dispatcher"] [dc-location=global]
[2022/01/14 18:14:48.262 +08:00] [INFO] [helper.go:51] ["init log"] [file=/data/cdc/8300/log/cdc.log] [level=info]
[2022/01/14 18:14:48.263 +08:00] [INFO] [version.go:47] ["Welcome to Change Data Capture (CDC)"] [release-version=v5.3.0] [git-hash=20626babf21fc381d4364646c40dd84598533d66] [git-branch=heads/refs/tags/v5.3.0] [utc-build-time="2021-11-22 10:37:02"] [go-version="go version go1.16.4 linux/amd64"] [failpoint-build=false]
[2022/01/14 18:14:48.263 +08:00] [INFO] [server.go:67] ["creating CDC server"] [pd-addrs="[http://10.59.111.225:2379,http://10.59.111.226:2379,http://10.59.111.227:2379]"] [config="{\"addr\":\"0.0.0.0:8300\",\"advertise-addr\":\"10.59.110.93:8300\",\"log-file\":\"/data/cdc/8300/log/cdc.log\",\"log-level\":\"info\",\"log\":{\"file\":{\"max-size\":300,\"max-days\":0,\"max-backups\":0}},\"data-dir\":\"/data/cdc/8300/store\",\"gc-ttl\":86400,\"tz\":\"System\",\"capture-session-ttl\":10,\"owner-flush-interval\":200000000,\"processor-flush-interval\":100000000,\"sorter\":{\"num-concurrent-worker\":4,\"chunk-size-limit\":134217728,\"max-memory-percentage\":30,\"max-memory-consumption\":17179869184,\"num-workerpool-goroutine\":16,\"sort-dir\":\"/tmp/sorter\"},\"security\":{\"ca-path\":\"\",\"cert-path\":\"\",\"key-path\":\"\",\"cert-allowed-cn\":null},\"per-table-memory-quota\":10485760,\"kv-client\":{\"worker-concurrent\":8,\"worker-pool-size\":0,\"region-scan-limit\":40}}"]
[2022/01/14 18:14:48.263 +08:00] [INFO] [client.go:355] ["[pd] create pd client with endpoints"] [pd-address="[http://10.59.111.225:2379,http://10.59.111.226:2379,http://10.59.111.227:2379]"]
[2022/01/14 18:14:48.270 +08:00] [INFO] [base_client.go:349] ["[pd] switch leader"] [new-leader=http://10.59.111.227:2379] [old-leader=]
[2022/01/14 18:14:48.270 +08:00] [INFO] [base_client.go:104] ["[pd] init cluster id"] [cluster-id=6825873712670396671]
[2022/01/14 18:14:48.270 +08:00] [INFO] [client.go:651] ["[pd] tso dispatcher created"] [dc-location=global]
[2022/01/14 18:14:48.271 +08:00] [INFO] [server.go:274] ["/data/cdc/8300/store is set as data-dir (99GB available), sort-dir=/data/cdc/8300/store/tmp/sorter. It is recommended that the disk for data-dir at least have 500GB available space"]
[2022/01/14 18:14:48.275 +08:00] [INFO] [store.go:74] ["new store"] [path="tikv://10.59.111.225:2379,10.59.111.226:2379,10.59.111.227:2379?disableGC=true"]
[2022/01/14 18:14:48.275 +08:00] [INFO] [client.go:355] ["[pd] create pd client with endpoints"] [pd-address="[10.59.111.225:2379,10.59.111.226:2379,10.59.111.227:2379]"]
[2022/01/14 18:14:48.278 +08:00] [INFO] [base_client.go:349] ["[pd] switch leader"] [new-leader=http://10.59.111.227:2379] [old-leader=]
[2022/01/14 18:14:48.278 +08:00] [INFO] [base_client.go:104] ["[pd] init cluster id"] [cluster-id=6825873712670396671]
[2022/01/14 18:14:48.278 +08:00] [INFO] [client.go:651] ["[pd] tso dispatcher created"] [dc-location=global]
[2022/01/14 18:14:48.281 +08:00] [INFO] [store.go:80] ["new store with retry success"]
[2022/01/14 18:14:48.281 +08:00] [INFO] [http_status.go:80] ["http server is running"] [addr=0.0.0.0:8300]
[2022/01/14 18:14:48.284 +08:00] [INFO] [capture.go:107] ["init capture"] [capture-id=c2335fe8-c4e9-406b-aad1-b6d235b10f46] [capture-addr=10.59.110.93:8300]
[2022/01/14 18:14:48.292 +08:00] [INFO] [reactor_state.go:74] ["remote capture online"] [capture-id=c2335fe8-c4e9-406b-aad1-b6d235b10f46] [info="{\"id\":\"c2335fe8-c4e9-406b-aad1-b6d235b10f46\",\"address\":\"10.59.110.93:8300\",\"version\":\"v5.3.0\"}"]
[2022/01/14 18:14:48.294 +08:00] [INFO] [capture.go:189] ["the processor routine has exited"] [error="json: cannot unmarshal number into Go struct field TaskStatus.tables of type model.TableReplicaInfo"] [errorVerbose="json: cannot unmarshal number into Go struct field TaskStatus.tables of type model.TableReplicaInfo\
github.com/pingcap/errors.AddStack\
\tgithub.com/pingcap/errors@v0.11.5-0.20210513014640-40f9a1999b3b/errors.go:174\
github.com/pingcap/errors.Trace\
\tgithub.com/pingcap/errors@v0.11.5-0.20210513014640-40f9a1999b3b/juju_adaptor.go:15\
github.com/pingcap/ticdc/pkg/orchestrator.(*ChangefeedReactorState).UpdateCDCKey\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/reactor_state.go:209\
github.com/pingcap/ticdc/pkg/orchestrator.(*GlobalReactorState).Update\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/reactor_state.go:89\
github.com/pingcap/ticdc/pkg/orchestrator.(*EtcdWorker).syncRawState\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/etcd_worker.go:266\
github.com/pingcap/ticdc/pkg/orchestrator.(*EtcdWorker).Run\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/etcd_worker.go:100\
github.com/pingcap/ticdc/cdc/capture.(*Capture).runEtcdWorker\
\tgithub.com/pingcap/ticdc/cdc/capture/capture.go:273\
github.com/pingcap/ticdc/cdc/capture.(*Capture).run.func3\
\tgithub.com/pingcap/ticdc/cdc/capture/capture.go:188\
runtime.goexit\
\truntime/asm_amd64.s:1371"]
[2022/01/14 18:14:48.299 +08:00] [INFO] [capture.go:178] ["the owner routine has exited"] []
[2022/01/14 18:14:48.301 +08:00] [INFO] [capture.go:145] ["the capture routine has exited"]
[2022/01/14 18:14:48.302 +08:00] [ERROR] [client.go:750] ["[pd] fetch pending tso requests error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]context canceled: context canceled"]
[2022/01/14 18:14:48.302 +08:00] [INFO] [client.go:669] ["[pd] exit tso dispatcher"] [dc-location=global]
[2022/01/14 18:14:48.302 +08:00] [ERROR] [server.go:130] ["run server"] [error="json: cannot unmarshal number into Go struct field TaskStatus.tables of type model.TableReplicaInfo\
github.com/pingcap/errors.AddStack\
\tgithub.com/pingcap/errors@v0.11.5-0.20210513014640-40f9a1999b3b/errors.go:174\
github.com/pingcap/errors.Trace\
\tgithub.com/pingcap/errors@v0.11.5-0.20210513014640-40f9a1999b3b/juju_adaptor.go:15\
github.com/pingcap/ticdc/pkg/orchestrator.(*ChangefeedReactorState).UpdateCDCKey\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/reactor_state.go:209\
github.com/pingcap/ticdc/pkg/orchestrator.(*GlobalReactorState).Update\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/reactor_state.go:89\
github.com/pingcap/ticdc/pkg/orchestrator.(*EtcdWorker).syncRawState\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/etcd_worker.go:266\
github.com/pingcap/ticdc/pkg/orchestrator.(*EtcdWorker).Run\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/etcd_worker.go:100\
github.com/pingcap/ticdc/cdc/capture.(*Capture).runEtcdWorker\
\tgithub.com/pingcap/ticdc/cdc/capture/capture.go:273\
github.com/pingcap/ticdc/cdc/capture.(*Capture).run.func3\
\tgithub.com/pingcap/ticdc/cdc/capture/capture.go:188\
runtime.goexit\
\truntime/asm_amd64.s:1371\
processor exited with error"]
[2022/01/14 18:14:48.302 +08:00] [ERROR] [client.go:750] ["[pd] fetch pending tso requests error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]context canceled: context canceled"]
[2022/01/14 18:14:48.302 +08:00] [INFO] [client.go:669] ["[pd] exit tso dispatcher"] [dc-location=global]
[2022/01/14 18:15:03.507 +08:00] [INFO] [helper.go:51] ["init log"] [file=/data/cdc/8300/log/cdc.log] [level=info]
[2022/01/14 18:15:03.508 +08:00] [INFO] [version.go:47] ["Welcome to Change Data Capture (CDC)"] [release-version=v5.3.0] [git-hash=20626babf21fc381d4364646c40dd84598533d66] [git-branch=heads/refs/tags/v5.3.0] [utc-build-time="2021-11-22 10:37:02"] [go-version="go version go1.16.4 linux/amd64"] [failpoint-build=false]
[2022/01/14 18:15:03.508 +08:00] [INFO] [server.go:67] ["creating CDC server"] [pd-addrs="[http://10.59.111.225:2379,http://10.59.111.226:2379,http://10.59.111.227:2379]"] [config="{\"addr\":\"0.0.0.0:8300\",\"advertise-addr\":\"10.59.110.93:8300\",\"log-file\":\"/data/cdc/8300/log/cdc.log\",\"log-level\":\"info\",\"log\":{\"file\":{\"max-size\":300,\"max-days\":0,\"max-backups\":0}},\"data-dir\":\"/data/cdc/8300/store\",\"gc-ttl\":86400,\"tz\":\"System\",\"capture-session-ttl\":10,\"owner-flush-interval\":200000000,\"processor-flush-interval\":100000000,\"sorter\":{\"num-concurrent-worker\":4,\"chunk-size-limit\":134217728,\"max-memory-percentage\":30,\"max-memory-consumption\":17179869184,\"num-workerpool-goroutine\":16,\"sort-dir\":\"/tmp/sorter\"},\"security\":{\"ca-path\":\"\",\"cert-path\":\"\",\"key-path\":\"\",\"cert-allowed-cn\":null},\"per-table-memory-quota\":10485760,\"kv-client\":{\"worker-concurrent\":8,\"worker-pool-size\":0,\"region-scan-limit\":40}}"]
[2022/01/14 18:15:03.508 +08:00] [INFO] [client.go:355] ["[pd] create pd client with endpoints"] [pd-address="[http://10.59.111.225:2379,http://10.59.111.226:2379,http://10.59.111.227:2379]"]
[2022/01/14 18:15:03.515 +08:00] [INFO] [base_client.go:349] ["[pd] switch leader"] [new-leader=http://10.59.111.227:2379] [old-leader=]
[2022/01/14 18:15:03.515 +08:00] [INFO] [base_client.go:104] ["[pd] init cluster id"] [cluster-id=6825873712670396671]
[2022/01/14 18:15:03.515 +08:00] [INFO] [client.go:651] ["[pd] tso dispatcher created"] [dc-location=global]
[2022/01/14 18:15:03.517 +08:00] [INFO] [server.go:274] ["/data/cdc/8300/store is set as data-dir (99GB available), sort-dir=/data/cdc/8300/store/tmp/sorter. It is recommended that the disk for data-dir at least have 500GB available space"]
[2022/01/14 18:15:03.520 +08:00] [INFO] [store.go:74] ["new store"] [path="tikv://10.59.111.225:2379,10.59.111.226:2379,10.59.111.227:2379?disableGC=true"]
[2022/01/14 18:15:03.520 +08:00] [INFO] [client.go:355] ["[pd] create pd client with endpoints"] [pd-address="[10.59.111.225:2379,10.59.111.226:2379,10.59.111.227:2379]"]
[2022/01/14 18:15:03.524 +08:00] [INFO] [base_client.go:349] ["[pd] switch leader"] [new-leader=http://10.59.111.227:2379] [old-leader=]
[2022/01/14 18:15:03.524 +08:00] [INFO] [base_client.go:104] ["[pd] init cluster id"] [cluster-id=6825873712670396671]
[2022/01/14 18:15:03.524 +08:00] [INFO] [client.go:651] ["[pd] tso dispatcher created"] [dc-location=global]
[2022/01/14 18:15:03.528 +08:00] [INFO] [store.go:80] ["new store with retry success"]
[2022/01/14 18:15:03.528 +08:00] [INFO] [http_status.go:80] ["http server is running"] [addr=0.0.0.0:8300]
[2022/01/14 18:15:03.531 +08:00] [INFO] [capture.go:107] ["init capture"] [capture-id=49b58df0-1b2a-403a-94ba-45046613c08e] [capture-addr=10.59.110.93:8300]
[2022/01/14 18:15:03.548 +08:00] [INFO] [reactor_state.go:74] ["remote capture online"] [capture-id=49b58df0-1b2a-403a-94ba-45046613c08e] [info="{\"id\":\"49b58df0-1b2a-403a-94ba-45046613c08e\",\"address\":\"10.59.110.93:8300\",\"version\":\"v5.3.0\"}"]
[2022/01/14 18:15:03.551 +08:00] [INFO] [capture.go:189] ["the processor routine has exited"] [error="json: cannot unmarshal number into Go struct field TaskStatus.tables of type model.TableReplicaInfo"] [errorVerbose="json: cannot unmarshal number into Go struct field TaskStatus.tables of type model.TableReplicaInfo\
github.com/pingcap/errors.AddStack\
\tgithub.com/pingcap/errors@v0.11.5-0.20210513014640-40f9a1999b3b/errors.go:174\
github.com/pingcap/errors.Trace\
\tgithub.com/pingcap/errors@v0.11.5-0.20210513014640-40f9a1999b3b/juju_adaptor.go:15\
github.com/pingcap/ticdc/pkg/orchestrator.(*ChangefeedReactorState).UpdateCDCKey\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/reactor_state.go:209\
github.com/pingcap/ticdc/pkg/orchestrator.(*GlobalReactorState).Update\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/reactor_state.go:89\
github.com/pingcap/ticdc/pkg/orchestrator.(*EtcdWorker).syncRawState\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/etcd_worker.go:266\
github.com/pingcap/ticdc/pkg/orchestrator.(*EtcdWorker).Run\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/etcd_worker.go:100\
github.com/pingcap/ticdc/cdc/capture.(*Capture).runEtcdWorker\
\tgithub.com/pingcap/ticdc/cdc/capture/capture.go:273\
github.com/pingcap/ticdc/cdc/capture.(*Capture).run.func3\
\tgithub.com/pingcap/ticdc/cdc/capture/capture.go:188\
runtime.goexit\
\truntime/asm_amd64.s:1371"]
[2022/01/14 18:15:03.556 +08:00] [INFO] [capture.go:178] ["the owner routine has exited"] []
[2022/01/14 18:15:03.559 +08:00] [INFO] [capture.go:145] ["the capture routine has exited"]
[2022/01/14 18:15:03.559 +08:00] [ERROR] [client.go:750] ["[pd] fetch pending tso requests error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]context canceled: context canceled"]
[2022/01/14 18:15:03.559 +08:00] [INFO] [client.go:669] ["[pd] exit tso dispatcher"] [dc-location=global]
[2022/01/14 18:15:03.559 +08:00] [ERROR] [server.go:130] ["run server"] [error="json: cannot unmarshal number into Go struct field TaskStatus.tables of type model.TableReplicaInfo\
github.com/pingcap/errors.AddStack\
\tgithub.com/pingcap/errors@v0.11.5-0.20210513014640-40f9a1999b3b/errors.go:174\
github.com/pingcap/errors.Trace\
\tgithub.com/pingcap/errors@v0.11.5-0.20210513014640-40f9a1999b3b/juju_adaptor.go:15\
github.com/pingcap/ticdc/pkg/orchestrator.(*ChangefeedReactorState).UpdateCDCKey\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/reactor_state.go:209\
github.com/pingcap/ticdc/pkg/orchestrator.(*GlobalReactorState).Update\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/reactor_state.go:89\
github.com/pingcap/ticdc/pkg/orchestrator.(*EtcdWorker).syncRawState\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/etcd_worker.go:266\
github.com/pingcap/ticdc/pkg/orchestrator.(*EtcdWorker).Run\
\tgithub.com/pingcap/ticdc/pkg/orchestrator/etcd_worker.go:100\
github.com/pingcap/ticdc/cdc/capture.(*Capture).runEtcdWorker\
\tgithub.com/pingcap/ticdc/cdc/capture/capture.go:273\
github.com/pingcap/ticdc/cdc/capture.(*Capture).run.func3\
\tgithub.com/pingcap/ticdc/cdc/capture/capture.go:188\
runtime.goexit\
\truntime/asm_amd64.s:1371\
processor exited with error"]
[2022/01/14 18:15:03.559 +08:00] [ERROR] [client.go:750] ["[pd] fetch pending tso requests error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]context canceled: context canceled"]
[2022/01/14 18:15:03.559 +08:00] [INFO] [client.go:669] ["[pd] exit tso dispatcher"] [dc-location=global]

请问pd能访问cdc的8300端口么

其中一个节点是起来的可以访问,另外两个ticdc没起来访问不了,ping都是通的

可以把配置文件发下么,扩容cdc的配置文件,感觉写的有问题

cdc_servers:
   - host: 10.59.110.178
     deploy_dir: "/data/cdc/8300"
     log_dir: "/data/cdc/8300/log"
     data_dir: "/data/cdc/8300/store"
   - host: 10.59.110.17
     deploy_dir: "/data/cdc/8300"
     log_dir: "/data/cdc/8300/log"
     data_dir: "/data/cdc/8300/store"
   - host: 10.59.110.93
     deploy_dir: "/data/cdc/8300"
     log_dir: "/data/cdc/8300/log"
     data_dir: "/data/cdc/8300/store"

5.3的没试过,感觉像是这一步出问题了


你可以先试下加个gc-ttl看下

还是不行啊

检查是不是防火墙的问题,确认下所有的cdc节点和pd之间的网络是否是通的,telnet检查一下

是的,都是通的

看错误应该是 json 反序列化失败了,然后导致启动失败,我们排查一下。

请问是第一次部署 cdc 么?还是说是从某个版本上级上来的呢,如果是的话,原始版本是什么?

能否同时提供一下完整的 tiup cluster deploy 使用的 yaml 文件,:pray:

是第一次部署ticdc

global:
  user: "tidb"
  ssh_port: 22
  deploy_dir: "/data/tidb_deploy"
  data_dir: "/data/tidb_data"

server_configs:
  tidb:
    log.slow-threshold: 300
    binlog.enable: false
    binlog.ignore-error: false
  pd:
    schedule.leader-schedule-limit: 4
    schedule.region-schedule-limit: 2048
    schedule.replica-schedule-limit: 64
    replication.enable-placement-rules: true
  tiflash:
    path_realtime_mode: false
    logger.level: "info"

pd_servers:
  - host: 10.59.111.225
  - host: 10.59.111.226
  - host: 10.59.111.227

tidb_servers:
  - host: 10.59.111.225
  - host: 10.59.111.226
  - host: 10.59.111.227

tikv_servers:
  - host: 10.59.111.132
  - host: 10.59.111.133
  - host: 10.59.111.224

tiflash_servers:
  - host: 10.59.111.10

monitoring_servers:
  - host: 10.59.111.10

grafana_servers:
  - host: 10.59.111.10

alertmanager_servers:
  - host: 10.59.111.10

你的 ticdc 如何部署的? 为什么 tiup deploy 里面没有 ticdc 的配置信息?

1、创建cdc.yml

cdc_servers:
   - host: 10.59.110.178
     gc-ttl: 86400
     deploy_dir: "/data/cdc/8300"
     log_dir: "/data/cdc/8300/log"
     data_dir: "/data/cdc/8300/store"
   - host: 10.59.110.17
     gc-ttl: 86400
     deploy_dir: "/data/cdc/8300"
     log_dir: "/data/cdc/8300/log"
     data_dir: "/data/cdc/8300/store"
   - host: 10.59.110.93
     gc-ttl: 86400
     deploy_dir: "/data/cdc/8300"
     log_dir: "/data/cdc/8300/log"
     data_dir: "/data/cdc/8300/store"

2、tiup 部署ticdc

 tiup cluster scale-out demo cdc.yml

tidb版本:

Starting component `cluster`: /root/.tiup/components/cluster/v1.8.2/tiup-cluster display demo
Cluster type:       tidb
Cluster name:       demo
Cluster version:    v5.3.0
Deploy user:        tidb
SSH type:           builtin

cdc版本:

Release Version: v5.3.0
Git Commit Hash: 20626babf21fc381d4364646c40dd84598533d66
Git Branch: heads/refs/tags/v5.3.0
UTC Build Time: 2021-11-22 10:37:02
Go Version: go version go1.16.4 linux/amd64
Failpoint Build: false

这种部署不行么,是要写在deploy文件中?