【 TiDB 使用环境】
测试环境 tidb-4.0.9集群
【概述】 场景 + 问题概述
对tidb集群使用sysbench进行造数据，通过ticdc工具进行同步，只同步了完成了3个表 后面不同步
【背景】 做过哪些操作
1、重启过ticdc
2、对任务添加配置，支持sync-ddl参数
【现象】 业务和数据库现象
目标端TIDB集群为4.0.14版本，无压力
【问题】 当前遇到的问题
数据不同步，源端TIDB已经造了几千数据和表，CDC任务显示：
[
{
“id”: “tidb-task15”,
“summary”: {
“state”: “normal”,
“tso”: 429461400660738064,
“checkpoint”: “2021-11-30 17:40:30.792”,
“error”: null
}
}
]
目前只有发现ticdc日志有error，其他都是info信息
[2021/11/30 18:19:14.737 +08:00] [ERROR] [client.go:320] ["[pd] getTS error"] [error="[PD:client:ErrClientGetTSO]rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]gene
[2021/11/30 18:19:14.738 +08:00] [ERROR] [pd.go:130] [“updateTS error”] [error=“rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, not the pdests\
\tgithub.com/tikv/pd@v1.1.0-beta.0.20200921100508-9ee41c4144f3/client/client.go:356\
github.com/tikv/pd/client.(*client).tsLoop\
\tgithub.com/tikv/pd@v1.1.0-beta.0.202009211e41c4144f3/client/client.go:466\
github.com/tikv/pd/client.(*client).GetTS\
\tgithub.com/tikv/pd@v1.1.0-beta.0.20200921100508-9ee41c4144f3/client/client.go:484\
github.com/pingcapore/tikv/oracle/oracles.(*pdOracle).getTimestamp\
\tgithub.com/pingcap/tidb@v1.1.0-beta.0.20201214152324-ce2f365189d3/store/tikv/oracle/oracles/pd.go:103\
github.com/pingcap/tidb/amd64.s:1374”]
[2021/11/30 18:19:14.942 +08:00] [ERROR] [client.go:320] ["[pd] getTS error"] [error="[PD:client:ErrClientGetTSO]rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]gene
[2021/11/30 18:19:14.964 +08:00] [ERROR] [base_client.go:140] ["[pd] failed updateLeader"] [error="[PD:client:ErrClientGetLeader]get leader from [http://172.20.112.255:2379 http:/
[2021/11/30 18:19:15.317 +08:00] [ERROR] [client.go:320] ["[pd] getTS error"] [error="[PD:client:ErrClientGetTSO]rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]gene
[2021/11/30 18:19:15.317 +08:00] [ERROR] [pd.go:130] [“updateTS error”] [error=“rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, timestamp /tikv/pd/client.(*client).processTSORequests\
\tgithub.com/tikv/pd@v1.1.0-beta.0.20200921100508-9ee41c4144f3/client/client.go:356\
github.com/tikv/pd/client.(*client).tsLoop\
\tgi/tikv/pd@v1.1.0-beta.0.20200921100508-9ee41c4144f3/client/client.go:466\ngithub.com/tikv/pd/client.(*client).GetTS\
\tgithub.com/tikv/pd@v1.1.0-beta.0.20200921100508-9ee41c4144f3/ceptor.go:60\
github.com/pingcap/tidb/store/tikv/oracle/oracles.(*pdOracle).getTimestamp\
\tgithub.com/pingcap/tidb@v1.1.0-beta.0.20201214152324-ce2f365189d3/store/tikv/oracle/orad.go:128\
runtime.goexit\
\truntime/asm_amd64.s:1374”]
[2021/11/30 18:19:15.337 +08:00] [ERROR] [base_client.go:140] ["[pd] failed updateLeader"] [error="[PD:client:ErrClientGetLeader]get leader from [http://172.20.112.255:2379 http:/
[2021/11/30 18:19:16.190 +08:00] [ERROR] [client.go:320] ["[pd] getTS error"] [error="[PD:client:ErrClientGetTSO]rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]gene
[2021/11/30 18:19:17.318 +08:00] [ERROR] [client.go:320] ["[pd] getTS error"] [error="[PD:client:ErrClientGetTSO]rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]gene
[2021/11/30 18:19:17.318 +08:00] [ERROR] [pd.go:130] [“updateTS error”] [error=“rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, timestamp /tikv/pd/client.(*client).processTSORequests\
\tgithub.com/tikv/pd@v1.1.0-beta.0.20200921100508-9ee41c4144f3/client/client.go:356\
github.com/tikv/pd/client.(*client).tsLoop\
\tgi/tikv/pd@v1.1.0-beta.0.20200921100508-9ee41c4144f3/client/client.go:466\ngithub.com/tikv/pd/client.(*client).GetTS\
\tgithub.com/tikv/pd@v1.1.0-beta.0.20200921100508-9ee41c4144f3/ceptor.go:60\
github.com/pingcap/tidb/store/tikv/oracle/oracles.(*pdOracle).getTimestamp\
\tgithub.com/pingcap/tidb@v1.1.0-beta.0.20201214152324-ce2f365189d3/store/tikv/oracle/orad.go:128\
runtime.goexit\
\truntime/asm_amd64.s:1374”]
[2021/11/30 18:21:08.773 +08:00] [ERROR] [client.go:1076] [“failed to receive from stream”] [addr=172.20.118.103:20171] [storeID=82685] [error=“rpc error: code = Unavailable desc
[2021/11/30 18:21:09.996 +08:00] [ERROR] [client.go:1076] [“failed to receive from stream”] [addr=172.20.118.103:20171] [storeID=82685] [error=“rpc error: code = Unavailable desc
[2021/11/30 18:21:10.301 +08:00] [ERROR] [client.go:1076] [“failed to receive from stream”] [addr=172.20.118.103:20171] [storeID=82685] [error=“rpc error: code = Unavailable desc
[2021/11/30 18:21:10.610 +08:00] [ERROR] [client.go:1076] [“failed to receive from stream”] [addr=172.20.118.103:20171] [storeID=82685] [error=“rpc error: code = Unavailable desc
[2021/11/30 18:41:34.439 +08:00] [ERROR] [client.go:1076] [“failed to receive from stream”] [addr=172.20.118.103:20171] [storeID=82685] [error=“rpc error: code = Unavailable desc
[2021/11/30 18:41:34.546 +08:00] [ERROR] [client.go:1076] [“failed to receive from stream”] [addr=172.20.112.255:20171] [storeID=84153] [error=“rpc error: code = Unavailable desc
[2021/11/30 18:41:34.660 +08:00] [ERROR] [client.go:764] [“send request to stream failed”] [addr=172.20.118.103:20171] [storeID=82685] [regionID=137557] [requestID=6060] [error=EO
[2021/11/30 18:42:01.195 +08:00] [ERROR] [client.go:201] [“tso request is canceled due to timeout”] [error=”[PD:client:ErrClientGetTSOTimeout]get TSO timeout”]
[2021/11/30 18:42:01.195 +08:00] [ERROR] [client.go:320] [”[pd] getTS error”] [error=”[PD:client:ErrClientGetTSO]rpc error: code = Canceled desc = context canceled”]
[2021/11/30 18:52:03.597 +08:00] [ERROR] [client.go:1076] [“failed to receive from stream”] [addr=172.20.235.147:20171] [storeID=67201] [error=“rpc error: code = Unavailable desc
[2021/11/30 18:52:03.617 +08:00] [ERROR] [client.go:764] [“send request to stream failed”] [addr=172.20.235.147:20171] [storeID=67201] [regionID=137773] [requestID=6087] [error=EO
[2021/11/30 18:52:03.736 +08:00] [ERROR] [client.go:764] [“send request to stream failed”] [addr=172.20.112.255:20171] [storeID=84153] [regionID=138841] [requestID=6093] [error=EO
[2021/11/30 18:52:49.363 +08:00] [ERROR] [client.go:1076] [“failed to receive from stream”] [addr=172.20.235.147:20171] [storeID=67201] [error=“rpc error: code = Unavailable desc
[root@xhhost75 logs]#
WARN日志提示：
[2021/11/30 19:16:48.907 +08:00] [WARN] [client.go:729] [“get grpc stream client failed”] [regionID=137901] [requestID=6313] [storeID=82685] [error=”[CDC:ErrTiKVEventFeed]rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing dial tcp 172.20.118.103:20171: connect: connection refused””]
[2021/11/30 19:16:49.167 +08:00] [WARN] [client.go:729] [“get grpc stream client failed”] [regionID=137581] [requestID=6315] [storeID=82685] [error="[CDC:ErrTiKVEventFeed]rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing dial tcp 172.20.118.103:20171: connect: connection refused”"]
[2021/11/30 19:16:49.544 +08:00] [WARN] [client.go:729] [“get grpc stream client failed”] [regionID=138421] [requestID=6317] [storeID=82685] [error="[CDC:ErrTiKVEventFeed]rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing dial tcp 172.20.118.103:20171: connect: connection refused”"]
[2021/11/30 19:16:49.874 +08:00] [WARN] [client.go:729] [“get grpc stream client failed”] [regionID=137829] [requestID=6319] [storeID=82685] [error="[CDC:ErrTiKVEventFeed]rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing dial tcp 172.20.118.103:20171: connect: connection refused”"]
[2021/11/30 19:16:50.192 +08:00] [WARN] [client.go:729] [“get grpc stream client failed”] [regionID=138985] [requestID=6321] [storeID=82685] [error="[CDC:ErrTiKVEventFeed]rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing dial tcp 172.20.118.103:20171: connect: connection refused”"]
[2021/11/30 19:16:50.511 +08:00] [WARN] [client.go:729] [“get grpc stream client failed”] [regionID=138233] [requestID=6323] [storeID=82685] [error="[CDC:ErrTiKVEventFeed]rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing dial tcp 172.20.118.103:20171: connect: connection refused”"]
[2021/11/30 19:16:50.808 +08:00] [WARN] [client.go:729] [“get grpc stream client failed”] [regionID=138329] [requestID=6325] [storeID=82685] [error="[CDC:ErrTiKVEventFeed]rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing dial tcp 172.20.118.103:20171: connect: connection refused”"]
[2021/11/30 19:16:51.140 +08:00] [WARN] [client.go:729] [“get grpc stream client failed”] [regionID=138861] [requestID=6327] [storeID=82685] [error="[CDC:ErrTiKVEventFeed]rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing dial tcp 172.20.118.103:20171: connect: connection refused”"]
[2021/11/30 19:16:51.440 +08:00] [WARN] [client.go:729] [“get grpc stream client failed”] [regionID=138669] [requestID=6329] [storeID=82685] [error="[CDC:ErrTiKVEventFeed]rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing dial tcp 172.20.118.103:20171: connect: connection refused”"]
[2021/11/30 19:16:51.762 +08:00] [WARN] [client.go:729] [“get grpc stream client failed”] [regionID=137673] [requestID=6331] [storeID=82685] [error="[CDC:ErrTiKVEventFeed]rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing dial tcp 172.20.118.103:20171: connect: connection refused”"]
[2021/11/30 19:16:52.151 +08:00] [WARN] [client.go:729] [“get grpc stream client failed”] [regionID=138741] [requestID=6333] [storeID=82685] [error="[CDC:ErrTiKVEventFeed]rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing dial tcp 172.20.118.103:20171: connect: connection refused”"]
[2021/11/30 19:16:52.488 +08:00] [WARN] [client.go:729] [“get grpc stream client failed”] [regionID=137501] [requestID=6335] [storeID=82685] [error="[CDC:ErrTiKVEventFeed]rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing dial tcp 172.20.118.103:20171: connect: connection refused”"]
[2021/11/30 19:16:52.816 +08:00] [WARN] [client.go:729] [“get grpc stream client failed”] [regionID=137749] [requestID=6337] [storeID=82685] [error="[CDC:ErrTiKVEventFeed]rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing dial tcp 172.20.118.103:20171: connect: connection refused”"]
[2021/11/30 19:16:53.099 +08:00] [WARN] [client.go:729] [“get grpc stream client failed”] [regionID=138605] [requestID=6341] [storeID=82685] [error="[CDC:ErrTiKVEventFeed]rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing dial tcp 172.20.118.103:20171: connect: connection refused”"]
[2021/11/30 19:16:53.406 +08:00] [WARN] [client.go:729] [“get grpc stream client failed”] [regionID=138801] [requestID=6344] [storeID=82685] [error="[CDC:ErrTiKVEventFeed]rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing dial tcp 172.20.118.103:20171: connect: connection refused”"]
[2021/11/30 19:16:53.923 +08:00] [WARN] [client.go:729] [“get grpc stream client failed”] [regionID=137901] [requestID=6347] [storeID=82685] [error="[CDC:ErrTiKVEventFeed]rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing dial tcp 172.20.118.103:20171: connect: connection refused”"]
[2021/11/30 19:16:54.774 +08:00] [WARN] [client.go:729] [“get grpc stream client failed”] [regionID=137501] [requestID=6350] [storeID=82685] [error="[CDC:ErrTiKVEventFeed]rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing dial tcp 172.20.118.103:20171: connect: connection refused”"]
[2021/11/30 19:16:55.851 +08:00] [WARN] [client.go:729] [“get grpc stream client failed”] [regionID=138669] [requestID=6353] [storeID=82685] [error="[CDC:ErrTiKVEventFeed]rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing dial tcp 172.20.118.103:20171: connect: connection refused”"]
[2021/11/30 19:16:57.260 +08:00] [WARN] [client.go:729] [“get grpc stream client failed”] [regionID=137673] [requestID=6358] [storeID=82685] [error="[CDC:ErrTiKVEventFeed]rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing dial tcp 172.20.118.103:20171: connect: connection refused”"]
[2021/11/30 19:16:59.455 +08:00] [WARN] [client.go:729] [“get grpc stream client failed”] [regionID=138861] [requestID=6361] [storeID=82685] [error="[CDC:ErrTiKVEventFeed]rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing dial tcp 172.20.118.103:20171: connect: connection refused”"]
[2021/11/30 19:21:29.510 +08:00] [WARN] [client.go:1357] [“The time cost of initializing is too mush”] [timeCost=4m48.862201773s] [regionID=138913]
[2021/11/30 19:21:30.648 +08:00] [WARN] [client.go:1325] [“region not receiving resolved event from tikv or resolved ts is not pushing for too long time, try to resolve lock”] [regionID=138913] [span="[748000000000002eff245f728000000000ff3362980000000000fa, 748000000000002eff245f728000000000ff39f0050000000000fa)"] [duration=1h40m58.417s] [resolvedTs=429461400660738064]
[2021/11/30 19:21:35.648 +08:00] [WARN] [client.go:1325] [“region not receiving resolved event from tikv or resolved ts is not pushing for too long time, try to resolve lock”] [regionID=138913] [span="[748000000000002eff245f728000000000ff3362980000000000fa, 748000000000002eff245f728000000000ff39f0050000000000fa)"] [duration=1h41m3.417s] [resolvedTs=429461400660738064]
[2021/11/30 19:21:40.648 +08:00] [WARN] [client.go:1325] [“region not receiving resolved event from tikv or resolved ts is not pushing for too long time, try to resolve lock”] [regionID=138913] [span="[748000000000002eff245f728000000000ff3362980000000000fa, 748000000000002eff245f728000000000ff39f0050000000000fa)"] [duration=1h41m8.417s] [resolvedTs=429461400660738064]
[2021/11/30 19:21:45.648 +08:00] [WARN] [client.go:1325] [“region not receiving resolved event from tikv or resolved ts is not pushing for too long time, try to resolve lock”] [regionID=138913] [span="[748000000000002eff245f728000000000ff3362980000000000fa, 748000000000002eff245f728000000000ff39f0050000000000fa)"] [duration=1h41m13.617s] [resolvedTs=429461400660738064]
[2021/11/30 19:21:50.648 +08:00] [WARN] [client.go:1314] [“region not receiving event from tikv for too long time”] [regionID=138913] [span="[748000000000002eff245f728000000000ff3362980000000000fa, 748000000000002eff245f728000000000ff39f0050000000000fa)"] [duration=21.138075317s]
[2021/11/30 19:21:50.648 +08:00] [WARN] [client.go:1325] [“region not receiving resolved event from tikv or resolved ts is not pushing for too long time, try to resolve lock”] [regionID=138913] [span="[748000000000002eff245f728000000000ff3362980000000000fa, 748000000000002eff245f728000000000ff39f0050000000000fa)"] [duration=1h41m18.417s] [resolvedTs=429461400660738064]
[2021/11/30 19:21:55.648 +08:00] [WARN] [client.go:1314] [“region not receiving event from tikv for too long time”] [regionID=138913] [span="[748000000000002eff245f728000000000ff3362980000000000fa, 748000000000002eff245f728000000000ff39f0050000000000fa)"] [duration=26.138076431s]
[2021/11/30 19:21:55.648 +08:00] [WARN] [client.go:1325] [“region not receiving resolved event from tikv or resolved ts is not pushing for too long time, try to resolve lock”] [regionID=138913] [span="[748000000000002eff245f728000000000ff3362980000000000fa, 748000000000002eff245f728000000000ff39f0050000000000fa)"] [duration=1h41m23.417s] [resolvedTs=429461400660738064]
[2021/11/30 19:22:00.648 +08:00] [WARN] [client.go:1314] [“region not receiving event from tikv for too long time”] [regionID=138913] [span="[748000000000002eff245f728000000000ff3362980000000000fa, 748000000000002eff245f728000000000ff39f0050000000000fa)"] [duration=31.138129719s]
[2021/11/30 19:22:00.648 +08:00] [WARN] [client.go:1325] [“region not receiving resolved event from tikv or resolved ts is not pushing for too long time, try to resolve lock”] [regionID=138913] [span="[748000000000002eff245f728000000000ff3362980000000000fa, 748000000000002eff245f728000000000ff39f0050000000000fa)"] [duration=1h41m28.417s] [resolvedTs=429461400660738064]
实际TIKV是存活着,在CDC机器上telnet也是通的
【TiDB 版本】
源端4.0.9
