ti cdc异常不消费,tso不动

【 TiDB 使用环境】生产环境
【 TiDB 版本】7.5.3
【遇到的问题】
任务有正常再跑的,最近晚上总是有异常积压的情况,但是观察日志层面没用error的日志 ,只有如下可能的日志

[2024/11/20 03:13:18.386 +08:00] [WARN] [shared_client.go:675] ["event feed resolve lock fail"] [namespace=default] [changefeed=dgdg-test] [regionID=20168796] [error="unexpected scanlock error: error:<locked:<primary_lock:\"t\\200\\000\\000\\000\\000\\000\\001\\030_i\\200\\000\\000\\000\\000\\000\\000\\001\\0010d5f78c2\\377750c846f\\3775bbac63a\\377cf418037\\377\\000\\000\\000\\000\\000\\000\\000\\000\\367\" lock_version:454044832182042663 key:\"t\\200\\000\\000\\000\\000\\000\\001\\030_r\\200\\000\\000\\001\\351\\003\\252\\246\" lock_ttl:29588 txn_size:1 lock_type:Lock lock_for_update_ts:454044832182042663 use_async_commit:true min_commit_ts:454044837044551774 > > exec_details_v2:<time_detail:<total_rpc_wall_time_ns:59238 > time_detail_v2:<total_rpc_wall_time_ns:59238 > > "] [errorVerbose="unexpected scanlock error: error:<locked:<primary_lock:\"t\\200\\000\\000\\000\\000\\000\\001\\030_i\\200\\000\\000\\000\\000\\000\\000\\001\\0010d5f78c2\\377750c846f\\3775bbac63a\\377cf418037\\377\\000\\000\\000\\000\\000\\000\\000\\000\\367\" lock_version:454044832182042663 key:\"t\\200\\000\\000\\000\\000\\000\\001\\030_r\\200\\000\\000\\001\\351\\003\\252\\246\" lock_ttl:29588 txn_size:1 lock_type:Lock lock_for_update_ts:454044832182042663 use_async_commit:true min_commit_ts:454044837044551774 > > exec_details_v2:<time_detail:<total_rpc_wall_time_ns:59238 > time_detail_v2:<total_rpc_wall_time_ns:59238 > > \ngithub.com/pingcap/tiflow/pkg/txnutil.(*resolver).Resolve\n\tgithub.com/pingcap/tiflow/pkg/txnutil/lock_resolver.go:126\ngithub.com/pingcap/tiflow/cdc/kv.(*SharedClient).resolveLock.func2\n\tgithub.com/pingcap/tiflow/cdc/kv/shared_client.go:674\ngithub.com/pingcap/tiflow/cdc/kv.(*SharedClient).resolveLock\n\tgithub.com/pingcap/tiflow/cdc/kv/shared_client.go:695\ngithub.com/pingcap/tiflow/cdc/kv.(*SharedClient).Run.func6\n\tgithub.com/pingcap/tiflow/cdc/kv/shared_client.go:251\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.7.0/errgroup/errgroup.go:78\nruntime.goexit\n\truntime/asm_amd64.s:1650"]
[2024/11/20 03:13:18.387 +08:00] [INFO] [lock_resolver.go:63] ["resolve lock finishes"] [regionID=20300376] [lockCount=0] [locks=null] [maxVersion=454044835393306624] [namespace=default] [changefeed=dgdg-test] [duration=575.506µs] [error="unexpected scanlock error: error:<locked:<primary_lock:\"t\\200\\000\\000\\000\\000\\000\\001\\030_i\\200\\000\\000\\000\\000\\000\\000\\001\\0010d5f78c2\\377750c846f\\3775bbac63a\\377cf418037\\377\\000\\000\\000\\000\\000\\000\\000\\000\\367\" lock_version:454044832182042663 key:\"t\\200\\000\\000\\000\\000\\000\\001\\030_r\\200\\000\\000\\001\\351\\003\\252\\246\" lock_ttl:29588 txn_size:1 lock_type:Lock lock_for_update_ts:454044832182042663 use_async_commit:true min_commit_ts:454044837044551774 > > exec_details_v2:<time_detail:<total_rpc_wall_time_ns:44319 > time_detail_v2:<total_rpc_wall_time_ns:44319 > > "] [errorVerbose="unexpected scanlock error: error:<locked:<primary_lock:\"t\\200\\000\\000\\000\\000\\000\\001\\030_i\\200\\000\\000\\000\\000\\000\\000\\001\\0010d5f78c2\\377750c846f\\3775bbac63a\\377cf418037\\377\\000\\000\\000\\000\\000\\000\\000\\000\\367\" lock_version:454044832182042663 key:\"t\\200\\000\\000\\000\\000\\000\\001\\030_r\\200\\000\\000\\001\\351\\003\\252\\246\" lock_ttl:29588 txn_size:1 lock_type:Lock lock_for_update_ts:454044832182042663 use_async_commit:true min_commit_ts:454044837044551774 > > exec_details_v2:<time_detail:<total_rpc_wall_time_ns:44319 > time_detail_v2:<total_rpc_wall_time_ns:44319 > > \ngithub.com/pingcap/tiflow/pkg/txnutil.(*resolver).Resolve\n\tgithub.com/pingcap/tiflow/pkg/txnutil/lock_resolver.go:126\ngithub.com/pingcap/tiflow/cdc/kv.(*SharedClient).resolveLock.func2\n\tgithub.com/pingcap/tiflow/cdc/kv/shared_client.go:674\ngithub.com/pingcap/tiflow/cdc/kv.(*SharedClient).resolveLock\n\tgithub.com/pingcap/tiflow/cdc/kv/shared_client.go:695\ngithub.com/pingcap/tiflow/cdc/kv.(*SharedClient).Run.func6\n\tgithub.com/pingcap/tiflow/cdc/kv/shared_client.go:251\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.7.0/errgroup/errgroup.go:78\nruntime.goexit\n\truntime/asm_amd64.s:1650"]
[2024/11/20 03:13:18.387 +08:00] [WARN] [shared_client.go:675] ["event feed resolve lock fail"] [namespace=default] [changefeed=dgdg-test] [regionID=20300376] [error="unexpected scanlock error: error:<locked:<primary_lock:\"t\\200\\000\\000\\000\\000\\000\\001\\030_i\\200\\000\\000\\000\\000\\000\\000\\001\\0010d5f78c2\\377750c846f\\3775bbac63a\\377cf418037\\377\\000\\000\\000\\000\\000\\000\\000\\000\\367\" lock_version:454044832182042663 key:\"t\\200\\000\\000\\000\\000\\000\\001\\030_r\\200\\000\\000\\001\\351\\003\\252\\246\" lock_ttl:29588 txn_size:1 lock_type:Lock lock_for_update_ts:454044832182042663 use_async_commit:true min_commit_ts:454044837044551774 > > exec_details_v2:<time_detail:<total_rpc_wall_time_ns:44319 > time_detail_v2:<total_rpc_wall_time_ns:44319 > > "] [errorVerbose="unexpected scanlock error: error:<locked:<primary_lock:\"t\\200\\000\\000\\000\\000\\000\\001\\030_i\\200\\000\\000\\000\\000\\000\\000\\001\\0010d5f78c2\\377750c846f\\3775bbac63a\\377cf418037\\377\\000\\000\\000\\000\\000\\000\\000\\000\\367\" lock_version:454044832182042663 key:\"t\\200\\000\\000\\000\\000\\000\\001\\030_r\\200\\000\\000\\001\\351\\003\\252\\246\" lock_ttl:29588 txn_size:1 lock_type:Lock lock_for_update_ts:454044832182042663 use_async_commit:true min_commit_ts:454044837044551774 > > exec_details_v2:<time_detail:<total_rpc_wall_time_ns:44319 > time_detail_v2:<total_rpc_wall_time_ns:44319 > > \ngithub.com/pingcap/tiflow/pkg/txnutil.(*resolver).Resolve\n\tgithub.com/pingcap/tiflow/pkg/txnutil/lock_resolver.go:126\ngithub.com/pingcap/tiflow/cdc/kv.(*SharedClient).resolveLock.func2\n\tgithub.com/pingcap/tiflow/cdc/kv/shared_client.go:674\ngithub.com/pingcap/tiflow/cdc/kv.(*SharedClient).resolveLock\n\tgithub.com/pingcap/tiflow/cdc/kv/shared_client.go:695\ngithub.com/pingcap/tiflow/cdc/kv.(*SharedClient).Run.func6\n\tgithub.com/pingcap/tiflow/cdc/kv/shared_client.go:251\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.7.0/errgroup/errgroup.go:78\nruntime.goexit\n\truntime/asm_amd64.s:1650"]
[2024/11/20 03:13:18.387 +08:00] [INFO] [lock_resolver.go:63] ["resolve lock finishes"] [regionID=20214416] [lockCount=0] [locks=null] [maxVersion=454044835393306624] [namespace=default] [changefeed=dgdg-test] [duration=437.632µs] [error="unexpected scanlock error: error:<locked:<primary_lock:\"t\\200\\000\\000\\000\\000\\000\\001\\030_i\\200\\000\\000\\000\\000\\000\\000\\001\\0010d5f78c2\\377750c846f\\3775bbac63a\\377cf418037\\377\\000\\000\\000\\000\\000\\000\\000\\000\\367\" lock_version:454044832182042663 key:\"t\\200\\000\\000\\000\\000\\000\\001\\030_r\\200\\000\\000\\001\\351\\003\\252\\246\" lock_ttl:29588 txn_size:1 lock_type:Lock lock_for_update_ts:454044832182042663 use_async_commit:true min_commit_ts:454044837044551774 > > exec_details_v2:<time_detail:<total_rpc_wall_time_ns:35670 > time_detail_v2:<total_rpc_wall_time_ns:35670 > > "] [errorVerbose="unexpected scanlock error: error:<locked:<primary_lock:\"t\\200\\000\\000\\000\\000\\000\\001\\030_i\\200\\000\\000\\000\\000\\000\\000\\001\\0010d5f78c2\\377750c846f\\3775bbac63a\\377cf418037\\377\\000\\000\\000\\000\\000\\000\\000\\000\\367\" lock_version:454044832182042663 key:\"t\\200\\000\\000\\000\\000\\000\\001\\030_r\\200\\000\\000\\001\\351\\003\\252\\246\" lock_ttl:29588 txn_size:1 lock_type:Lock lock_for_update_ts:454044832182042663 use_async_commit:true min_commit_ts:454044837044551774 > > exec_details_v2:<time_detail:<total_rpc_wall_time_ns:35670 > time_detail_v2:<total_rpc_wall_time_ns:35670 > > \ngithub.com/pingcap/tiflow/pkg/txnutil.(*resolver).Resolve\n\tgithub.com/pingcap/tiflow/pkg/txnutil/lock_resolver.go:126\ngithub.com/pingcap/tiflow/cdc/kv.(*SharedClient).resolveLock.func2\n\tgithub.com/pingcap/tiflow/cdc/kv/shared_client.go:674\ngithub.com/pingcap/tiflow/cdc/kv.(*SharedClient).resolveLock\n\tgithub.com/pingcap/tiflow/cdc/kv/shared_client.go:695\ngithub.com/pingcap/tiflow/cdc/kv.(*SharedClient).Run.func6\n\tgithub.com/pingcap/tiflow/cdc/kv/shared_client.go:251\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.7.0/errgroup/errgroup.go:78\nruntime.goexit\n\truntime/asm_amd64.s:1650"]
[2024/11/20 03:13:18.387 +08:00] [WARN] [shared_client.go:675] ["event feed resolve lock fail"] [namespace=default] [changefeed=dgdg-test] [regionID=20214416] [error="unexpected scanlock error: error:<locked:<primary_lock:\"t\\200\\000\\000\\000\\000\\000\\001\\030_i\\200\\000\\000\\000\\000\\000\\000\\001\\0010d5f78c2\\377750c846f\\3775bbac63a\\377cf418037\\377\\000\\000\\000\\000\\000\\000\\000\\000\\367\" lock_version:454044832182042663 key:\"t\\200\\000\\000\\000\\000\\000\\001\\030_r\\200\\000\\000\\001\\351\\003\\252\\246\" lock_ttl:29588 txn_size:1 lock_type:Lock lock_for_update_ts:454044832182042663 use_async_commit:true min_commit_ts:454044837044551774 > > exec_details_v2:<time_detail:<total_rpc_wall_time_ns:35670 > time_detail_v2:<total_rpc_wall_time_ns:35670 > > "] [errorVerbose="unexpected scanlock error: error:<locked:<primary_lock:\"t\\200\\000\\000\\000\\000\\000\\001\\030_i\\200\\000\\000\\000\\000\\000\\000\\001\\0010d5f78c2\\377750c846f\\3775bbac63a\\377cf418037\\377\\000\\000\\000\\000\\000\\000\\000\\000\\367\" lock_version:454044832182042663 key:\"t\\200\\000\\000\\000\\000\\000\\001\\030_r\\200\\000\\000\\001\\351\\003\\252\\246\" lock_ttl:29588 txn_size:1 lock_type:Lock lock_for_update_ts:454044832182042663 use_async_commit:true min_commit_ts:454044837044551774 > > exec_details_v2:<time_detail:<total_rpc_wall_time_ns:35670 > time_detail_v2:<total_rpc_wall_time_ns:35670 > > \ngithub.com/pingcap/tiflow/pkg/txnutil.(*resolver).Resolve\n\tgithub.com/pingcap/tiflow/pkg/txnutil/lock_resolver.go:126\ngithub.com/pingcap/tiflow/cdc/kv.(*SharedClient).resolveLock.func2\n\tgithub.com/pingcap/tiflow/cdc/kv/shared_client.go:674\ngithub.com/pingcap/tiflow/cdc/kv.(*SharedClient).resolveLock\n\tgithub.com/pingcap/tiflow/cdc/kv/shared_client.go:695\ngithub.com/pingcap/tiflow/cdc/kv.(*SharedClient).Run.func6\n\tgithub.com/pingcap/tiflow/cdc/kv/shared_client.go:251\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.7.0/errgroup/errgroup.go:78\nruntime.goexit\n\truntime/asm_amd64.s:1650"]
[2024/11/20 03:13:18.408 +08:00] [INFO] [lock_resolver.go:63] ["resolve lock finishes"] [regionID=20233210] [lockCount=0] [locks=null] [maxVersion=454044835393306624] [namespace=default] [changefeed=dgdg-test] [duration=20.690481ms] [error="unexpected scanlock error: error:<locked:<primary_lock:\"t\\200\\000\\000\\000\\000\\000\\001\\030_i\\200\\000\\000\\000\\000\\000\\000\\001\\0010d5f78c2\\377750c846f\\3775bbac63a\\377cf418037\\377\\000\\000\\000\\000\\000\\000\\000\\000\\367\" lock_version:454044832182042663 key:\"t\\200\\000\\000\\000\\000\\000\\001\\030_r\\200\\000\\000\\001\\351\\003\\252\\246\" lock_ttl:29588 txn_size:1 lock_type:Lock lock_for_update_ts:454044832182042663 use_async_commit:true min_commit_ts:454044837044551774 > > exec_details_v2:<time_detail:<total_rpc_wall_time_ns:3638660 > time_detail_v2:<total_rpc_wall_time_ns:3638660 > > "] [errorVerbose="unexpected scanlock error: error:<locked:<primary_lock:\"t\\200\\000\\000\\000\\000\\000\\001\\030_i\\200\\000\\000\\000\\000\\000\\000\\001\\0010d5f78c2\\377750c846f\\3775bbac63a\\377cf418037\\377\\000\\000\\000\\000\\000\\000\\000\\000\\367\" lock_version:454044832182042663 key:\"t\\200\\000\\000\\000\\000\\000\\001\\030_r\\200\\000\\000\\001\\351\\003\\252\\246\" lock_ttl:29588 txn_size:1 lock_type:Lock lock_for_update_ts:454044832182042663 use_async_commit:true min_commit_ts:454044837044551774 > > exec_details_v2:<time_detail:<total_rpc_wall_time_ns:3638660 > time_detail_v2:<total_rpc_wall_time_ns:3638660 > > \ngithub.com/pingcap/tiflow/pkg/txnutil.(*resolver).Resolve\n\tgithub.com/pingcap/tiflow/pkg/txnutil/lock_resolver.go:126\ngithub.com/pingcap/tiflow/cdc/kv.(*SharedClient).resolveLock.func2\n\tgithub.com/pingcap/tiflow/cdc/kv/shared_client.go:674\ngithub.com/pingcap/tiflow/cdc/kv.(*SharedClient).resolveLock\n\tgithub.com/pingcap/tiflow/cdc/kv/shared_client.go:695\ngithub.com/pingcap/tiflow/cdc/kv.(*SharedClient).Run.func6\n\tgithub.com/pingcap/tiflow/cdc/kv/shared_client.go:251\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.7.0/errgroup/errgroup.go:78\nruntime.goexit\n\truntime/asm_amd64.s:1650"]
[2024/11/20 03:13:18.408 +08:00] [WARN] [shared_client.go:675] ["event feed resolve lock fail"] [namespace=default] [changefeed=dgdg-test] [regionID=20233210] [error="unexpected scanlock error: error:<locked:<primary_lock:\"t\\200\\000\\000\\000\\000\\000\\001\\030_i\\200\\000\\000\\000\\000\\000\\000\\001\\0010d5f78c2\\377750c846f\\3775bbac63a\\377cf418037\\377\\000\\000\\000\\000\\000\\000\\000\\000\\367\" lock_version:454044832182042663 key:\"t\\200\\000\\000\\000\\000\\000\\001\\030_r\\200\\000\\000\\001\\351\\003\\252\\246\" lock_ttl:29588 txn_size:1 lock_type:Lock lock_for_update_ts:454044832182042663 use_async_commit:true min_commit_ts:454044837044551774 > > exec_details_v2:<time_detail:<total_rpc_wall_time_ns:3638660 > time_detail_v2:<total_rpc_wall_time_ns:3638660 > > "] [errorVerbose="unexpected scanlock error: error:<locked:<primary_lock:\"t\\200\\000\\000\\000\\000\\000\\001\\030_i\\200\\000\\000\\000\\000\\000\\000\\001\\0010d5f78c2\\377750c846f\\3775bbac63a\\377cf418037\\377\\000\\000\\000\\000\\000\\000\\000\\000\\367\" lock_version:454044832182042663 key:\"t\\200\\000\\000\\000\\000\\000\\001\\030_r\\200\\000\\000\\001\\351\\003\\252\\246\" lock_ttl:29588 txn_size:1 lock_type:Lock lock_for_update_ts:454044832182042663 use_async_commit:true min_commit_ts:454044837044551774 > > exec_details_v2:<time_detail:<total_rpc_wall_time_ns:3638660 > time_detail_v2:<total_rpc_wall_time_ns:3638660 > > \ngithub.com/pingcap/tiflow/pkg/txnutil.(*resolver).Resolve\n\tgithub.com/pingcap/tiflow/pkg/txnutil/lock_resolver.go:126\ngithub.com/pingcap/tiflow/cdc/kv.(*SharedClient).resolveLock.func2\n\tgithub.com/pingcap/tiflow/cdc/kv/shared_client.go:674\ngithub.com/pingcap/tiflow/cdc/kv.(*SharedClient).resolveLock\n\tgithub.com/pingcap/tiflow/cdc/kv/shared_client.go:695\ngithub.com/pingcap/tiflow/cdc/kv.(*SharedClient).Run.func6\n\tgithub.com/pingcap/tiflow/cdc/kv/shared_client.go:251\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.7.0/errgroup/errgroup.go:78\nruntime.goexit\n\truntime/asm_amd64.s:1650"]
[2024/11/20 03:13:18.411 +08:00] [INFO] [lock_resolver.go:63] ["resolve lock finishes"] [regionID=20287400] [lockCount=0] [locks=null] [maxVersion=454044835393306624] [namespace=default] [changefeed=dgdg-test] [duration=3.022563ms] [error="unexpected scanlock error: error:<locked:<primary_lock:\"t\\200\\000\\000\\000\\000\\000\\001\\030_i\\200\\000\\000\\000\\000\\000\\000\\001\\0010d5f78c2\\377750c846f\\3775bbac63a\\377cf418037\\377\\000\\000\\000\\000\\000\\000\\000\\000\\367\" lock_version:454044832182042663 key:\"t\\200\\000\\000\\000\\000\\000\\001\\030_r\\200\\000\\000\\001\\351\\003\\252\\246\" lock_ttl:29588 txn_size:1 lock_type:Lock lock_for_update_ts:454044832182042663 use_async_commit:true min_commit_ts:454044837044551774 > > exec_details_v2:<time_detail:<total_rpc_wall_time_ns:217106 > time_detail_v2:<total_rpc_wall_time_ns:217106 > > "] [errorVerbose="unexpected scanlock error: error:<locked:<primary_lock:\"t\\200\\000\\000\\000\\000\\000\\001\\030_i\\200\\000\\000\\000\\000\\000\\000\\001\\0010d5f78c2\\377750c846f\\3775bbac63a\\377cf418037\\377\\000\\000\\000\\000\\000\\000\\000\\000\\367\" lock_version:454044832182042663 key:\"t\\200\\000\\000\\000\\000\\000\\001\\030_r\\200\\000\\000\\001\\351\\003\\252\\246\" lock_ttl:29588 txn_size:1 lock_type:Lock lock_for_update_ts:454044832182042663 use_async_commit:true min_commit_ts:454044837044551774 > > exec_details_v2:<time_detail:<total_rpc_wall_time_ns:217106 > time_detail_v2:<total_rpc_wall_time_ns:217106 > > \ngithub.com/pingcap/tiflow/pkg/txnutil.(*resolver).Resolve\n\tgithub.com/pingcap/tiflow/pkg/txnutil/lock_resolver.go:126\ngithub.com/pingcap/tiflow/cdc/kv.(*SharedClient).resolveLock.func2\n\tgithub.com/pingcap/tiflow/cdc/kv/shared_client.go:674\ngithub.com/pingcap/tiflow/cdc/kv.(*SharedClient).resolveLock\n\tgithub.com/pingcap/tiflow/cdc/kv/shared_client.go:695\ngithub.com/pingcap/tiflow/cdc/kv.(*SharedClient).Run.func6\n\tgithub.com/pingcap/tiflow/cdc/kv/shared_client.go:251\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.7.0/errgroup/errgroup.go:78\nruntime.goexit\n\truntime/asm_amd64.s:1650"]
[2024/11/20 03:13:18.411 +08:00] [WARN] [shared_client.go:675] ["event feed resolve lock fail"] [namespace=default] [changefeed=dgdg-test] [regionID=20287400] [error="unexpected scanlock error: error:<locked:<primary_lock:\"t\\200\\000\\000\\000\\000\\000\\001\\030_i\\200\\000\\000\\000\\000\\000\\000\\001\\0010d5f78c2\\377750c846f\\3775bbac63a\\377cf418037\\377\\000\\000\\000\\000\\000\\000\\000\\000\\367\" lock_version:454044832182042663 key:\"t\\200\\000\\000\\000\\000\\000\\001\\030_r\\200\\000\\000\\001\\351\\003\\252\\246\" lock_ttl:29588 txn_size:1 lock_type:Lock lock_for_update_ts:454044832182042663 use_async_commit:true min_commit_ts:454044837044551774 > > exec_details_v2:<time_detail:<total_rpc_wall_time_ns:217106 > time_detail_v2:<total_rpc_wall_time_ns:217106 > > "] [errorVerbose="unexpected scanlock error: error:<locked:<primary_lock:\"t\\200\\000\\000\\000\\000\\000\\001\\030_i\\200\\000\\000\\000\\000\\000\\000\\001\\0010d5f78c2\\377750c846f\\3775bbac63a\\377cf418037\\377\\000\\000\\000\\000\\000\\000\\000\\000\\367\" lock_version:454044832182042663 key:\"t\\200\\000\\000\\000\\000\\000\\001\\030_r\\200\\000\\000\\001\\351\\003\\252\\246\" lock_ttl:29588 txn_size:1 lock_type:Lock lock_for_update_ts:454044832182042663 use_async_commit:true min_commit_ts:454044837044551774 > > exec_details_v2:<time_detail:<total_rpc_wall_time_ns:217106 > time_detail_v2:<total_rpc_wall_time_ns:217106 > > \ngithub.com/pingcap/tiflow/pkg/txnutil.(*resolver).Resolve\n\tgithub.com/pingcap/tiflow/pkg/txnutil/lock_resolver.go:126\ngithub.com/pingcap/tiflow/cdc/kv.(*SharedClient).resolveLock.func2\n\tgithub.com/pingcap/tiflow/cdc/kv/shared_client.go:674\ngithub.com/pingcap/tiflow/cdc/kv.(*SharedClient).resolveLock\n\tgithub.com/pingcap/tiflow/cdc/kv/shared_client.go:695\ngithub.com/pingcap/tiflow/cdc/kv.(*SharedClient).Run.func6\n\tgithub.com/pingcap/tiflow/cdc/kv/shared_client.go:251\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.7.0/errgroup/errgroup.go:78\nruntime.goexit\n\truntime/asm_amd64.s:1650"]
[2024/11/20 03:13:18.412 +08:00] [INFO] [lock_resolver.go:63] ["resolve lock finishes"] [regionID=20289343] [lockCount=0] [locks=null] [maxVersion=454044835393306624] [namespace=default] [changefeed=dgdg-test] [duration=911.452µs] [error="unexpected scanlock error: error:<locked:<primary_lock:\"t\\200\\000\\000\\000\\000\\000\\001\\030_i\\200\\000\\000\\000\\000\\000\\000\\001\\0010d5f78c2\\377750c846f\\3775bbac63a\\377cf418037\\377\\000\\000\\000\\000\\000\\000\\000\\000\\367\" lock_version:454044832182042663 key:\"t\\200\\000\\000\\000\\000\\000\\001\\030_r\\200\\000\\000\\001\\351\\003\\252\\246\" lock_ttl:29588 txn_size:1 lock_type:Lock lock_for_update_ts:454044832182042663 use_async_commit:true min_commit_ts:454044837044551774 > > exec_details_v2:<time_detail:<total_rpc_wall_time_ns:29529 > time_detail_v2:<total_rpc_wall_time_ns:29529 > > "] [errorVerbose="unexpected scanlock error: error:<locked:<primary_lock:\"t\\200\\000\\000\\000\\000\\000\\001\\030_i\\200\\000\\000\\000\\000\\000\\000\\001\\0010d5f78c2\\377750c846f\\3775bbac63a\\377cf418037\\377\\000\\000\\000\\000\\000\\000\\000\\000\\367\" lock_version:454044832182042663 key:\"t\\200\\000\\000\\000\\000\\000\\001\\030_r\\200\\000\\000\\001\\351\\003\\252\\246\" lock_ttl:29588 txn_size:1 lock_type:Lock lock_for_update_ts:454044832182042663 use_async_commit:true min_commit_ts:454044837044551774 > > exec_details_v2:<time_detail:<total_rpc_wall_time_ns:29529 > time_detail_v2:<total_rpc_wall_time_ns:29529 > > \ngithub.com/pingcap/tiflow/pkg/txnutil.(*resolver).Resolve\n\tgithub.com/pingcap/tiflow/pkg/txnutil/lock_resolver.go:126\ngithub.com/pingcap/tiflow/cdc/kv.(*SharedClient).resolveLock.func2\n\tgithub.com/pingcap/tiflow/cdc/kv/shared_client.go:674\ngithub.com/pingcap/tiflow/cdc/kv.(*SharedClient).resolveLock\n\tgithub.com/pingcap/tiflow/cdc/kv/shared_client.go:695\ngithub.com/pingcap/tiflow/cdc/kv.(*SharedClient).Run.func6\n\tgithub.com/pingcap/tiflow/cdc/kv/shared_client.go:251\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.7.0/errgroup/errgroup.go:78\nruntime.goexit\n\truntime/asm_amd64.s:1650"]
[2024/11/20 03:13:18.412 +08:00] [WARN] [shared_client.go:675] ["event feed resolve lock fail"] [namespace=default] [changefeed=dgdg-test] [regionID=20289343] [error="unexpected scanlock error: error:<locked:<primary_lock:\"t\\200\\000\\000\\000\\000\\000\\001\\030_i\\200\\000\\000\\000\\000\\000\\000\\001\\0010d5f78c2\\377750c846f\\3775bbac63a\\377cf418037\\377\\000\\000\\000\\000\\000\\000\\000\\000\\367\" lock_version:454044832182042663 key:\"t\\200\\000\\000\\000\\000\\000\\001\\030_r\\200\\000\\000\\001\\351\\003\\252\\246\" lock_ttl:29588 txn_size:1 lock_type:Lock lock_for_update_ts:454044832182042663 use_async_commit:true min_commit_ts:454044837044551774 > > exec_details_v2:<time_detail:<total_rpc_wall_time_ns:29529 > time_detail_v2:<total_rpc_wall_time_ns:29529 > > "] [errorVerbose="unexpected scanlock error: error:<locked:<primary_lock:\"t\\200\\000\\000\\000\\000\\000\\001\\030_i\\200\\000\\000\\000\\000\\000\\000\\001\\0010d5f78c2\\377750c846f\\3775bbac63a\\377cf418037\\377\\000\\000\\000\\000\\000\\000\\000\\000\\367\" lock_version:454044832182042663 key:\"t\\200\\000\\000\\000\\000\\000\\001\\030_r\\200\\000\\000\\001\\351\\003\\252\\246\" lock_ttl:29588 txn_size:1 lock_type:Lock lock_for_update_ts:454044832182042663 use_async_commit:true min_commit_ts:454044837044551774 > > exec_details_v2:<time_detail:<total_rpc_wall_time_ns:29529 > time_detail_v2:<total_rpc_wall_time_ns:29529 > > \ngithub.com/pingcap/tiflow/pkg/txnutil.(*resolver).Resolve\n\tgithub.com/pingcap/tiflow/pkg/txnutil/lock_resolver.go:126\ngithub.com/pingcap/tiflow/cdc/kv.(*SharedClient).resolveLock.func2\n\tgithub.com/pingcap/tiflow/cdc/kv/shared_client.go:674\ngithub.com/pingcap/tiflow/cdc/kv.(*SharedClient).resolveLock\n\tgithub.com/pingcap/tiflow/cdc/kv/shared_client.go:695\ngithub.com/pingcap/tiflow/cdc/kv.(*SharedClient).Run.func6\n\tgithub.com/pingcap/tiflow/cdc/kv/shared_client.go:251\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.7.0/errgroup/errgroup.go:78\nruntime.goexit\n\truntime/asm_amd64.s:1650"]
[2024/11/20 03:13:18.429 +08:00] [INFO] [lock_resolver.go:63] ["resolve lock finishes"] [regionID=33574171] [lockCount=0] [locks=null] [maxVersion=454044835393306624] [namespace=default] [changefeed=dgdg-test] [duration=16.276187ms] [error="unexpected scanlock error: error:<locked:<primary_lock:\"t\\200\\000\\000\\000\\000\\000\\001\\030_i\\200\\000\\000\\000\\000\\000\\000\\001\\0010d5f78c2\\377750c846f\\3775bbac63a\\377cf418037\\377\\000\\000\\000\\000\\000\\000\\000\\000\\367\" lock_version:454044832182042663 key:\"t\\200\\000\\000\\000\\000\\000\\001\\030_r\\200\\000\\000\\001\\351\\003\\252\\246\" lock_ttl:29588 txn_size:1 lock_type:Lock lock_for_update_ts:454044832182042663 use_async_commit:true min_commit_ts:454044837044551774 > > exec_details_v2:<time_detail:<total_rpc_wall_time_ns:7133691 > time_detail_v2:<total_rpc_wall_time_ns:7133691 > > "] [errorVerbose="unexpected scanlock error: error:<locked:<primary_lock:\"t\\200\\000\\000\\000\\000\\000\\001\\030_i\\200\\000\\000\\000\\000\\000\\000\\001\\0010d5f78c2\\377750c846f\\3775bbac63a\\377cf418037\\377\\000\\000\\000\\000\\000\\000\\000\\000\\367\" lock_version:454044832182042663 key:\"t\\200\\000\\000\\000\\000\\000\\001\\030_r\\200\\000\\000\\001\\351\\003\\252\\246\" lock_ttl:29588 txn_size:1 lock_type:Lock lock_for_update_ts:454044832182042663 use_async_commit:true min_commit_ts:454044837044551774 > > exec_details_v2:<time_detail:<total_rpc_wall_time_ns:7133691 > time_detail_v2:<total_rpc_wall_time_ns:7133691 > > \ngithub.com/pingcap/tiflow/pkg/txnutil.(*resolver).Resolve\n\tgithub.com/pingcap/tiflow/pkg/txnutil/lock_resolver.go:126\ngithub.com/pingcap/tiflow/cdc/kv.(*SharedClient).resolveLock.func2\n\tgithub.com/pingcap/tiflow/cdc/kv/shared_client.go:674\ngithub.com/pingcap/tiflow/cdc/kv.(*SharedClient).resolveLock\n\tgithub.com/pingcap/tiflow/cdc/kv/shared_client.go:695\ngithub.com/pingcap/tiflow/cdc/kv.(*SharedClient).Run.func6\n\tgithub.com/pingcap/tiflow/cdc/kv/shared_client.go:251\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.7.0/errgroup/errgroup.go:78\nruntime.goexit\n\truntime/asm_amd64.s:1650"]

上面的warn就是锁等待,看看相同时间tikv上有类似的锁吗

看下监控问题时段 Grafana 的 TiDB=》Transiaction=》Duration 面板,把监控表达式中的 0.99 调整为 1


这样异常不

有长事务?是长时间不提交还是大事务?这个应该是会影响 cdc resolved ts 推进的,从而影响同步进度

1 个赞

有什么快速查询的命令不

有个系统视图查询当前活跃事务

SELECT * FROM INFORMATION_SCHEMA.TIDB_TRX\G