个人猜测大致原因如下:
可能是下面原因引起的。目前该集群设置了set global tidb_replica_read=‘leader-and-follower’。
follower节点上面可能由于还存在着primary key的信息导致的。查看报错的tidb.log日志中storeAddr显示的follower的地址。
修改参数set global tidb_replica_read='leader‘,重启tidb节点之后。告警消失。
报错日志error="other error: key is locked (backoff or cleanup) 。中storeAddr=xx.xx.xx.xx显示的正好是follower的地址。
[2022/11/01 08:42:02.985 +08:00] [WARN] [coprocessor.go:914] ["other error"] [conn=8233] [txnStartTS=437063076535336968] [regionID=22127261] [storeAddr=10.15
0.xx.xx:20160] [error="other error: key is locked (backoff or cleanup) primary_lock: 7480000000000001515F698000000000000004038000000000432DF8 lock_version: 4
37063076522229789 key: 7480000000000001515F698000000000000004038000000000432DFC lock_ttl: 3008 txn_size: 2 use_async_commit: true min_commit_ts: 437063076522
229790"]
[2022/11/01 08:42:03.006 +08:00] [INFO] [conn.go:1069] ["command dispatched failed"] [conn=8233] [connInfo="id:8233, addr:10.205.xx.xx:38718 status:10, col
lation:utf8_general_ci, user:prod_xx"] [command=Query] [status="inTxn:0, autocommit:1"] [sql="SELECT MIN(CheckPoint)-1,MAX(CheckPoint) FROM po_xx_0
2 WHERE CheckPoint>4402683"] [txn_mode=PESSIMISTIC] [err="other error: key is locked (backoff or cleanup) primary_lock: 7480000000000001515F69800000000000000
4038000000000432DF8 lock_version: 437063076522229789 key: 7480000000000001515F698000000000000004038000000000432DFC lock_ttl: 3008 txn_size: 2 use_async_commi
t: true min_commit_ts: 437063076522229790\ngithub.com/pingcap/tidb/store/copr.(*copIteratorWorker).handleCopResponse\n\t/home/jenkins/agent/workspace/optimiz
ation-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/store/copr/coprocessor.go:913\ngithub.com/pingcap/tidb/store/copr.(*copIteratorWorker).handleTaskOn
ce\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/store/copr/coprocessor.go:755\ngithub.com/pingcap/tidb/s
tore/copr.(*copIteratorWorker).handleTask\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/store/copr/coproc
essor.go:645\ngithub.com/pingcap/tidb/store/copr.(*copIteratorWorker).run\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.co
m/pingcap/tidb/store/copr/coprocessor.go:382\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371"]
[2022/11/01 08:43:56.124 +08:00] [WARN] [coprocessor.go:914] ["other error"] [conn=8233] [txnStartTS=437063106183823366] [regionID=22083063] [storeAddr=10.15
0.xx.xx:20160] [error="other error: key is locked (backoff or cleanup) primary_lock: 7480000000000002585F69800000000000000203800000000024FD73 lock_version: 4
37063106183823365 key: 7480000000000002585F69800000000000000203800000000024FDA7 lock_ttl: 3003 txn_size: 3 use_async_commit: true min_commit_ts: 437063106183
823366"]
[2022/11/01 08:43:56.124 +08:00] [WARN] [coprocessor.go:914] ["other error"] [conn=8233] [txnStartTS=437063106183823366] [regionID=22083063] [storeAddr=10.15
0.xx.xx:20160] [error="other error: key is locked (backoff or cleanup) primary_lock: 7480000000000002585F69800000000000000203800000000024FD73 lock_version: 4
37063106183823365 key: 7480000000000002585F69800000000000000203800000000024FDA7 lock_ttl: 3003 txn_size: 3 use_async_commit: true min_commit_ts: 437063106183
823366"]