TiDB 5.3.3 TiKV节点抖动导致读写超时

【 TiDB 使用环境】生产环境
【 TiDB 版本】V5.3.3
【遇到的问题:问题现象及影响】
出现问题的时间点,集群负载正常,怀疑是锁原因导致SQL执行耗时变长。tikv节点锁冲突导致业务出现读写超时【代码中读操作与写操作存在悲观事务与乐观事务的混用的情况】,如果全部调整为显示事务的方式,是否可以解决该问题。
【资源配置】进入到 TiDB Dashboard -集群信息 (Cluster Info) -主机(Hosts) 截图此页面
【复制黏贴 ERROR 报错的日志】
TiKV一直在刷清锁的日志
[2025/02/23 12:33:56.849 +08:00] [WARN] [endpoint.rs:638] [error-response] [err=“Key is locked (will clean up) primary_lock: 7480000000000000455F698000000000000001013131303136363638FF3730323339303738FF3438000000000000F9038000001939E8DCE8 lock_version: 456205333091319950 key: 7480000000000000455F698000000000000001013131303136363638FF3730323339303738FF3438000000000000F9038000001939E8DCE8 lock_ttl: 3000 txn_size: 1 use_async_commit: true min_commit_ts: 456205333091319951 secondaries: 7480000000000000455F6980000000000000020419B5EEC8780AF3C7038000001939E8DCE8 secondaries: 7480000000000000455F728000001939E8DCE8”]
[2025/02/23 12:33:57.094 +08:00] [WARN] [endpoint.rs:638] [error-response] [err=“Key is locked (will clean up) primary_lock: 7480000000000000455F698000000000000001013131303136363638FF3938323637313938FF3239000000000000F9038000001939E71828 lock_version: 456205333156855962 key: 7480000000000000455F698000000000000001013131303136363638FF3938323637313938FF3239000000000000F9038000001939E71828 lock_ttl: 3000 txn_size: 1 use_async_commit: true min_commit_ts: 456205333156855963 secondaries: 7480000000000000455F6980000000000000020419B5EEC879017F04038000001939E71828 secondaries: 7480000000000000455F728000001939E71828”]
[2025/02/23 12:33:57.587 +08:00] [WARN] [endpoint.rs:638] [error-response] [err=“Key is locked (will clean up) primary_lock: 7480000000000000455F698000000000000001013131303136363639FF3037383931363637FF3633000000000000F9038000001939E68833 lock_version: 456205333287927882 key: 7480000000000000455F698000000000000001013131303136363639FF3037383931363637FF3633000000000000F9038000001939E68833 lock_ttl: 3000 txn_size: 1 use_async_commit: true min_commit_ts: 456205333287927883 secondaries: 7480000000000000455F6980000000000000020419B5EEC87909471E038000001939E68833 secondaries: 7480000000000000455F728000001939E68833”]
[2025/02/23 12:33:57.855 +08:00] [WARN] [endpoint.rs:638] [error-response] [err=“Key is locked (will clean up) primary_lock: 7480000000000000455F698000000000000001013131303136363638FF3938323833383730FF3435000000000000F9038000001939E11B2E lock_version: 456205333353463994 key: 7480000000000000455F698000000000000001013131303136363638FF3938323833383730FF3435000000000000F9038000001939E11B2E lock_ttl: 3000 txn_size: 1 use_async_commit: true min_commit_ts: 456205333353463997 secondaries: 7480000000000000455F6980000000000000020419B5EEC8790D5A7C038000001939E11B2E secondaries: 7480000000000000455F728000001939E11B2E”]
[2025/02/23 12:33:57.970 +08:00] [WARN] [endpoint.rs:638] [error-response] [err=“Key is locked (will clean up) primary_lock: 7480000000000000455F698000000000000001013131303136363638FF3535353038333535FF3639000000000000F9038000001939E11B33 lock_version: 456205333379678514 key: 7480000000000000455F698000000000000001013131303136363638FF3535353038333535FF3639000000000000F9038000001939E11B33 lock_ttl: 3000 txn_size: 1 use_async_commit: true min_commit_ts: 456205333379678515 secondaries: 7480000000000000455F6980000000000000020419B5EEC8790CCBA3038000001939E11B33 secondaries: 7480000000000000455F728000001939E11B33”]
[2025/02/23 12:33:58.076 +08:00] [WARN] [endpoint.rs:638] [error-response] [err=“Key is locked (will clean up) primary_lock: 7480000000000000455F698000000000000001013131303136363638FF3730323531313837FF3239000000000000F9038000001939E2E6C2 lock_version: 456205333405893009 key: 7480000000000000455F698000000000000001013131303136363638FF3730323531313837FF3239000000000000F9038000001939E2E6C2 lock_ttl: 3000 txn_size: 1 use_async_commit: true min_commit_ts: 456205333405893010 secondaries: 7480000000000000455F6980000000000000020419B5EEC8790E70BD038000001939E2E6C2 secondaries: 7480000000000000455F728000001939E2E6C2”]
[2025/02/23 12:33:58.085 +08:00] [WARN] [endpoint.rs:638] [error-response] [err=“Key is locked (will clean up) primary_lock: 7480000000000000455F698000000000000001013131303136363638FF3730323237393336FF3538000000000000F9038000001939E11B36 lock_version: 456205333418999878 key: 7480000000000000455F698000000000000001013131303136363638FF3730323237393336FF3538000000000000F9038000001939E11B36 lock_ttl: 3000 txn_size: 1 use_async_commit: true min_commit_ts: 456205333418999879 secondaries: 7480000000000000455F6980000000000000020419B5EEC8790E93D7038000001939E11B36 secondaries: 7480000000000000455F728000001939E11B36”]
[2025/02/23 12:33:58.100 +08:00] [WARN] [endpoint.rs:638] [error-response] [err=“Key is locked (will clean up) primary_lock: 7480000000000000455F698000000000000001013131303136363638FF3938323536363438FF3032000000000000F9038000001939E2E6C3 lock_version: 456205333418999961 key: 7480000000000000455F698000000000000001013131303136363638FF3938323536363438FF3032000000000000F9038000001939E2E6C3 lock_ttl: 3000 txn_size: 1 use_async_commit: true min_commit_ts: 456205333418999962 secondaries: 7480000000000000455F6980000000000000020419B5EEC87A01D074038000001939E2E6C3 secondaries: 7480000000000000455F728000001939E2E6C3”]
[2025/02/23 12:33:58.816 +08:00] [WARN] [endpoint.rs:638] [error-response] [err=“Key is locked (will clean up) primary_lock: 7480000000000000455F698000000000000001013131303136363639FF3037383934333738FF3336000000000000F9038000001939E8DD15 lock_version: 456205333602500839 key: 7480000000000000455F698000000000000001013131303136363639FF3037383934333738FF3336000000000000F9038000001939E8DD15 lock_ttl: 3000 txn_size: 1 use_async_commit: true min_commit_ts: 456205333602500840 secondaries: 7480000000000000455F6980000000000000020419B5EEC87A0CBE16038000001939E8DD15 secondaries: 7480000000000000455F728000001939E8DD15”]
[2025/02/23 12:33:58.887 +08:00] [WARN] [endpoint.rs:638] [error-response] [err=“Key is locked (will clean up) primary_lock: 7480000000000000455F698000000000000001013131303136363639FF3037373936323035FF3830000000000000F9038000001939E11B48 lock_version: 456205333628715085 key: 7480000000000000455F698000000000000001013131303136363639FF3037373936323035FF3830000000000000F9038000001939E11B48 lock_ttl: 3000 txn_size: 1 use_async_commit: true min_commit_ts: 456205333628715086 secondaries: 7480000000000000455F6980000000000000020419B5EEC87A0B88DB038000001939E11B48 secondaries: 7480000000000000455F728000001939E11B48”]
[2025/02/23 12:33:59.090 +08:00] [WARN] [endpoint.rs:638] [error-response] [err=“Key is locked (will clean up) primary_lock: 7480000000000000455F698000000000000001013131303136363638FF3938323735333737FF3139000000000000F9038000001939E2E6D9 lock_version: 456205333681143886 key: 7480000000000000455F698000000000000001013131303136363638FF3938323735333737FF3139000000000000F9038000001939E2E6D9 lock_ttl: 3000 txn_size: 1 use_async_commit: true min_commit_ts: 456205333681143887 secondaries: 7480000000000000455F6980000000000000020419B5EEC87B016F57038000001939E2E6D9 secondaries: 7480000000000000455F728000001939E2E6D9”]
[2025/02/23 12:33:59.184 +08:00] [WARN] [endpoint.rs:638] [error-response] [err=“Key is locked (will clean up) primary_lock: 7480000000000000455F698000000000000001013131303136363638FF3838303533353230FF3334000000000000F9038000001939E11B50 lock_version: 456205333707358255 key: 7480000000000000455F698000000000000001013131303136363638FF3838303533353230FF3334000000000000F9038000001939E11B50 lock_ttl: 3000 txn_size: 1 use_async_commit: true min_commit_ts: 456205333707358256 secondaries: 7480000000000000455F6980000000000000020419B5EEC87B00D062038000001939E11B50 secondaries: 7480000000000000455F728000001939E11B50”]
[2025/02/23 12:34:00.420 +08:00] [WARN] [endpoint.rs:638] [error-response] [err=“Key is locked (will clean up) primary_lock: 7480000000000000455F698000000000000001013131303136363638FF3630353332303239FF3232000000000000F9038000001939E84ED1 lock_version: 456205334021931346 key: 7480000000000000455F698000000000000001013131303136363638FF3630353332303239FF3232000000000000F9038000001939E84ED1 lock_ttl: 3000 txn_size: 1 use_async_commit: true min_commit_ts: 456205334021931347 secondaries: 7480000000000000455F6980000000000000020419B5EEC8800672E0038000001939E84ED1 secondaries: 7480000000000000455F728000001939E84ED1”]
[2025/02/23 12:34:00.509 +08:00] [WARN] [endpoint.rs:638] [error-response] [err=“Key is locked (will clean up) primary_lock: 7480000000000000455F698000000000000001013131303136363638FF3630303738323935FF3833000000000000F9038000001939E9708F lock_version: 456205334048145604 key: 7480000000000000455F698000000000000001013131303136363638FF3630303738323935FF3833000000000000F9038000001939E9708F lock_ttl: 3000 txn_size: 1 use_async_commit: true min_commit_ts: 456205334048145605 secondaries: 7480000000000000455F6980000000000000020419B5EEC88007CCC8038000001939E9708F secondaries: 7480000000000000455F728000001939E9708F”]
【其他附件:截图/日志/监控】








写冲突和事务模式,以及是不是显示事务都没关系。

本质就是你同一时间写了同一个key。这个一般都是要求应用端查一下是否有必要一定要在这个时间段内写同一个key。先排除业务处理逻辑上的bug。

如果确定实际业务的并发就是这样,就是要在同一时间写同一个key,那么现在的常见做法是把分布式锁放到数据库外,例如zookeeper/redis之类的系统上去做。而不是在db上频繁的触发写冲突,浪费数据库的资源。

2 个赞

我有遇到相关联问题解决没有

还在查,目前还没有解决,怀疑是锁的问题导致的。

https://github.com/oh-my-tidb/mok

用这个工具解析一下冲突的key,

primary_lock: 7480000000000000455F698000000000000001013131303136363638FF3630353332303239FF3232000000000000F9038000001939E84ED1

类似这种,应该很容易定位到冲突的表,缩小一下范围,应该是很容易看出来是否符合预期的。