v7.5.7集群抖动

Hacker_Yv76YjBL · 2025 年9 月 24 日 01:30

【TiDB 使用环境】生产环境
【TiDB 版本】v7.5.7
【遇到的问题：问题现象及影响】
2025-09-23 20:59与2025-09-23 21:09，业务TPS均没有突刺，但集群出现抖动，导致DML语句执行耗时增加，查看监控，发现部分异常。

抖动的时间点正好是在做GC。查看PD并未发现异常，只有部分tikv监控存在可疑。
10.221.202.74为TiDB Server节点
根据唯一约束更新语句

UPDATE
  trade_order_seller
SET
  seller_feature = 'xxxxxxxxxxx'
WHERE
  seller_id = 170488813
  AND sub_order_no = '110182'
  AND modify_time = '2025-09-24 08:47:25.479';

执行时间：

TiKV监控

Hibernate Peers

Region average written bytes

Region average written keys

gRPC message count

Raft store CPU

Raft read proposals per server

Storage command total

TiDB GC seconds

99% Thread Pool Schedule Wait Duration

Load base split duration

Txn Scheduler Pool Wait Duration

TiDB监控

KV Request OPS

Load Region Duration

GC Duration 99

GC Push Task Duration 95

DDL job table duration

tidb server节点10.221.202.74日志

[2025/09/23 20:58:20.800 +08:00] [INFO] [coprocessor.go:1332] ["[TIME_COP_WAIT] resp_time:1.685808975s txnStartTS:461014905385910485 region_id:1895193 store_addr:10.221.202.122:20160 stats:Cop:{num_rpc:1, total_time:1.69s} kv_process_ms:0 kv_wait_ms:29 kv_read_ms:0 processed_versions:1 total_versions:1 rocksdb_delete_skipped_count:0 rocksdb_key_skipped_count:0 rocksdb_cache_hit_count:8 rocksdb_read_count:1 rocksdb_read_byte:35949"] [conn=2990378460] [session_alias=]
[2025/09/23 20:58:20.801 +08:00] [INFO] [coprocessor.go:1332] ["[TIME_COP_WAIT] resp_time:1.733018211s txnStartTS:461014905372803205 region_id:1890721 store_addr:10.221.202.122:20160 stats:Cop:{num_rpc:1, total_time:1.73s} kv_process_ms:0 kv_wait_ms:29 kv_read_ms:0 processed_versions:1 total_versions:1 rocksdb_delete_skipped_count:0 rocksdb_key_skipped_count:0 rocksdb_cache_hit_count:10 rocksdb_read_count:0 rocksdb_read_byte:0"] [conn=2990378114] [session_alias=]
[2025/09/23 20:58:20.812 +08:00] [INFO] [coprocessor.go:1332] ["[TIME_COP_WAIT] resp_time:1.462925766s txnStartTS:461014905425231998 region_id:1795363 store_addr:10.221.202.122:20160 stats:Cop:{num_rpc:1, total_time:1.46s} kv_process_ms:0 kv_wait_ms:39 kv_read_ms:0 processed_versions:1 total_versions:1 rocksdb_delete_skipped_count:0 rocksdb_key_skipped_count:0 rocksdb_cache_hit_count:10 rocksdb_read_count:0 rocksdb_read_byte:0"] [conn=2990378638] [session_alias=]
[2025/09/23 20:58:20.822 +08:00] [INFO] [coprocessor.go:1332] ["[TIME_COP_WAIT] resp_time:679.308093ms txnStartTS:461014905634947162 region_id:1834415 store_addr:10.221.202.122:20160 stats:Cop:{num_rpc:1, total_time:679.3ms} kv_process_ms:1 kv_wait_ms:39 kv_read_ms:0 processed_versions:2 total_versions:2 rocksdb_delete_skipped_count:0 rocksdb_key_skipped_count:0 rocksdb_cache_hit_count:20 rocksdb_read_count:0 rocksdb_read_byte:0"] [conn=2990378688] [session_alias=]
[2025/09/23 20:58:20.851 +08:00] [INFO] [coprocessor.go:1332] ["[TIME_COP_WAIT] resp_time:515.656873ms txnStartTS:461014905713590293 region_id:180631 store_addr:10.221.202.122:20160 stats:Cop:{num_rpc:1, total_time:515.6ms} kv_process_ms:5 kv_wait_ms:67 kv_read_ms:5 processed_versions:0 total_versions:105 rocksdb_delete_skipped_count:0 rocksdb_key_skipped_count:0 rocksdb_cache_hit_count:1259 rocksdb_read_count:1 rocksdb_read_byte:32722"] [conn=2990378644] [session_alias=]
[2025/09/23 20:58:33.522 +08:00] [INFO] [gc_worker.go:421] ["starts the whole job"] [category="gc worker"] [uuid=664cec10930098e] [safePoint=460992259919183872] [concurrency=20]
[2025/09/23 20:58:33.523 +08:00] [INFO] [gc_worker.go:1178] ["start resolve locks"] [category="gc worker"] [uuid=664cec10930098e] [safePoint=460992259919183872] [concurrency=20]
[2025/09/23 20:58:33.523 +08:00] [INFO] [range_task.go:138] ["range task started"] [name=resolve-locks-runner] [startKey=] [endKey=] [concurrency=20]
[2025/09/23 20:58:35.900 +08:00] [INFO] [coprocessor.go:1332] ["[TIME_COP_WAIT] resp_time:906.893238ms txnStartTS:461014909553999954 region_id:1471295 store_addr:10.221.202.122:20160 stats:Cop:{num_rpc:1, total_time:906.9ms} kv_process_ms:0 kv_wait_ms:1 kv_read_ms:1 processed_versions:1 total_versions:1 rocksdb_delete_skipped_count:0 rocksdb_key_skipped_count:0 rocksdb_cache_hit_count:12 rocksdb_read_count:0 rocksdb_read_byte:0"] [conn=2990378382] [session_alias=]
[2025/09/23 20:58:35.938 +08:00] [INFO] [coprocessor.go:1332] ["[TIME_COP_WAIT] resp_time:1.150199885s txnStartTS:461014909501571115 region_id:1480962 store_addr:10.221.202.122:20160 stats:Cop:{num_rpc:1, total_time:1.15s} kv_process_ms:1 kv_wait_ms:5 kv_read_ms:7 processed_versions:1 total_versions:1 rocksdb_delete_skipped_count:0 rocksdb_key_skipped_count:0 rocksdb_cache_hit_count:11 rocksdb_read_count:1 rocksdb_read_byte:11120"] [conn=2990378638] [session_alias=]
[2025/09/23 20:58:35.938 +08:00] [INFO] [coprocessor.go:1332] ["[TIME_COP_WAIT] resp_time:849.816662ms txnStartTS:461014909580214316 region_id:1472100 store_addr:10.221.202.122:20160 stats:Cop:{num_rpc:1, total_time:849.8ms} kv_process_ms:0 kv_wait_ms:5 kv_read_ms:0 processed_versions:1 total_versions:1 rocksdb_delete_skipped_count:0 rocksdb_key_skipped_count:0 rocksdb_cache_hit_count:12 rocksdb_read_count:0 rocksdb_read_byte:0"] [conn=2990378460] [session_alias=]
[2025/09/23 20:58:36.471 +08:00] [WARN] [server.go:704] ["Server.onConn handshake"] [conn=2990378796] [error="write tcp 10.221.202.74:4000->100.127.67.131:36813: write: connection reset by peer"] ["remote addr"=100.127.67.131:36813]
[2025/09/23 20:58:38.984 +08:00] [WARN] [server.go:704] ["Server.onConn handshake"] [conn=2990378798] [error="write tcp 10.221.202.74:4000->100.127.67.0:60491: write: connection reset by peer"] ["remote addr"=100.127.67.0:60491]
[2025/09/23 20:58:40.470 +08:00] [WARN] [server.go:704] ["Server.onConn handshake"] [conn=2990378800] [error="write tcp 10.221.202.74:4000->100.127.67.131:56160: write: connection reset by peer"] ["remote addr"=100.127.67.131:56160]
[2025/09/23 20:58:40.997 +08:00] [WARN] [server.go:704] ["Server.onConn handshake"] [conn=2990378802] [error="write tcp 10.221.202.74:4000->100.127.67.0:36069: write: connection reset by peer"] ["remote addr"=100.127.67.0:36069]
[2025/09/23 20:58:41.149 +08:00] [INFO] [coprocessor.go:1332] ["[TIME_COP_WAIT] resp_time:369.932453ms txnStartTS:461014911074435078 region_id:1890915 store_addr:10.221.202.122:20160 stats:Cop:{num_rpc:1, total_time:369.9ms} kv_process_ms:0 kv_wait_ms:15 kv_read_ms:0 processed_versions:1 total_versions:1 rocksdb_delete_skipped_count:0 rocksdb_key_skipped_count:0 rocksdb_cache_hit_count:11 rocksdb_read_count:0 rocksdb_read_byte:0"] [conn=2990378644] [session_alias=]
[2025/09/23 20:58:41.150 +08:00] [INFO] [coprocessor.go:1332] ["[TIME_COP_WAIT] resp_time:1.187593139s txnStartTS:461014910851612862 region_id:1472060 store_addr:10.221.202.122:20160 stats:Cop:{num_rpc:1, total_time:1.19s} kv_process_ms:0 kv_wait_ms:0 kv_read_ms:0 processed_versions:1 total_versions:1 rocksdb_delete_skipped_count:0 rocksdb_key_skipped_count:0 rocksdb_cache_hit_count:12 rocksdb_read_count:1 rocksdb_read_byte:13809"] [conn=2990378190] [session_alias=]
[2025/09/23 20:58:42.481 +08:00] [WARN] [server.go:704] ["Server.onConn handshake"] [conn=2990378804] [error="write tcp 10.221.202.74:4000->100.127.67.131:31293: write: connection reset by peer"] ["remote addr"=100.127.67.131:31293]
[2025/09/23 20:58:43.081 +08:00] [WARN] [server.go:704] ["Server.onConn handshake"] [conn=2990378806] [error="write tcp 10.221.202.74:4000->100.127.67.0:31093: write: connection reset by peer"] ["remote addr"=100.127.67.0:31093]
[2025/09/23 20:58:46.034 +08:00] [INFO] [coprocessor.go:1332] ["[TIME_COP_WAIT] resp_time:867.803892ms txnStartTS:461014912214761611 region_id:1738442 store_addr:10.221.202.122:20160 stats:Cop:{num_rpc:1, total_time:867.8ms} kv_process_ms:0 kv_wait_ms:0 kv_read_ms:0 processed_versions:1 total_versions:2 rocksdb_delete_skipped_count:0 rocksdb_key_skipped_count:1 rocksdb_cache_hit_count:10 rocksdb_read_count:0 rocksdb_read_byte:0"] [conn=2990378190] [session_alias=]
[2025/09/23 20:58:46.035 +08:00] [INFO] [coprocessor.go:1332] ["[TIME_COP_WAIT] resp_time:396.456282ms txnStartTS:461014912345833504 region_id:1709235 store_addr:10.221.202.122:20160 stats:Cop:{num_rpc:1, total_time:396.4ms} kv_process_ms:0 kv_wait_ms:0 kv_read_ms:0 processed_versions:1 total_versions:1 rocksdb_delete_skipped_count:0 rocksdb_key_skipped_count:0 rocksdb_cache_hit_count:13 rocksdb_read_count:0 rocksdb_read_byte:0"] [conn=2990378114] [session_alias=]
[2025/09/23 20:58:46.049 +08:00] [INFO] [coprocessor.go:1332] ["[TIME_COP_WAIT] resp_time:577.162127ms txnStartTS:461014912293404828 region_id:1772355 store_addr:10.221.202.122:20160 stats:Cop:{num_rpc:1, total_time:577.2ms} kv_process_ms:0 kv_wait_ms:6 kv_read_ms:0 processed_versions:1 total_versions:1 rocksdb_delete_skipped_count:0 rocksdb_key_skipped_count:0 rocksdb_cache_hit_count:10 rocksdb_read_count:1 rocksdb_read_byte:11816"] [conn=2990378568] [session_alias=]
[2025/09/23 20:58:56.725 +08:00] [INFO] [region_request.go:1764] ["send request meet region error without retry"] [conn=2990378558] [session_alias=] [req-ts=461014914980380728] [req-type=Get] [region="{ region id: 1871358, ver: 5000, confVer: 1775 }"] [replica-read-type=leader] [stale-read=false] [request-sender="{rpcError:<nil>,replicaSelector: replicaSelector{selectorStateStr: accessKnownLeader, cacheRegionIsValid: false, replicaStatus: [peer: 1875736, store: 3, isEpochStale: false, attempts: 0, attempts_time: 0s, replica-epoch: 0, store-epoch: 0, store-state: resolved, store-liveness-state: reachable peer: 1887937, store: 16, isEpochStale: false, attempts: 1, attempts_time: 1.04s, replica-epoch: 0, store-epoch: 0, store-state: resolved, store-liveness-state: reachable peer: 1895890, store: 14, isEpochStale: false, attempts: 0, attempts_time: 0s, replica-epoch: 0, store-epoch: 0, store-state: resolved, store-liveness-state: reachable]}}"] [total-round-stats="{total-backoff: 0s, total-backoff-times: 0, total-rpc: {Get:{num_rpc:1, total_time:1.04s}, rpc_errors:{region_not_found:1}}}"] [current-round-stats="{time: 1.04s, backoff: 0s, timeout: 30s, req-max-exec-timeout: 30s, retry-times: 0, replica-access: {{peer:1887937, store:16, err:region_not_found}}}"]
[2025/09/23 20:58:56.725 +08:00] [INFO] [coprocessor.go:1332] ["[TIME_COP_WAIT] resp_time:552.652943ms txnStartTS:461014915098345617 region_id:1773895 store_addr:10.221.202.122:20160 stats:Cop:{num_rpc:1, total_time:552.6ms} kv_process_ms:0 kv_wait_ms:1 kv_read_ms:0 processed_versions:1 total_versions:1 rocksdb_delete_skipped_count:0 rocksdb_key_skipped_count:0 rocksdb_cache_hit_count:11 rocksdb_read_count:0 rocksdb_read_byte:0"] [conn=2990378460] [session_alias=]
[2025/09/23 20:58:56.733 +08:00] [INFO] [coprocessor.go:1332] ["[TIME_COP_WAIT] resp_time:561.418515ms txnStartTS:461014915098345617 region_id:1482756 store_addr:10.221.202.122:20160 stats:Cop:{num_rpc:1, total_time:561.4ms} kv_process_ms:0 kv_wait_ms:3 kv_read_ms:0 processed_versions:1 total_versions:1 rocksdb_delete_skipped_count:0 rocksdb_key_skipped_count:0 rocksdb_cache_hit_count:12 rocksdb_read_count:1 rocksdb_read_byte:13324"] [conn=2990378460] [session_alias=]
[2025/09/23 20:58:56.734 +08:00] [INFO] [coprocessor.go:1332] ["[TIME_COP_WAIT] resp_time:522.814973ms txnStartTS:461014915111452766 region_id:1823089 store_addr:10.221.202.122:20160 stats:Cop:{num_rpc:1, total_time:522.8ms} kv_process_ms:0 kv_wait_ms:3 kv_read_ms:0 processed_versions:1 total_versions:1 rocksdb_delete_skipped_count:0 rocksdb_key_skipped_count:0 rocksdb_cache_hit_count:13 rocksdb_read_count:0 rocksdb_read_byte:0"] [conn=2990378822] [session_alias=]
[2025/09/23 20:58:56.744 +08:00] [INFO] [coprocessor.go:1332] ["[TIME_COP_WAIT] resp_time:535.138173ms txnStartTS:461014915111452766 region_id:1471019 store_addr:10.221.202.122:20160 stats:Cop:{num_rpc:1, total_time:535.1ms} kv_process_ms:0 kv_wait_ms:3 kv_read_ms:1 processed_versions:1 total_versions:1 rocksdb_delete_skipped_count:0 rocksdb_key_skipped_count:0 rocksdb_cache_hit_count:13 rocksdb_read_count:1 rocksdb_read_byte:13874"] [conn=2990378822] [session_alias=]
[2025/09/23 20:58:56.789 +08:00] [INFO] [coprocessor.go:1332] ["[TIME_COP_WAIT] resp_time:332.507812ms txnStartTS:461014915163881590 region_id:1728456 store_addr:10.221.202.122:20160 stats:Cop:{num_rpc:1, total_time:332.5ms} kv_process_ms:55 kv_wait_ms:3 kv_read_ms:55 processed_versions:402 total_versions:1717 rocksdb_delete_skipped_count:0 rocksdb_key_skipped_count:413 rocksdb_cache_hit_count:10434 rocksdb_read_count:0 rocksdb_read_byte:0"] [conn=2990378688] [session_alias=]
[2025/09/23 20:58:56.794 +08:00] [INFO] [coprocessor.go:1332] ["[TIME_COP_WAIT] resp_time:584.145896ms txnStartTS:461014915111452766 region_id:1479483 store_addr:10.221.202.122:20160 stats:Cop:{num_rpc:1, total_time:584.1ms} kv_process_ms:0 kv_wait_ms:3 kv_read_ms:0 processed_versions:1 total_versions:1 rocksdb_delete_skipped_count:0 rocksdb_key_skipped_count:0 rocksdb_cache_hit_count:14 rocksdb_read_count:0 rocksdb_read_byte:0"] [conn=2990378822] [session_alias=]
[2025/09/23 20:59:01.950 +08:00] [INFO] [coprocessor.go:1332] ["[TIME_COP_WAIT] resp_time:554.24051ms txnStartTS:461014916474601515 region_id:1860584 store_addr:10.221.202.122:20160 stats:Cop:{num_rpc:1, total_time:554.2ms} kv_process_ms:0 kv_wait_ms:0 kv_read_ms:0 processed_versions:1 total_versions:1 rocksdb_delete_skipped_count:0 rocksdb_key_skipped_count:0 rocksdb_cache_hit_count:12 rocksdb_read_count:0 rocksdb_read_byte:0"] [conn=2990378382] [session_alias=]
[2025/09/23 20:59:01.954 +08:00] [INFO] [coprocessor.go:1332] ["[TIME_COP_WAIT] resp_time:696.968683ms txnStartTS:461014916435279968 region_id:1896664 store_addr:10.221.202.122:20160 stats:Cop:{num_rpc:1, total_time:697ms} kv_process_ms:1 kv_wait_ms:0 kv_read_ms:2 processed_versions:1 total_versions:1 rocksdb_delete_skipped_count:0 rocksdb_key_skipped_count:0 rocksdb_cache_hit_count:11 rocksdb_read_count:0 rocksdb_read_byte:0"] [conn=2990378754] [session_alias=]
[2025/09/23 20:59:01.955 +08:00] [INFO] [coprocessor.go:1332] ["[TIME_COP_WAIT] resp_time:561.440022ms txnStartTS:461014916474601515 region_id:1718861 store_addr:10.221.202.122:20160 stats:Cop:{num_rpc:1, total_time:561.4ms} kv_process_ms:0 kv_wait_ms:0 kv_read_ms:0 processed_versions:1 total_versions:1 rocksdb_delete_skipped_count:0 rocksdb_key_skipped_count:0 rocksdb_cache_hit_count:9 rocksdb_read_count:0 rocksdb_read_byte:0"] [conn=2990378382] [session_alias=]
[2025/09/23 20:59:01.958 +08:00] [INFO] [coprocessor.go:1332] ["[TIME_COP_WAIT] resp_time:441.145321ms txnStartTS:461014916500816015 region_id:1485150 store_addr:10.221.202.122:20160 stats:Cop:{num_rpc:1, total_time:441.1ms} kv_process_ms:2 kv_wait_ms:0 kv_read_ms:4 processed_versions:1 total_versions:1 rocksdb_delete_skipped_count:0 rocksdb_key_skipped_count:0 rocksdb_cache_hit_count:10 rocksdb_read_count:1 rocksdb_read_byte:10654"] [conn=2990378394] [session_alias=]
[2025/09/23 20:59:01.964 +08:00] [INFO] [coprocessor.go:1332] ["[TIME_COP_WAIT] resp_time:1.031029168s txnStartTS:461014916356636676 region_id:1695893 store_addr:10.221.202.122:20160 stats:Cop:{num_rpc:1, total_time:1.03s} kv_process_ms:1 kv_wait_ms:0 kv_read_ms:3 processed_versions:1 total_versions:1 rocksdb_delete_skipped_count:0 rocksdb_key_skipped_count:0 rocksdb_cache_hit_count:10 rocksdb_read_count:0 rocksdb_read_byte:0"] [conn=2990378638] [session_alias=]
[2025/09/23 20:59:01.967 +08:00] [INFO] [coprocessor.go:1332] ["[TIME_COP_WAIT] resp_time:372.96676ms txnStartTS:461014916527030331 region_id:191573 store_addr:10.221.202.122:20160 stats:Cop:{num_rpc:1, total_time:373ms} kv_process_ms:11 kv_wait_ms:1 kv_read_ms:11 processed_versions:0 total_versions:105 rocksdb_delete_skipped_count:0 rocksdb_key_skipped_count:0 rocksdb_cache_hit_count:1155 rocksdb_read_count:0 rocksdb_read_byte:0"] [conn=2990378808] [session_alias=]
[2025/09/23 20:59:01.976 +08:00] [INFO] [coprocessor.go:1332] ["[TIME_COP_WAIT] resp_time:719.896417ms txnStartTS:461014916435279968 region_id:1865074 store_addr:10.221.202.122:20160 stats:Cop:{num_rpc:1, total_time:719.9ms} kv_process_ms:0 kv_wait_ms:0 kv_read_ms:1 processed_versions:1 total_versions:1 rocksdb_delete_skipped_count:0 rocksdb_key_skipped_count:0 rocksdb_cache_hit_count:9 rocksdb_read_count:0 rocksdb_read_byte:0"] [conn=2990378754] [session_alias=]
[2025/09/23 20:59:01.976 +08:00] [INFO] [coprocessor.go:1332] ["[TIME_COP_WAIT] resp_time:579.872097ms txnStartTS:461014916474601515 region_id:1874602 store_addr:10.221.202.122:20160 stats:Cop:{num_rpc:1, total_time:579.9ms} kv_process_ms:0 kv_wait_ms:0 kv_read_ms:0 processed_versions:1 total_versions:1 rocksdb_delete_skipped_count:0 rocksdb_key_skipped_count:0 rocksdb_cache_hit_count:9 rocksdb_read_count:0 rocksdb_read_byte:0"] [conn=2990378382] [session_alias=]

WalterWj · 2025 年9 月 24 日 02:58

看下监控，tikv detail >> error 下面的监控面板有什么。

Hacker_Yv76YjBL · 2025 年9 月 24 日 03:06

未发现异常

Hacker_Yv76YjBL · 2025 年9 月 24 日 08:37

由于业务逻辑，所有的DML操作都是显示事务。但出现执行耗时较长的DML时，提示有大量的锁。

tidb server节点日志

[2025/09/24 14:40:34.996 +08:00] [INFO] [prewrite.go:444] ["prewrite encounters lock"] [session=3971944250] [txnID=461031613395894546] [lock="key: 7480000000000000485F728000000097457EF7, primary: 7480000000000000485F698000000000000001013131303138323534FF3934303232363239FF3636000000000000F9, txnStartTS: 461031613395894545, lockForUpdateTS:461031613395894545, minCommitTs:461031613395894546, ttl: 20000, type: PessimisticLock, UseAsyncCommit: false, txnSize: 0"]
[2025/09/24 14:40:35.084 +08:00] [INFO] [prewrite.go:444] ["prewrite encounters lock"] [session=3971944250] [txnID=461031613395894546] [lock="key: 7480000000000000485F728000000097457EF7, primary: 7480000000000000485F698000000000000001013131303138323534FF3934303232363239FF3636000000000000F9, txnStartTS: 461031613395894545, lockForUpdateTS:461031613395894545, minCommitTs:461031613395894574, ttl: 20000, type: Put, UseAsyncCommit: true, txnSize: 1"]
[2025/09/24 14:40:35.278 +08:00] [INFO] [prewrite.go:444] ["prewrite encounters lock"] [session=3971944250] [txnID=461031613395894546] [lock="key: 7480000000000000485F728000000097457EF7, primary: 7480000000000000485F698000000000000001013131303138323534FF3934303232363239FF3636000000000000F9, txnStartTS: 461031613395894545, lockForUpdateTS:461031613395894545, minCommitTs:461031613395894574, ttl: 20000, type: Put, UseAsyncCommit: true, txnSize: 1"]
[2025/09/24 14:40:35.555 +08:00] [INFO] [prewrite.go:444] ["prewrite encounters lock"] [session=3971944250] [txnID=461031613395894546] [lock="key: 7480000000000000485F728000000097457EF7, primary: 7480000000000000485F698000000000000001013131303138323534FF3934303232363239FF3636000000000000F9, txnStartTS: 461031613395894545, lockForUpdateTS:461031613395894545, minCommitTs:461031613395894574, ttl: 20000, type: Put, UseAsyncCommit: true, txnSize: 1"]
[2025/09/24 14:40:36.265 +08:00] [INFO] [prewrite.go:444] ["prewrite encounters lock"] [session=3971944250] [txnID=461031613395894546] [lock="key: 7480000000000000485F728000000097457EF7, primary: 7480000000000000485F698000000000000001013131303138323534FF3934303232363239FF3636000000000000F9, txnStartTS: 461031613395894545, lockForUpdateTS:461031613395894545, minCommitTs:461031613395894574, ttl: 20000, type: Put, UseAsyncCommit: true, txnSize: 1"]
[2025/09/24 14:40:37.810 +08:00] [INFO] [prewrite.go:444] ["prewrite encounters lock"] [session=3971944250] [txnID=461031613395894546] [lock="key: 7480000000000000485F728000000097457EF7, primary: 7480000000000000485F698000000000000001013131303138323534FF3934303232363239FF3636000000000000F9, txnStartTS: 461031613395894545, lockForUpdateTS:461031613395894545, minCommitTs:461031613395894574, ttl: 20000, type: Put, UseAsyncCommit: true, txnSize: 1"]

WalterWj · 2025 年9 月 24 日 08:39

去看下监控，这个地方每次高的都是一台还是多台。

Hacker_Yv76YjBL · 2025 年9 月 24 日 09:17

以下时间update耗时均为5秒+
09-23 10：23 均正常
09-23 13：49 store-5
09-23 15：03 均正常
09-23 20：17 均正常
09-23 20：58 store-16
09-23 21：02 store-16
09-23 21：25 store-16
09-24 08：47 均正常
09-24 14：40 均正常
09-24 15：03 均正常

我是火炎焱燚 · 2025 年9 月 24 日 16:27

看下GC cycle duration，GC worker count，GC deleted keys三个指标呢

像风一样的男子 · 2025 年9 月 25 日 01:22

查看下kv的监控指标 TiKV-Details Thread CPU下有个Unified read pool CPU 这个cpu使用是否到瓶颈了