tidb_tikvclient_backoff_seconds_count 经常报警，type="regionMiss" 这个怎么解

🪐 TiDB 技术问题 K8s
1

image
image1920×942 78 KB

【 TiDB 使用环境】
【概述】：场景 + 问题概述
【背景】：做过哪些操作
【现象】：业务和数据库现象
【问题】：当前遇到的问题
【业务影响】：
【TiDB 版本】：
【TiDB Operator 版本】：
【K8s 版本】：
【附件】：

2

leader 抖动会导致 leader 迁移，可能会出现 regionmiss，可以看下监控 tidb里的kv error 和 tikv 里的 error，看下造成 backoff 的类型，或者查下tidb的日志，如果只是写冲突，没有太大风险。

3

[2022/02/23 11:51:08.448 +08:00] [WARN] [grpclogger.go:81] [“grpc: Server.Serve failed to create ServerTransport: connection error: desc = “transport: http2Server.HandleStreams failed to receive the preface from client: EOF””] [system=grpc] [grpc_log=true]
[2022/02/23 11:51:18.111 +08:00] [WARN] [grpclogger.go:81] [“grpc: Server.Serve failed to create ServerTransport: connection error: desc = “transport: http2Server.HandleStreams failed to receive the preface from client: EOF””] [system=grpc] [grpc_log=true]
[2022/02/23 11:51:38.429 +08:00] [WARN] [grpclogger.go:81] [“grpc: Server.Serve failed to create ServerTransport: connection error: desc = “transport: http2Server.HandleStreams failed to receive the preface from client: EOF””] [system=grpc] [grpc_log=true]
[2022/02/23 11:51:48.129 +08:00] [WARN] [grpclogger.go:81] [“grpc: Server.Serve failed to create ServerTransport: connection error: desc = “transport: http2Server.HandleStreams failed to receive the preface from client: EOF””] [system=grpc] [grpc_log=true]
[2022/02/23 11:52:08.356 +08:00] [WARN] [grpclogger.go:81] [“grpc: Server.Serve failed to create ServerTransport: connection error: desc = “transport: http2Server.HandleStreams failed to receive the preface from client: EOF””] [system=grpc] [grpc_log=true]
[2022/02/23 11:52:18.169 +08:00] [WARN] [grpclogger.go:81] [“grpc: Server.Serve failed to create ServerTransport: connection error: desc = “transport: http2Server.HandleStreams failed to receive the preface from client: EOF””] [system=grpc] [grpc_log=true]

4

参考：集群状态正常，没有节点下线的情况下，出现大量 regionMiss 的？

5

集群状态是正常的， 以下是tikv 的错误信息
[2022/02/23 12:05:42.922 +08:00] [WARN] [endpoint.rs:632] [error-response] [err=“Key is locked (will clean up) primary_lock: 7480000000000000155F69800000000000000203800000000000047A lock_version: 431381320271134729 key: 7480000000000000155F6980000000000000010405FC930617C0001503800000000001D4C5 lock_ttl: 3001 txn_size: 4 lock_type: Del lock_for_update_ts: 431381320271134729 use_async_commit: true min_commit_ts: 431381320271134731”]
[2022/02/23 12:06:42.925 +08:00] [WARN] [endpoint.rs:632] [error-response] [err=“Key is locked (will clean up) primary_lock: 7480000000000000175F728000000000293480 lock_version: 431381335999774744 key: 7480000000000000175F728000000000293480 lock_ttl: 3000 txn_size: 4 min_commit_ts: 431381335999774745”]
[2022/02/23 12:08:27.395 +08:00] [WARN] [endpoint.rs:632] [error-response] [err=“Key is locked (will clean up) primary_lock: 7480000000000000155F69800000000000000203800000000000047C lock_version: 431381362528747521 key: 7480000000000000155F69800000000000000203800000000000047C lock_ttl: 20000 txn_size: 1281 lock_for_update_ts: 431381363341393925 min_commit_ts: 431381363380715523”]
[2022/02/23 12:09:42.925 +08:00] [WARN] [endpoint.rs:632] [error-response] [err=“Key is locked (will clean up) primary_lock: 7480000000000000175F72800000000029348F lock_version: 431381383185694729 key: 7480000000000000175F72800000000029348F lock_ttl: 3000 txn_size: 4 min_commit_ts: 431381383185694730”]
[2022/02/23 12:11:54.226 +08:00] [WARN] [endpoint.rs:632] [error-response] [err=“Region error (will back off and retry) message: "peer is not leader for region 110005, leader may Some(id: 110008 store_id: 80)" not_leader { region_id: 110005 leader { id: 110008 store_id: 80 } }”]
[2022/02/23 12:11:54.923 +08:00] [WARN] [endpoint.rs:632] [error-response] [err=“Region error (will back off and retry) message: "peer is not leader for region 110005, leader may Some(id: 110008 store_id: 80)" not_leader { region_id: 110005 leader { id: 110008 store_id: 80 } }”]
[2022/02/23 12:15:43.145 +08:00] [WARN] [endpoint.rs:632] [error-response] [err=“Region error (will back off and retry) message: "peer is not leader for region 110205, leader may Some(id: 110208 store_id: 80)" not_leader { region_id: 110205 leader { id: 110208 store_id: 80 } }”]
[2022/02/23 12:15:48.236 +08:00] [WARN] [endpoint.rs:632] [error-response] [err=“Region error (will back off and retry) message: "peer is not leader for region 110205, leader may Some(id: 110208 store_id: 80)" not_leader { region_id: 110205 leader { id: 110208 store_id: 80 } }”]
[2022/02/23 12:15:48.933 +08:00] [WARN] [endpoint.rs:632] [error-response] [err=“Region error (will back off and retry) message: "peer is not leader for region 110205, leader may Some(id: 110208 store_id: 80)" not_leader { region_id: 110205 leader { id: 110208 store_id: 80 } }”]
[2022/02/23 12:19:16.308 +08:00] [WARN] [endpoint.rs:632] [error-response] [err=“Region error (will back off and retry) message: "peer is not leader for region 110673, leader may Some(id: 110675 store_id: 54)" not_leader { region_id: 110673 leader { id: 110675 store_id: 54 } }”]
[2022/02/23 12:19:21.341 +08:00] [WARN] [endpoint.rs:632] [error-response] [err=“Region error (will back off and retry) message: "peer is not leader for region 110673, leader may Some(id: 110675 store_id: 54)" not_leader { region_id: 110673 leader { id: 110675 store_id: 54 } }”]
[2022/02/23 12:19:22.035 +08:00] [WARN] [endpoint.rs:632] [error-response] [err=“Region error (will back off and retry) message: "peer is not leader for region 110673, leader may Some(id: 110675 store_id: 54)" not_leader { region_id: 110673 leader { id: 110675 store_id: 54 } }”]