TiKV持续报“call GetStoreSafeTS failed”

🪐 TiDB 技术问题 部署&运维管理
, ,
1

【 TiDB 使用环境】生产环境
【 TiDB 版本】v7.5.0
【遇到的问题：问题现象及影响】
TiKV的日志中持续输出以下信息，不确定是什么原因以及如何解决？
【附件：截图/日志/监控】

[2024/05/13 15:51:56.244 +08:00] [INFO] [kv.rs:955] [“kv rpc failed”] [err=RemoteStopped] [request=batch_commands] [thread_id=0x5]
[2024/05/13 15:51:59.703 +08:00] [WARN] [kv.rs:1065] [“call GetStoreSafeTS failed”] [err=Grpc(RemoteStopped)] [thread_id=0x5]
[2024/05/13 15:52:01.780 +08:00] [WARN] [kv.rs:1065] [“call GetStoreSafeTS failed”] [err=Grpc(RemoteStopped)] [thread_id=0x5]
[2024/05/13 15:52:07.941 +08:00] [INFO] [compaction_filter.rs:625] [“Compaction filter reports”] [filtered=36074] [total=347280] [thread_id=0x5]
[2024/05/13 15:52:18.348 +08:00] [INFO] [trend.rs:291] [“history window flipping: enter”] [increasing_rate=-78333.77012258547] [flip_margin_error=4083.6295946947325] [delta=4190.166666666657] [name=L2] [thread_id=0x5]
[2024/05/13 15:52:23.256 +08:00] [INFO] [kv.rs:955] [“kv rpc failed”] [err=RemoteStopped] [request=batch_commands] [thread_id=0x5]
[2024/05/13 15:52:42.867 +08:00] [WARN] [kv.rs:1065] [“call GetStoreSafeTS failed”] [err=Grpc(RemoteStopped)] [thread_id=0x5]
[2024/05/13 15:52:48.351 +08:00] [INFO] [trend.rs:272] [“history window flipping: end”] [flipping_duration=30] [increasing_rate=0] [time_based_multiple=0.00849625007211562] [flip_margin_error=2182.864407340349] [delta=2693.333333333343] [name=L2] [thread_id=0x5]
[2024/05/13 15:52:48.546 +08:00] [WARN] [kv.rs:1065] [“call GetStoreSafeTS failed”] [err=Grpc(RemoteStopped)] [thread_id=0x5]
[2024/05/13 15:53:05.765 +08:00] [INFO] [kv.rs:955] [“kv rpc failed”] [err=RemoteStopped] [request=batch_commands] [thread_id=0x5]
[2024/05/13 15:53:05.765 +08:00] [INFO] [kv.rs:955] [“kv rpc failed”] [err=RemoteStopped] [request=batch_commands] [thread_id=0x5]
[2024/05/13 15:53:05.829 +08:00] [WARN] [kv.rs:1065] [“call GetStoreSafeTS failed”] [err=Grpc(RemoteStopped)] [thread_id=0x5]
[2024/05/13 15:53:11.270 +08:00] [INFO] [kv.rs:955] [“kv rpc failed”] [err=RemoteStopped] [request=batch_commands] [thread_id=0x5]
[2024/05/13 15:53:11.270 +08:00] [INFO] [kv.rs:955] [“kv rpc failed”] [err=RemoteStopped] [request=batch_commands] [thread_id=0x5]

2

info日志可以忽略

3

好像不是报错信息吧

4

都是info级别的日志，没什么问题

5

调整下grpc的timeout可能会减少频次
请问有伴随内存增长不？

6

warning级别，问题不大

7

内存倒是没有什么增长趋势

8

虽然是info级别，不过持续在刷屏，而且带有fail字眼，还是担心会有什么隐患，感觉有思路的话还是想着把它处理一下，

9

还有很多WARN级别的，也是持续在刷屏
[2024/05/13 20:58:31.961 +08:00] [WARN] [scanner.rs:137] [“resolved_ts scan get snapshot failed”] [err=“Other("[components/resolved_ts/src/scanner.rs:193]: scan task cancelled")”] [thread_id=0x5]
[2024/05/13 20:58:32.633 +08:00] [WARN] [scanner.rs:137] [“resolved_ts scan get snapshot failed”] [err=“Other("[components/resolved_ts/src/scanner.rs:193]: scan task cancelled")”] [thread_id=0x5]
[2024/05/13 20:58:33.311 +08:00] [WARN] [scanner.rs:137] [“resolved_ts scan get snapshot failed”] [err=“Other("[components/resolved_ts/src/scanner.rs:193]: scan task cancelled")”] [thread_id=0x5]
[2024/05/13 20:58:33.999 +08:00] [WARN] [scanner.rs:137] [“resolved_ts scan get snapshot failed”] [err=“Other("[components/resolved_ts/src/scanner.rs:193]: scan task cancelled")”] [thread_id=0x5]
[2024/05/13 20:58:33.999 +08:00] [WARN] [scanner.rs:137] [“resolved_ts scan get snapshot failed”] [err=“Other("[components/resolved_ts/src/scanner.rs:193]: scan task cancelled")”] [thread_id=0x5]
[2024/05/13 20:58:34.677 +08:00] [WARN] [scanner.rs:137] [“resolved_ts scan get snapshot failed”] [err=“Other("[components/resolved_ts/src/scanner.rs:193]: scan task cancelled")”] [thread_id=0x5]
[2024/05/13 20:58:34.677 +08:00] [WARN] [scanner.rs:137] [“resolved_ts scan get snapshot failed”] [err=“Other("[components/resolved_ts/src/scanner.rs:193]: scan task cancelled")”] [thread_id=0x5]
[2024/05/13 20:58:35.348 +08:00] [WARN] [scanner.rs:137] [“resolved_ts scan get snapshot failed”] [err=“Other("[components/resolved_ts/src/scanner.rs:193]: scan task cancelled")”] [thread_id=0x5]
[2024/05/13 20:58:35.349 +08:00] [WARN] [scanner.rs:137] [“resolved_ts scan get snapshot failed”] [err=“Other("[components/resolved_ts/src/scanner.rs:193]: scan task cancelled")”] [thread_id=0x5]

10

没有太重要的日志信息

11

warn和info级别的日志不用管，主要看error级别的