TiFlash 报错后无法重启

🪐 TiDB 技术问题 TiFlash
1

【 TiDB 版本】
6.5.2

tiflash_error.log

[2023/09/04 10:11:31.005 +08:00] [ERROR] [<unknown>] ["DB::Exception: Unexpected eof"] [source=Application] [thread_id=1]
[2023/09/04 10:11:53.520 +08:00] [ERROR] [<unknown>] ["DB::Exception: Unexpected eof"] [source=Application] [thread_id=1]
[2023/09/04 10:12:15.817 +08:00] [ERROR] [<unknown>] ["DB::Exception: Unexpected eof"] [source=Application] [thread_id=1]
[2023/09/04 10:12:38.040 +08:00] [ERROR] [<unknown>] ["DB::Exception: Unexpected eof"] [source=Application] [thread_id=1]

tiflash_stderr.log

Logging debug to /data01/deploy/log/tiflash.log
Logging errors to /data01/deploy/log/tiflash_error.log
deprecated configuration, log-file has been moved to log.file.filename
override log.file.filename with log-file, "/data01/deploy/log/tiflash_tikv.log"

tiflash_tikv.log

[2023/09/04 08:36:32.760 +08:00] [ERROR] [observer.rs:290] ["transfer leader won't exec"] [req="cmd_type: TransferLeader transfer_leader { peer { id: 4900743611 store_id: 1151850759 } }"][region="id: 94768718 start_key: 7480000000000079FF675F72800000000AFFCDECB70000000000FA end_key: 7480000000000079FF6800000000000000F8 region_epoch { conf_ver: 236240 version: 7351 } peers { id: 4900743611 store_id: 1151850759 } peers { id: 4900826256 store_id: 4453837499 } peers { id: 4900875948 store_id: 3755910046 } peers { id: 4901151587 store_id: 4858181679 role: Learner } peers { id: 4901153408 store_id: 4745761616 role: Learner }"]

tiflash.log（反复报错并重启）

[2023/09/04 10:14:09.454 +08:00] [DEBUG] [Server.cpp:995] ["Destroyed global context."] [thread_id=1]
[2023/09/04 10:14:09.454 +08:00] [INFO] [Server.cpp:868] ["Let tiflash proxy shutdown"] [thread_id=1]
[2023/09/04 10:14:09.454 +08:00] [INFO] [Server.cpp:871] ["Wait for tiflash proxy thread to join"] [thread_id=1]
[2023/09/04 10:14:09.560 +08:00] [INFO] [Server.cpp:873] ["tiflash proxy thread is joined"] [thread_id=1]
[2023/09/04 10:14:09.560 +08:00] [ERROR] [<unknown>] ["DB::Exception: Unexpected eof"] [source=Application] [thread_id=1]
[2023/09/04 10:14:09.560 +08:00] [INFO] [<unknown>] ["shutting down"] [source=Application] [thread_id=1]
[2023/09/04 10:14:09.560 +08:00] [DEBUG] [<unknown>] ["Uninitializing subsystem: Logging Subsystem"] [source=Application] [thread_id=1]
[2023/09/04 10:14:09.560 +08:00] [INFO] [BaseDaemon.cpp:315] ["Stop SignalListener thread"] [source=BaseDaemon] [thread_id=67]
[2023/09/04 10:14:24.956 +08:00] [INFO] [<unknown>] ["Welcome to TiFlash"] [thread_id=1]
2

类似于这个

3

不一样吧，持续分析功能是关闭状态。设置的两副本，目前就是6.2版本呢。
暂时通过重新扩容处理了这个问题。

4

不过日志里有报这个，Tombstoned table db_45.t_68737(68737)，但只是INFO级别

5

重新启动 TiFlash 进程，再检查 TiFlash 的日志文件，看是否有其他错误信息

6

没有看到其他的报错信息，日志报的还是和这个一样

7

可能是这个原因，出问题的时候正在跑job，TiFlash经常触发OOM，重启依然无法阻止OOM发生

8

此话题已在最后回复的 60 天后被自动关闭。不再允许新回复。