TiDB 升级至6.5.1后,TiKV重启异常增加

【 TiDB 使用环境】生产环境 /测试/ Poc
【 TiDB 版本】v6.5.1
【复现路径】正常读写数据
【遇到的问题:问题现象及影响】TiKV异常重启,日志中看不到报错,但是系统层可见如下报错:

【资源配置】16C 64G
【附件:截图/日志/监控】

Apr 11 11:16:02 master2 systemd[1]: Started Session c41863 of user root.
Apr 11 11:16:08 master2 kernel: [3432600.511534] EXT4-fs error (device vdb) in ext4_free_blocks:4972: Corrupt filesystem
Apr 11 11:16:08 master2 kernel: [3432600.523842] vmsec-host-net. (3664293): drop_caches: 3
Apr 11 11:16:08 master2 kernel: [3432600.597301] [VMSECNFQ]: Version: 8.0.3
Apr 11 11:16:08 master2 kernel: [3432600.597307] [VMSECNFQ]: Hook nic prefix enp3s0: dev: 000000009c7a1abe, IP: 0.0.0.0
Apr 11 11:16:08 master2 kernel: [3432600.603000] [VMSECNFQ]: init success
Apr 11 11:16:08 master2 kernel: [3432600.617574] EXT4-fs error (device vdb) in ext4_free_blocks:4972: Corrupt filesystem
Apr 11 11:16:09 master2 kernel: [3432601.748412] EXT4-fs error (device vdb) in ext4_free_blocks:4972: Corrupt filesystem
Apr 11 11:16:09 master2 kernel: [3432601.750499] EXT4-fs error (device vdb) in ext4_free_blocks:4972: Corrupt filesystem
Apr 11 11:16:09 master2 kernel: [3432601.773392] [VMSECNFQ]: Safe to release netlink socket as there are no live reference!
Apr 11 11:16:09 master2 kernel: [3432601.773435] [VMSECNFQ]: Safe to quit as there are no onging operations!
Apr 11 11:16:10 master2 kernel: [3432601.887681] [VMSECNFQ]: g_vmsec_fw_ctx resource freee OK
Apr 11 11:16:14 master2 systemd[1]: session-c41863.scope: Succeeded.
Apr 11 11:16:18 master2 kernel: [3432610.065748] EXT4-fs error (device vdb) in ext4_free_blocks:4972: Corrupt filesystem
Apr 11 11:16:18 master2 systemd[1]: tikv-20160.service: Main process exited, code=exited, status=1/FAILURE
Apr 11 11:16:18 master2 systemd[1]: tikv-20160.service: Failed with result 'exit-code'.
Apr 11 11:16:33 master2 systemd[1]: tikv-20160.service: Service RestartSec=15s expired, scheduling restart.
Apr 11 11:16:33 master2 systemd[1]: tikv-20160.service: Scheduled restart job, restart counter is at 43.
Apr 11 11:16:33 master2 systemd[1]: Stopped tikv service.
Apr 11 11:16:33 master2 systemd[1]: Started tikv service.

有没有相同问题的小伙伴,有没有大佬知道可能的问题所在?

1 个赞

是不是硬盘故障了

查一下硬盘吧,不过看着像是虚拟磁盘

主机内存没达到瓶颈?

这个有找到原因吗

像是硬件有问题,看这段日志的话