请问有办法降级嘛？比如从6.5.3 降级到6.5.1

johnwa-CD · 2023 年6 月 23 日 11:45

看过了.确实是 0 ,但是这个文件是好久好久以前的了 …2021年.也是奇葩

johnwa-CD · 2023 年6 月 23 日 11:50

server report failture 就是这样,前面某一个点又那么一些,然后就正常了,都是 0

johnwa-CD · 2023 年6 月 23 日 11:56

还有这种,但是好像不影响

2023-06-23 19:07:09 (UTC+08:00)

TiKV 10.10.10.41:20160

[endpoint.rs:759] [“cdc initialize fail: Request error message: "EpochNotMatch current epoch of region 8902005922 is conf_ver: 45523 version: 178491, but you sent conf_ver: 45523 version: 178489" epoch_not_match { current_regions { id: 8902005922 start_key: 748000000000091FFF6500000000000000F8 end_key: 7480000000000922FF775F698000000000FF0000020131343032FF37303931FF383833FF3000000000FB0132FF3032322D30352DFFFF31362030373A3030FFFF3A303000000000FF00FA010000000000FF000000F700000000FB region_epoch { conf_ver: 45523 version: 178491 } peers { id: 8902005923 store_id: 4109233388 } peers { id: 8902005924 store_id: 170703971 } peers { id: 8902005925 store_id: 4009574399 } } current_regions { id: 8902007841 start_key: 748000000000091FFF1D5F728000000363FF02718B0000000000FA end_key: 748000000000091FFF1D5F728000000363FF0D1EB20000000000FA region_epoch { conf_ver: 81329 version: 184729 } peers { id: 8902007842 store_id: 4102664327 } peers { id: 8902007843 store_id: 4009574399 } peers { id: 8902007870 store_id: 170703971 } } current_regions { id: 8902015655 start_key: 748000000000091FFF1D5F728000000362FFEE6D6B0000000000FA end_key: 748000000000091FFF1D5F728000000363FF02718B0000000000FA region_epoch { conf_ver: 81329 version: 184729 } peers { id: 8902015656 store_id: 4102664327 } peers { id: 8902015657 store_id: 4009574399 } peers { id: 8902015658 store_id: 170703971 } } current_regions { id: 8902006069 start_key: 748000000000091FFF1D5F728000000362FF5D477A0000000000FA end_key: 748000000000091FFF1D5F728000000362FF6A86CA0000000000FA region_epoch { conf_ver: 81329 version: 184724 } peers { id: 8902006070 store_id: 4102664327 } peers { id: 8902006071 store_id: 4009574399 } peers { id: 8902006073 store_id: 170703971 } } current_regions { id: 8902008220 start_key: 748000000000091FFF1D5F728000000362FF4BD56A0000000000FA end_key: 748000000000091FFF1D5F728000000362FF5D477A0000000000FA region_epoch { conf_ver: 81329 version: 184724 } peers { id: 8902008221 store_id: 4102664327 } peers { id: 8902008222 store_id: 4009574399 } peers { id: 8902008223 store_id: 170703971 } } current_regions { id: 8901993451 start_key: 748000000000091FFF1D5F728000000360FF16C9970000000000FA end_key: 748000000000091FFF1D5F728000000360FF220EB10000000000FA region_epoch { conf_ver: 81329 version: 184705 } peers { id: 8901993453 store_id: 4009574399 } peers { id: 8901993454 store_id: 3525244877 } peers { id: 8901993472 store_id: 170703971 } } current_regions { id: 8901998211 start_key: 748000000000091FFF1D5F72800000035FFFF9903A0000000000FA end_key: 748000000000091FFF1D5F728000000360FF16C9970000000000FA region_epoch { conf_ver: 81329 version: 184705 } peers { id: 8901998212 store_id: 4009574399 } peers { id: 8901998213 store_id: 3525244877 } peers { id: 8901998214 store_id: 170703971 } } current_regions { id: 8901983316 start_key: 748000000000091FFF1D5F72800000035FFFB386330000000000FA end_key: 748000000000091FFF1D5F72800000035FFFBF512B0000000000FA region_epoch { conf_ver: 81329 version: 184702 } peers { id: 8901983317 store_id: 4102664327 } peers { id: 8901983318 store_id: 4009574399 } peers { id: 8901983320 store_id: 170703971 } } current_regions { id: 8901998110 start_key: 748000000000091FFF1D5F72800000035FFFA1D6580000000000FA end_key: 748000000000091FFF1D5F72800000035FFFB386330000000000FA region_epoch { conf_ver: 81329 version: 184702 } peers { id: 8901998111 store_id: 4102664327 } peers { id: 8901998112 store_id: 4009574399 } peers { id: 8901998113 store_id: 170703971 } } current_regions { id: 8901972068 start_key: 748000000000091FFF1D5F72800000035BFFF493EC0000000000FA end_key: 748000000000091FFF1D5F72800000035CFF03C4AD0000000000FA region_epoch { conf_ver: 81329 version: 184672 } peers { id: 8901972070 store_id: 4009574399 } peers { id: 8901972071 store_id: 3525244877 } peers { id: 8901972076 store_id: 170703971 } } current_regions { id: 8901947844 start_key: 748000000000091FFF1D5F728000000359FF191CBF0000000000FA end_key: 748000000000091FFF1D5F728000000359FF2DC0AB0000000000FA region_epoch { conf_ver: 81329 version: 184648 } peers { id: 8901947846 store_id: 4009574399 } peers { id: 8901947847 store_id: 3525244877 } peers { id: 8901970097 store_id: 170703971 } } }”] [region_id=8902005922]

johnwa-CD · 2023 年6 月 23 日 12:04

CDC不是上面的问题
是跟这位老哥一样的.会直接挂掉

TICDC 向一表一TOPIC分发数据，经常莫名卡死，不再有新的数据写入 - 建议反馈 / 产品缺陷 - TiDB 的问答社区 (asktug.com)

johnwa-CD · 2023 年6 月 23 日 12:14

升级之前都OK, 就是很平稳的这个,升级后就这样了各种飙升

zhanggame1 · 2023 年6 月 23 日 12:16

有问题tikv我觉得也可以考虑先扩容再缩容，从集群里面慢慢移除

johnwa-CD · 2023 年6 月 23 日 12:36

计划这么做了.但是 ,扩容一个节点,平衡数需要大概 1周时间, 巨蛋疼
请问有没有加速平衡的最佳实践

xfworld · 2023 年6 月 23 日 13:49

看起来问题并不是升级导致的，只不过升级把这些问题都放大了…

日志描述的情况，感觉是集群运维工作不是太得当…

加速平衡的文章上面已经推给你了… 可以参考…

有猫万事足 · 2023 年6 月 23 日 13:57

1111

每次升级的时候，其实只升级了bin目录，而且老的版本也没有删除只是改了名字。
你要降级，可以尝试把文件夹的名字互换一下，改回来试试。

不过这个方法有点冒险。没有文档，所以也肯定不会是官方推荐的做法。

我还是更推荐你先扩容2个tikv再说。哪怕平衡数据的时间长，弄不好有新机器顶上去，qps能恢复一些也比现在强的多。这问题要是在生产环境持续几天，确实令人抓狂。

johnwa-CD · 2023 年6 月 23 日 14:06

运维 … 我们本来平稳运行6.1.5
只是做了一个用tiup升级到6.5.3,也是严格按照官方工具执行升级
现在还有一个更好玩的, 2个TiFlash,另外一台一旦加入,立马就性能直接拉跨~~~
上面那位兄弟和我们一样的CDC严重无法使用 …
运维哪里做的错了吗?请指点一下?

xfworld · 2023 年6 月 23 日 14:08

[cleanup_sst.rs:119] [“get region failed”] [err=“failed to load full sst meta from disk for [242, 148, 145, 51, 128, 141, 72, 6, 165, 188, 254, 164, 106, 120, 179, 243] and there isn’t extra information provided: Engine Engine(Status { code: IoError, sub_code: None, sev: NoError, state: “Corruption: file is too short (0 bytes) to be an sstable: /data/tidb-data/tikv-20160/import/f2949133-808d-4806-a5bc-fea46a78b3f3_4080615702_42367_145285_write.sst” })”]

6.1.X → 6.5.X 直接升？

johnwa-CD · 2023 年6 月 23 日 14:14

是的, 6.1.x 后,不是只有 6.5 吗?

xfworld · 2023 年6 月 23 日 14:16

6.1 和 6.5 相差还是有点大的…

稳定的环境，升级操作…不准备点后备的资源做回滚计划和方案么？

不过现在说这些感觉没啥用了，你看看小伙伴们推荐的方案有没有合适的，先试试

johnwa-CD · 2023 年6 月 23 日 14:19

哎,算是大意了吧.确实没有做好备用方案
之前从5.1 升级到 6.1 是非常顺利的.先在测试环境操作了一遍 .

Anna · 2023 年6 月 24 日 05:32

如果是测试环境，可以手动回退试试，但是成功机率不会太高…

johnwa-CD · 2023 年6 月 24 日 05:50

生产,看了下Leader掉之前的日志
[2023/06/24 13:20:31.649 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 4110352114 store_id: 3525244877”] [peer_id=4110352115] [region_id=4110352112] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.649 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 4104275222 store_id: 3525244877”] [peer_id=4104275221] [region_id=4104275219] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.649 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 4055399997 store_id: 3525244877”] [peer_id=4055399999] [region_id=4055399996] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.649 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 4107973128 store_id: 3525244877”] [peer_id=4107973127] [region_id=4107973125] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.649 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 8892689573 store_id: 3525244877”] [peer_id=8892695595] [region_id=8892689571] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.649 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 4079841272 store_id: 3525244877”] [peer_id=4079841274] [region_id=4079841271] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.649 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 8890399175 store_id: 3525244877”] [peer_id=8890399174] [region_id=8890399173] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.649 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 4097038269 store_id: 3525244877”] [peer_id=4097038268] [region_id=4097038267] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.649 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 4107226083 store_id: 3525244877”] [peer_id=4107226084] [region_id=4107226081] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.649 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 4101579969 store_id: 3525244877”] [peer_id=8901970103] [region_id=4101579967] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.649 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 4109137572 store_id: 3525244877”] [peer_id=5452196106] [region_id=4109137570] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.649 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 4093367912 store_id: 3525244877”] [peer_id=4093367913] [region_id=4093367910] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.649 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 8890952443 store_id: 3525244877”] [peer_id=8890952444] [region_id=8890952442] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.649 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 4101596179 store_id: 3525244877”] [peer_id=4101596178] [region_id=4101596176] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.649 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 6898913506 store_id: 3525244877”] [peer_id=4103695167] [region_id=4103695166] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.649 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 4107340160 store_id: 3525244877”] [peer_id=4107341260] [region_id=4107340159] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.649 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 8893019259 store_id: 3525244877”] [peer_id=8893019260] [region_id=8893019257] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.649 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 8890700847 store_id: 3525244877”] [peer_id=8890700848] [region_id=8890700846] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.649 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 4101064441 store_id: 3525244877”] [peer_id=4101064439] [region_id=4101064438] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.649 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 4087309066 store_id: 3525244877”] [peer_id=4087309068] [region_id=4087309064] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.649 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 4093337000 store_id: 3525244877”] [peer_id=4093336998] [region_id=4093336997] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.649 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 4100165081 store_id: 3525244877”] [peer_id=4100165079] [region_id=4100165078] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.649 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 4093727783 store_id: 3525244877”] [peer_id=4093727784] [region_id=4093727781] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.649 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 4095762382 store_id: 3525244877”] [peer_id=7391211588] [region_id=4095762381] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.650 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 4390015095 store_id: 3525244877”] [peer_id=4097248500] [region_id=4097248498] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.650 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 4102635706 store_id: 3525244877”] [peer_id=8344819278] [region_id=4102635703] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.650 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 8892503335 store_id: 3525244877”] [peer_id=8892503334] [region_id=8892503332] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.650 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 8901891998 store_id: 3525244877”] [peer_id=8901891996] [region_id=8901891995] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.650 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 5424426001 store_id: 3525244877”] [peer_id=4104999928] [region_id=4104999927] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.650 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 8892364710 store_id: 3525244877”] [peer_id=5640742824] [region_id=4108288132] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.650 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 4102902919 store_id: 3525244877”] [peer_id=4102902920] [region_id=4102902917] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.650 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 5682202902 store_id: 3525244877”] [peer_id=5682202903] [region_id=5682202901] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.650 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 4097923468 store_id: 3525244877”] [peer_id=4097923159] [region_id=4097923158] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.650 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 4096981894 store_id: 3525244877”] [peer_id=4096981893] [region_id=4096981892] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.650 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 4096993271 store_id: 3525244877”] [peer_id=4096993270] [region_id=4096993269] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.650 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 8892105046 store_id: 3525244877”] [peer_id=8892105045] [region_id=8892105044] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.650 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 8892792679 store_id: 3525244877”] [peer_id=8892792677] [region_id=8892792676] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.650 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 4107720755 store_id: 3525244877”] [peer_id=4107720754] [region_id=4107720753] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.650 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 4095679253 store_id: 3525244877”] [peer_id=4095679252] [region_id=4095679251] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.650 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 8892511712 store_id: 3525244877”] [peer_id=8892511714] [region_id=8892511711] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.650 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 4212019760 store_id: 3525244877”] [peer_id=4096245933] [region_id=4096245931] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.650 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 4086052073 store_id: 3525244877”] [peer_id=4086052072] [region_id=4086052070] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.650 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 4096710954 store_id: 3525244877”] [peer_id=8890931255] [region_id=4096710952] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.650 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 8892821467 store_id: 3525244877”] [peer_id=8892821468] [region_id=8892821465] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.650 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 5715710423 store_id: 3525244877”] [peer_id=4100649314] [region_id=4100649302] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.650 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 4108652342 store_id: 3525244877”] [peer_id=4108652340] [region_id=4108652339] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.650 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 8892754530 store_id: 3525244877”] [peer_id=8892712355] [region_id=8892712352] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.650 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 8891901387 store_id: 3525244877”] [peer_id=8891901389] [region_id=8891901386] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.650 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 4051084103 store_id: 3525244877”] [peer_id=4051084104] [region_id=4051084101] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.650 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 4096451951 store_id: 3525244877”] [peer_id=4096451950] [region_id=4096451949] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.650 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 4090826537 store_id: 3525244877”] [peer_id=4090826538] [region_id=4090826536] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.651 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 4107588375 store_id: 3525244877”] [peer_id=4107588374] [region_id=4107588373] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.651 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 8903034050 store_id: 3525244877”] [peer_id=8903034049] [region_id=8903034047] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.651 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 4103562682 store_id: 3525244877”] [peer_id=4103562683] [region_id=4103562680] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.651 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 4096539379 store_id: 3525244877”] [peer_id=4096539380] [region_id=4096539378] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.651 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 8901956783 store_id: 3525244877”] [peer_id=8901956781] [region_id=8901956780] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.651 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 4086819585 store_id: 3525244877”] [peer_id=4086819584] [region_id=4086819583] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.651 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 4103201745 store_id: 3525244877”] [peer_id=4103201747] [region_id=4103201744] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.651 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 4103402133 store_id: 3525244877”] [peer_id=4103402134] [region_id=4103402132] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.651 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 4110102500 store_id: 3525244877”] [peer_id=4110102502] [region_id=4110102499] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.651 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 4085349970 store_id: 3525244877”] [peer_id=8891640582] [region_id=4085349969] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.651 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 4072490547 store_id: 3525244877”] [peer_id=4072490546] [region_id=4072490544] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.651 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 6288759163 store_id: 3525244877”] [peer_id=4055769067] [region_id=4055769064] [type=MsgHibernateResponse]
[2023/06/24 13:20:31.651 +08:00] [Error] [peer.rs:5243] [“failed to send extra message”] [err_code=KV:Raftstore:Transport] [err=Transport(Full)] [target=“id: 4100191252 store_id: 3525244877”] [peer_id=4100191251] [region_id=4100191249] [type=MsgHibernateResponse]

大量这个错误日志,是什么意思呢?可能什么原因?有大神能看下吗

dba-kit · 2023 年6 月 24 日 08:07

这两个节点有什么特殊的么？TiDB当检测到是慢节点时候，会自动将慢节点上的leader驱逐到其他节点。我也遇到过几次，不过都是在往SATA盘节点迁移数据时候，SATA盘的IO满导致的，正常情况下不会。

johnwa-CD · 2023 年6 月 24 日 08:10

没有,原来运行OK,已经几个月了.这次就是从 6.1 升级到 6.5
硬盘是 NVME,io显示util 就是20%左右

dba-kit · 2023 年6 月 24 日 08:14

如果是SATA盘节点问题，可以尝试增加一下raftstore.inspect-interval的值：

dba-kit · 2023 年6 月 24 日 08:15

我现在集群也是从6.1升到6.5的，不过是从6.1.2升级到6.5.0，倒是没出现问题。