tikv 扩容后 tiflash disconnect

🪐 TiDB 技术问题
1

[root@localhost tikv-20160]# date
Sat Dec 11 14:11:44 CST 2021
[root@localhost tikv-20160]# rm -rf ./*
[root@localhost tikv-20160]# pwd
/tidb-data/tikv-20160

ID Role Host Ports OS/Arch Status Data Dir Deploy Dir

192.168.135.132:2379 pd 192.168.135.132 2379/2380 linux/x86_64 Up|UI /tidb-data/pd-2379 /tidb-deploy/pd-2379
192.168.135.133:2379 pd 192.168.135.133 2379/2380 linux/x86_64 Up /tidb-data/pd-2379 /tidb-deploy/pd-2379
192.168.135.134:2379 pd 192.168.135.134 2379/2380 linux/x86_64 Up|L /tidb-data/pd-2379 /tidb-deploy/pd-2379
192.168.135.132:4000 tidb 192.168.135.132 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000
192.168.135.133:4000 tidb 192.168.135.133 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000
192.168.135.134:4000 tidb 192.168.135.134 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000
192.168.135.135:9000 tiflash 192.168.135.135 9000/8123/3930/20170/20292/8234 linux/x86_64 Up /tidb-data/tiflash-9000 /tidb-deploy/tiflash-9000
192.168.135.132:20160 tikv 192.168.135.132 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160
192.168.135.133:20160 tikv 192.168.135.133 20160/20180 linux/x86_64 Disconnected /tidb-data/tikv-20160 /tidb-deploy/tikv-20160
192.168.135.134:20160 tikv 192.168.135.134 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160

[tidb@tidb1 ~]$ cat scale2.yaml

tikv_servers:

  • host: 192.168.135.135
    [tidb@tidb1 ~]$ tiup cluster scale-out tidb-jiantest scale2.yaml
    Scaled cluster tidb-jiantest out successfully

192.168.135.132:2379 pd 192.168.135.132 2379/2380 linux/x86_64 Up|UI /tidb-data/pd-2379 /tidb-deploy/pd-2379
192.168.135.133:2379 pd 192.168.135.133 2379/2380 linux/x86_64 Up /tidb-data/pd-2379 /tidb-deploy/pd-2379
192.168.135.134:2379 pd 192.168.135.134 2379/2380 linux/x86_64 Up|L /tidb-data/pd-2379 /tidb-deploy/pd-2379
192.168.135.132:4000 tidb 192.168.135.132 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000
192.168.135.133:4000 tidb 192.168.135.133 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000
192.168.135.134:4000 tidb 192.168.135.134 4000/10080 linux/x86_64 Up - /tidb-deploy/tidb-4000
192.168.135.135:9000 tiflash 192.168.135.135 9000/8123/3930/20170/20292/8234 linux/x86_64 Disconnected /tidb-data/tiflash-9000 /tidb-deploy/tiflash-9000
192.168.135.132:20160 tikv 192.168.135.132 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160
192.168.135.133:20160 tikv 192.168.135.133 20160/20180 linux/x86_64 Disconnected /tidb-data/tikv-20160 /tidb-deploy/tikv-20160
192.168.135.134:20160 tikv 192.168.135.134 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160
192.168.135.135:20160 tikv 192.168.135.135 20160/20180 linux/x86_64 Up /tidb-data/tikv-20160 /tidb-deploy/tikv-20160

麻烦大佬帮忙看看

2

[2021/12/11 14:15:46.325 +08:00] [INFO] [] [“Application: encryption is disabled”] [thread_id=1]
[2021/12/11 14:15:46.335 +08:00] [WARN] [] [“Application: The configuration “path” is deprecated. Check [storage] section for new style.”] [thread_id=1]
…skipping…
[2021/12/11 14:15:47.435 +08:00] [INFO] [] [“Application: tiflash proxy is ready to serve, try to wake up all regions’
leader”] [thread_id=1]
[2021/12/11 14:15:47.443 +08:00] [INFO] [] [“WaitCheckRegionReady: start to check regions ready”] [thread_id=1]
[2021/12/11 14:15:47.454 +08:00] [INFO] [] [“WaitCheckRegionReady: finish to check 0 regions, time cost 0.00597419s”] [
thread_id=1]
[2021/12/11 14:15:47.462 +08:00] [INFO] [] [“Application: Start to wait for terminal signal”] [thread_id=1]
[2021/12/11 14:15:48.416 +08:00] [DEBUG] [] [“CfgReloader: Loading config `conf/tiflash.toml’”] [thread_id=6]
[2021/12/11 14:15:48.425 +08:00] [INFO] [] [“KVStore: setRegionCompactLogConfig: threshold config: period 120, rows 409
60, bytes 33554432”] [thread_id=6]
[2021/12/11 14:15:48.425 +08:00] [INFO] [] [“reloadConfig: read-index max thread num: 1, timeout: 10000ms; wait-index t
imeout: 300000ms; wait-region-ready timeout: 1200s”] [thread_id=6]
[2021/12/11 14:15:48.425 +08:00] [INFO] [] [“IORateLimiter: storage.io_rate_limit is not found in config, use default c
onfig.”] [thread_id=6]
[2021/12/11 14:15:48.425 +08:00] [INFO] [] [“IORateLimiter: storage.io_rate_limit is not changed.”] [thread_id=6]
[2021/12/11 14:15:48.425 +08:00] [INFO] [] ["Context: reload delta tree "] [thread_id=6]
[2021/12/11 14:16:06.899 +08:00] [INFO] [] [“Welcome to TiFlash”] [thread_id=1]
[2021/12/11 14:16:06.900 +08:00] [INFO] [] [“Starting daemon with revision 54381”] [thread_id=1]

3

有朋友帮忙看看么 或者我的操作有什么不对的地方

4

不好意思 最终发现是环境问题

5

具体的问题、排查方法以及根本原因可不可以帮忙补充一下 :+1:

6

看了tikv的日志发现tikv重启后一段时间自己就重启了，后来不经意发现系统日志中有OOM把tiflash给kill了。而且日志可以对应上这边kill掉那边就重启了。

7

我的tiflash节点只给了3G,tikv 和tiflash一起跑就不行了，如果只跑一个tiflash没有问题

8

此话题已在最后回复的 1 分钟后被自动关闭。不再允许新回复。