TiFlash 部署后一直down状态

[qwewer123234]

你好，使用tiup部署tiflash 服务器后，一直显示down状态，错误日志如下，请问可能是什么原因？谢谢。：

2020.08.24 12:24:41.842330 [ 4 ] <Error> BaseDaemon: ########################################

2020.08.24 12:24:41.842364 [ 4 ] <Error> BaseDaemon: (from thread 2) Received signal Segmentation fault (11).

2020.08.24 12:24:41.842375 [ 4 ] <Error> BaseDaemon: Address: 0x2f5

2020.08.24 12:24:41.842381 [ 4 ] <Error> BaseDaemon: Access: read.

2020.08.24 12:24:41.842395 [ 4 ] <Error> BaseDaemon: Address not mapped to object.

2020.08.24 12:24:41.846761 [ 4 ] <Error> BaseDaemon: 0. bin/tiflash/tiflash(DB::HandleCheckTerminated(DB::TiFlashServer*)+0xc) [0x66d44ec]

2020.08.24 12:24:41.846772 [ 4 ] <Error> BaseDaemon: 1. /home/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0x155769b) [0x7f9a3a55369b]

2020.08.24 12:24:41.846787 [ 4 ] <Error> BaseDaemon: 2. /home/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(run_tiflash_proxy_ffi+0x4e5a) [0x7f9a3a57c5aa]

2020.08.24 12:24:41.846794 [ 4 ] <Error> BaseDaemon: 3. bin/tiflash/tiflash() [0x30ee3de]

2020.08.24 12:24:41.846801 [ 4 ] <Error> BaseDaemon: 4. bin/tiflash/tiflash() [0x7a029cf]

2020.08.24 12:24:41.846808 [ 4 ] <Error> BaseDaemon: 5. /lib64/libpthread.so.0(+0x7e25) [0x7f9a38559e25]

2020.08.24 12:24:50.430278 [ 4 ] <Error> BaseDaemon: ########################################

2020.08.24 12:24:50.430296 [ 4 ] <Error> BaseDaemon: (from thread 5) Received signal Segmentation fault (11).

2020.08.24 12:24:50.430304 [ 4 ] <Error> BaseDaemon: Address: NULL pointer.

2020.08.24 12:24:50.430310 [ 4 ] <Error> BaseDaemon: Access: read.

2020.08.24 12:24:50.430316 [ 4 ] <Error> BaseDaemon: Address not mapped to object.

2020.08.24 12:24:50.433935 [ 4 ] <Error> BaseDaemon: 0. bin/tiflash/tiflash(DB::TMTContext::getContext()+0x1) [0x672a9b1]

2020.08.24 12:24:50.433947 [ 4 ] <Error> BaseDaemon: 1. bin/tiflash/tiflash(DB::HandleComputeFsStats(DB::TiFlashServer*)+0x27) [0x66d4687]

2020.08.24 12:24:50.433954 [ 4 ] <Error> BaseDaemon: 2. /home/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0xf79708) [0x7f9a39f75708]

2020.08.24 12:24:50.433961 [ 4 ] <Error> BaseDaemon: 3. /home/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0x12fb186) [0x7f9a3a2f7186]

2020.08.24 12:24:50.433968 [ 4 ] <Error> BaseDaemon: 4. /home/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0x1308139) [0x7f9a3a304139]

2020.08.24 12:24:50.433974 [ 4 ] <Error> BaseDaemon: 5. /home/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0x1038a97) [0x7f9a3a034a97]

2020.08.24 12:24:50.433981 [ 4 ] <Error> BaseDaemon: 6. /home/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0x107c95b) [0x7f9a3a07895b]

2020.08.24 12:24:50.433987 [ 4 ] <Error> BaseDaemon: 7. /home/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0xc1161f) [0x7f9a39c0d61f]

2020.08.24 12:24:50.433993 [ 4 ] <Error> BaseDaemon: 8. /home/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0xc2a31c) [0x7f9a39c2631c]

2020.08.24 12:24:50.434000 [ 4 ] <Error> BaseDaemon: 9. /lib64/libpthread.so.0(+0x7e25) [0x7f9a38559e25]

2020.08.24 12:25:22.156737 [ 5 ] <Error> BaseDaemon: ########################################

2020.08.24 12:25:22.156770 [ 5 ] <Error> BaseDaemon: (from thread 2) Received signal Segmentation fault (11).

2020.08.24 12:25:22.156778 [ 5 ] <Error> BaseDaemon: Address: 0x2f5

2020.08.24 12:25:22.156784 [ 5 ] <Error> BaseDaemon: Access: read.

2020.08.24 12:25:22.156794 [ 5 ] <Error> BaseDaemon: Address not mapped to object.

2020.08.24 12:25:22.160975 [ 5 ] <Error> BaseDaemon: 0. bin/tiflash/tiflash(DB::HandleCheckTerminated(DB::TiFlashServer*)+0xc) [0x66d44ec]

2020.08.24 12:25:22.160986 [ 5 ] <Error> BaseDaemon: 1. /home/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0x155769b) [0x7f4dfb23b69b]

2020.08.24 12:25:22.160998 [ 5 ] <Error> BaseDaemon: 2. /home/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(run_tiflash_proxy_ffi+0x4e5a) [0x7f4dfb2645aa]

2020.08.24 12:25:22.161006 [ 5 ] <Error> BaseDaemon: 3. bin/tiflash/tiflash() [0x30ee3de]

2020.08.24 12:25:22.161013 [ 5 ] <Error> BaseDaemon: 4. bin/tiflash/tiflash() [0x7a029cf]

2020.08.24 12:25:22.161020 [ 5 ] <Error> BaseDaemon: 5. /lib64/libpthread.so.0(+0x7e25) [0x7f4df9241e25]

2020.08.24 12:25:30.330425 [ 5 ] <Error> BaseDaemon: ########################################

2020.08.24 12:25:30.330463 [ 5 ] <Error> BaseDaemon: (from thread 6) Received signal Segmentation fault (11).

2020.08.24 12:25:30.330482 [ 5 ] <Error> BaseDaemon: Address: NULL pointer.

2020.08.24 12:25:30.330498 [ 5 ] <Error> BaseDaemon: Access: read.

2020.08.24 12:25:30.330515 [ 5 ] <Error> BaseDaemon: Address not mapped to object.

2020.08.24 12:25:30.340666 [ 5 ] <Error> BaseDaemon: 0. bin/tiflash/tiflash(DB::TMTContext::getContext()+0x1) [0x672a9b1]

2020.08.24 12:25:30.340680 [ 5 ] <Error> BaseDaemon: 1. bin/tiflash/tiflash(DB::HandleComputeFsStats(DB::TiFlashServer*)+0x27) [0x66d4687]

2020.08.24 12:25:30.340688 [ 5 ] <Error> BaseDaemon: 2. /home/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0xf79708) [0x7f4dfac5d708]

2020.08.24 12:25:30.340696 [ 5 ] <Error> BaseDaemon: 3. /home/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0x12fb186) [0x7f4dfafdf186]

2020.08.24 12:25:30.340703 [ 5 ] <Error> BaseDaemon: 4. /home/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0x1308139) [0x7f4dfafec139]

2020.08.24 12:25:30.340710 [ 5 ] <Error> BaseDaemon: 5. /home/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0x1038a97) [0x7f4dfad1ca97]

2020.08.24 12:25:30.340717 [ 5 ] <Error> BaseDaemon: 6. /home/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0x107c95b) [0x7f4dfad6095b]

2020.08.24 12:25:30.340724 [ 5 ] <Error> BaseDaemon: 7. /home/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0xc1161f) [0x7f4dfa8f561f]

2020.08.24 12:25:30.340730 [ 5 ] <Error> BaseDaemon: 8. /home/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0xc2a31c) [0x7f4dfa90e31c]

2020.08.24 12:25:30.340738 [ 5 ] <Error> BaseDaemon: 9. /lib64/libpthread.so.0(+0x7e25) [0x7f4df9241e25]

2020.08.24 12:25:59.172787 [ 5 ] <Error> BaseDaemon: ########################################

2020.08.24 12:25:59.172823 [ 5 ] <Error> BaseDaemon: (from thread 2) Received signal Segmentation fault (11).

2020.08.24 12:25:59.172840 [ 5 ] <Error> BaseDaemon: Address: 0x2f5

2020.08.24 12:25:59.172847 [ 5 ] <Error> BaseDaemon: Access: read.

2020.08.24 12:25:59.172856 [ 5 ] <Error> BaseDaemon: Address not mapped to object.

2020.08.24 12:25:59.177127 [ 5 ] <Error> BaseDaemon: 0. bin/tiflash/tiflash(DB::HandleCheckTerminated(DB::TiFlashServer*)+0xc) [0x66d44ec]

2020.08.24 12:25:59.177138 [ 5 ] <Error> BaseDaemon: 1. /home/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0x155769b) [0x7fa528a2669b]

2020.08.24 12:25:59.177152 [ 5 ] <Error> BaseDaemon: 2. /home/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(run_tiflash_proxy_ffi+0x4e5a) [0x7fa528a4f5aa]

2020.08.24 12:25:59.177160 [ 5 ] <Error> BaseDaemon: 3. bin/tiflash/tiflash() [0x30ee3de]

2020.08.24 12:25:59.177167 [ 5 ] <Error> BaseDaemon: 4. bin/tiflash/tiflash() [0x7a029cf]

2020.08.24 12:25:59.177174 [ 5 ] <Error> BaseDaemon: 5. /lib64/libpthread.so.0(+0x7e25) [0x7fa526a2ce25]

2020.08.24 12:26:07.340961 [ 5 ] <Error> BaseDaemon: ########################################

2020.08.24 12:26:07.340980 [ 5 ] <Error> BaseDaemon: (from thread 6) Received signal Segmentation fault (11).

2020.08.24 12:26:07.340987 [ 5 ] <Error> BaseDaemon: Address: NULL pointer.

2020.08.24 12:26:07.340993 [ 5 ] <Error> BaseDaemon: Access: read.

2020.08.24 12:26:07.340999 [ 5 ] <Error> BaseDaemon: Address not mapped to object.

2020.08.24 12:26:07.350874 [ 5 ] <Error> BaseDaemon: 0. bin/tiflash/tiflash(DB::TMTContext::getContext()+0x1) [0x672a9b1]

2020.08.24 12:26:07.350889 [ 5 ] <Error> BaseDaemon: 1. bin/tiflash/tiflash(DB::HandleComputeFsStats(DB::TiFlashServer*)+0x27) [0x66d4687]

2020.08.24 12:26:07.350897 [ 5 ] <Error> BaseDaemon: 2. /home/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0xf79708) [0x7fa528448708]

2020.08.24 12:26:07.350904 [ 5 ] <Error> BaseDaemon: 3. /home/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0x12fb186) [0x7fa5287ca186]

2020.08.24 12:26:07.350911 [ 5 ] <Error> BaseDaemon: 4. /home/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0x1308139) [0x7fa5287d7139]

2020.08.24 12:26:07.350930 [ 5 ] <Error> BaseDaemon: 5. /home/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0x1038a97) [0x7fa528507a97]

2020.08.24 12:26:07.350937 [ 5 ] <Error> BaseDaemon: 6. /home/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0x107c95b) [0x7fa52854b95b]

2020.08.24 12:26:07.350944 [ 5 ] <Error> BaseDaemon: 7. /home/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0xc1161f) [0x7fa5280e061f]

2020.08.24 12:26:07.350951 [ 5 ] <Error> BaseDaemon: 8. /home/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0xc2a31c) [0x7fa5280f931c]

2020.08.24 12:26:07.350958 [ 5 ] <Error> BaseDaemon: 9. /lib64/libpthread.so.0(+0x7e25) [0x7fa526a2ce25]

2020.08.24 12:26:36.422753 [ 5 ] <Error> BaseDaemon: ########################################

2020.08.24 12:26:36.422792 [ 5 ] <Error> BaseDaemon: (from thread 2) Received signal Segmentation fault (11).

2020.08.24 12:26:36.422808 [ 5 ] <Error> BaseDaemon: Address: 0x2f5

2020.08.24 12:26:36.422815 [ 5 ] <Error> BaseDaemon: Access: read.

2020.08.24 12:26:36.422822 [ 5 ] <Error> BaseDaemon: Address not mapped to object.

2020.08.24 12:26:36.427006 [ 5 ] <Error> BaseDaemon: 0. bin/tiflash/tiflash(DB::HandleCheckTerminated(DB::TiFlashServer*)+0xc) [0x66d44ec]

2020.08.24 12:26:36.427019 [ 5 ] <Error> BaseDaemon: 1. /home/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0x155769b) [0x7f4dd8c5d69b]

2020.08.24 12:26:36.427033 [ 5 ] <Error> BaseDaemon: 2. /home/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(run_tiflash_proxy_ffi+0x4e5a) [0x7f4dd8c865aa]

2020.08.24 12:26:36.427041 [ 5 ] <Error> BaseDaemon: 3. bin/tiflash/tiflash() [0x30ee3de]

2020.08.24 12:26:36.427048 [ 5 ] <Error> BaseDaemon: 4. bin/tiflash/tiflash() [0x7a029cf]

2020.08.24 12:26:36.427055 [ 5 ] <Error> BaseDaemon: 5. /lib64/libpthread.so.0(+0x7e25) [0x7f4dd6c63e25]

2020.08.24 12:26:44.597791 [ 5 ] <Error> BaseDaemon: ########################################

2020.08.24 12:26:44.597827 [ 5 ] <Error> BaseDaemon: (from thread 6) Received signal Segmentation fault (11).

2020.08.24 12:26:44.597888 [ 5 ] <Error> BaseDaemon: Address: NULL pointer.

2020.08.24 12:26:44.597904 [ 5 ] <Error> BaseDaemon: Access: read.

2020.08.24 12:26:44.597921 [ 5 ] <Error> BaseDaemon: Address not mapped to object.

2020.08.24 12:26:44.608951 [ 5 ] <Error> BaseDaemon: 0. bin/tiflash/tiflash(DB::TMTContext::getContext()+0x1) [0x672a9b1]

2020.08.24 12:26:44.608992 [ 5 ] <Error> BaseDaemon: 1. bin/tiflash/tiflash(DB::HandleComputeFsStats(DB::TiFlashServer*)+0x27) [0x66d4687]

2020.08.24 12:26:44.609014 [ 5 ] <Error> BaseDaemon: 2. /home/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0xf79708) [0x7f4dd867f708]

2020.08.24 12:26:44.609041 [ 5 ] <Error> BaseDaemon: 3. /home/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0x12fb186) [0x7f4dd8a01186]

2020.08.24 12:26:44.609061 [ 5 ] <Error> BaseDaemon: 4. /home/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0x1308139) [0x7f4dd8a0e139]

2020.08.24 12:26:44.609079 [ 5 ] <Error> BaseDaemon: 5. /home/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0x1038a97) [0x7f4dd873ea97]

2020.08.24 12:26:44.609098 [ 5 ] <Error> BaseDaemon: 6. /home/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0x107c95b) [0x7f4dd878295b]

2020.08.24 12:26:44.609117 [ 5 ] <Error> BaseDaemon: 7. /home/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0xc1161f) [0x7f4dd831761f]

2020.08.24 12:26:44.609136 [ 5 ] <Error> BaseDaemon: 8. /home/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0xc2a31c) [0x7f4dd833031c]

2020.08.24 12:26:44.609174 [ 5 ] <Error> BaseDaemon: 9. /lib64/libpthread.so.0(+0x7e25) [0x7f4dd6c63e25]

2020.08.24 12:27:13.666793 [ 5 ] <Error> BaseDaemon: ########################################

2020.08.24 12:27:13.666846 [ 5 ] <Error> BaseDaemon: (from thread 2) Received signal Segmentation fault (11).

2020.08.24 12:27:13.666857 [ 5 ] <Error> BaseDaemon: Address: 0x2f5

2020.08.24 12:27:13.666866 [ 5 ] <Error> BaseDaemon: Access: read.

2020.08.24 12:27:13.666886 [ 5 ] <Error> BaseDaemon: Address not mapped to object.

2020.08.24 12:27:13.671432 [ 5 ] <Error> BaseDaemon: 0. bin/tiflash/tiflash(DB::HandleCheckTerminated(DB::TiFlashServer*)+0xc) [0x66d44ec]

2020.08.24 12:27:13.671455 [ 5 ] <Error> BaseDaemon: 1. /home/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(+0x155769b) [0x7fe3f0fbf69b]

2020.08.24 12:27:13.671467 [ 5 ] <Error> BaseDaemon: 2. /home/tidb-deploy/tiflash-9000/bin/tiflash/libtiflash_proxy.so(run_tiflash_proxy_ffi+0x4e5a) [0x7fe3f0fe85aa]

2020.08.24 12:27:13.671474 [ 5 ] <Error> BaseDaemon: 3. bin/tiflash/tiflash() [0x30ee3de]

2020.08.24 12:27:13.671481 [ 5 ] <Error> BaseDaemon: 4. bin/tiflash/tiflash() [0x7a029cf]

2020.08.24 12:27:13.671489 [ 5 ] <Error> BaseDaemon: 5. /lib64/libpthread.so.0(+0x7e25) [0x7fe3eefc5e25]

2020.08.24 12:27:21.844130 [ 5 ] <Error> BaseDaemon: ########################################

2020.08.24 12:27:21.844147 [ 5 ] <Error> BaseDaemon: (from thread 6) Received signal Segmentation fault (11).

2020.08.24 12:27:21.844154 [ 5 ] <Error> BaseDaemon: Address: NULL pointer.

2020.08.24 12:27:21.844160 [ 5 ] <Error> BaseDaemon: Access: read.

2020.08.24 12:27:21.844173 [ 5 ] <Error> BaseDaemon: Address not mapped to object.

[来了老弟]

你好，
可以提供下 tidb 的版本，是 scale out 还是 deploy 部署的 tiflash 节点
通过 pd-ctl 自行检查下 pd placement rules 参数是否开启。
将 tiflash deploy dir log 下的文件打包上传下，感谢

[qwewer123234]

你好，是通过scale out部署的，之前直接通过deploy也是down的状态,附件为Log 以及部署和scale out模板，谢谢
placement rules 为true
pd 172.16.200.213:2379 replication.enable-placement-rules true
Log.rar (691.1 KB)

[来了老弟]

hi，store 的信息和 tiup display 也反馈下，感谢

[qwewer123234]

你好，附件是store及display信息，谢谢

store.txt (3.8 KB)

[JaySon-Huang (JaySon-PingCAP)]

请检查一下 172.16.200.214 机器的 9000 端口是否被其他进程占用了。lsof -i :9000

[qwewer123234]

你好，9000端口有一个cslistener，sdw-1.keboda.com是172.16.200.213的域名，请问这个是TIDB自带的还是？谢谢

[JaySon-Huang (JaySon-PingCAP)]

process - What is cslistener? - Ask Ubuntu cslistener 应该是服务器上运行的其他进程。可以用 PID ps -ef | grep 1920 查询，确认其他进程的信息。TiFlash 一直为down是因为在尝试监听 9000 端口时因为绑定失败而退出了。

如果不能停止 9000 端口的服务，可以用 tiup 先对 TiFlash 节点进行缩容，然后重新扩容，并修改 TiFlash 绑定的 tcp_port 端口。最好部署前先确认端口没有冲突。
https://docs.pingcap.com/zh/tidb/stable/scale-tidb-using-tiup

请问tiflash 端口相关文档在哪里？

请问一下tiflash使用了哪些端口那个文档在哪里？我找了很久没找到，以前见过的

[qwewer123234]

可以了，解决了，感谢

[yilong (yi888long)]