【 TiDB 使用环境】V5.1.0
【概述】tiflash down重启依然显示离线
【背景】利用dm同步数据到tidb，但未做tiflash同步
【现象】业务和数据库现象
【业务影响】
【TiDB 版本】
【附件】
- 相关日志 和 监控
- TiUP Cluster Display 信息
ID Role Host Ports OS/Arch Status Data Dir Deploy Dir
10.10.20.82:9093 alertmanager 10.10.20.82 9093/9094 linux/x86_64 Up /data/tidb-data/alertmanager-9093 /data/tidb-deploy/alertmanager-9093
10.10.20.82:3000 grafana 10.10.20.82 3000 linux/x86_64 Up - /data/tidb-deploy/grafana-3000
10.10.20.82:2379 pd 10.10.20.82 2379/2380 linux/x86_64 Up /data/tidb-data/pd-2379 /data/tidb-deploy/pd-2379
10.10.20.83:2379 pd 10.10.20.83 2379/2380 linux/x86_64 Up|L|UI /data/tidb-data/pd-2379 /data/tidb-deploy/pd-2379
10.10.20.85:2379 pd 10.10.20.85 2379/2380 linux/x86_64 Up /data/deploy/install/data/pd-2379 /data/deploy/install/deploy/pd-2379
10.10.20.82:9090 prometheus 10.10.20.82 9090 linux/x86_64 Up /data/tidb-data/prometheus-9090 /data/tidb-deploy/prometheus-9090
10.10.20.81:4000 tidb 10.10.20.81 4000/10080 linux/x86_64 Up - /data/deploy/install/deploy/tidb-4000
10.10.20.82:4000 tidb 10.10.20.82 4000/10080 linux/x86_64 Up - /data/tidb-deploy/tidb-4000
10.10.20.83:4000 tidb 10.10.20.83 4000/10080 linux/x86_64 Up - /data/tidb-deploy/tidb-4000
10.10.20.83:9000 tiflash 10.10.20.83 9000/8123/3930/20170/20292/8234 linux/x86_64 Down /data/tidb-data/tiflash-9000 /data/tidb-deploy/tiflash-9000
10.10.20.87:9000 tiflash 10.10.20.87 9000/8123/3930/20170/20292/8234 linux/x86_64 Up /data/tidb-data/tiflash-9000 /data/tidb-deploy/tiflash-9000
10.10.20.81:20160 tikv 10.10.20.81 20160/20180 linux/x86_64 Up /data/deploy/install/data/tikv-20160 /data/deploy/install/deploy/tikv-20160
10.10.20.82:20160 tikv 10.10.20.82 20160/20180 linux/x86_64 Up /data/tidb-data/tikv-20160 /data/tidb-deploy/tikv-20160
10.10.20.86:20160 tikv 10.10.20.86 20160/20180 linux/x86_64 Up /data/deploy/install/data/tikv-20160 /data/deploy/install/deploy/tikv-20160
10.10.20.82:7077 tispark-master 10.10.20.82 7077/8080 linux/x86_64 Up - /data/tidb-deploy/tispark-master-7077
10.10.20.83:7078 tispark-worker 10.10.20.83 7078/8081 linux/x86_64 Up - /data/tidb-deploy/tispark-worker-7078
-
TiUP Cluster Edit Config 信息
-
TiDB- Overview 监控
- 对应模块日志（包含问题前后1小时日志）
[2021/07/30 04:41:48.076 +08:00] [ERROR] [TiFlashManager] [can not report replica sync status to tidb: [(‘10.10.20.82:10080’, Exception(‘unexpected status code 400 from http://10.10.20.82:10080/tiflash/replica’,)), (‘10.10.20.83:10080’, Exception(‘unexpected status code 400 from http://10.10.20.83:10080/tiflash/replica’,)), (‘10.10.20.81:10080’, Exception(‘unexpected status code 400 from http://10.10.20.81:10080/tiflash/replica’,))]]
[2021/07/30 04:41:51.956 +08:00] [INFO] [TiFlashManager] [report to tidb: id 82276, region_count 1, flash_region_count 1]
[2021/07/30 04:43:34.278 +08:00] [ERROR] [TiFlashManager] [can not report replica sync status to tidb: [(‘10.10.20.82:10080’, ReadTimeout(ReadTimeoutError(“HTTPConnectionPool(host=‘10.10.20.82’, port=10080): Read timed out. (read timeout=60)”,),)), (‘10.10.20.83:10080’, Exception(‘unexpected status code 400 from http://10.10.20.83:10080/tiflash/replica’,)), (‘10.10.20.81:10080’, Exception(‘unexpected status code 400 from http://10.10.20.81:10080/tiflash/replica’,))]]
若提问为性能优化、故障排查类问题，请下载脚本运行。终端输出的打印结果，请务必全选并复制粘贴上传。