Tikv节点释放问题

系统版本: CentOS Linux release 7.3.1611 (Core) tidb 版本:2.1.14 tidb集群服务器列表: 192.168.144.199 (ansible+monitor) 10.111.9.248 (kv1) 10.111.22.39(kv2) 10.111.9.251(kv3) 10.111.22.49(kv4) 10.111.10.7(kv5) 10.111.9.247(pd1) 10.111.22.40(pd2) 10.111.9.253(pd3) 10.111.9.254(tidb1) 10.111.22.41(tidb2)

中控机执行 ./pd-ctl -u “http://10.111.9.247:2379” -d store delete 11990 ,对应节点信息

{ “store”: { “id”: 11990, “address”: “10.111.9.248:20160”, “state”: 1, “version”: “2.1.14”, “state_name”: “Offline” }, “status”: { “capacity”: “504 GiB”, “available”: “502 GiB”, “leader_weight”: 1, “region_count”: 8639, “region_weight”: 1, “region_score”: 18690, “region_size”: 18690, “start_ts”: “2019-11-07T19:26:55+08:00”, “last_heartbeat_ts”: “2019-11-08T11:03:40.096119862+08:00”, “uptime”: “15h36m45.096119862s” } }, 这个状态保持了差不多12个小时,

查看 tikv节点日志: tail -f /datadisk1/tidb/deploy/log/tikv.log 2019/11/08 07:41:51.283 INFO apply.rs:922: [region 26] 23576 execute admin command cmd_type: CompactLog compact_log {compact_index: 23738 compact_term: 37} at [term: 37, index: 23740] 2019/11/08 08:02:50.786 INFO apply.rs:922: [region 26] 23576 execute admin command cmd_type: CompactLog compact_log {compact_index: 23793 compact_term: 37} at [term: 37, index: 23795] 2019/11/08 08:23:46.083 INFO apply.rs:922: [region 26] 23576 execute admin command cmd_type: CompactLog compact_log {compact_index: 23844 compact_term: 37} at [term: 37, index: 23846] 2019/11/08 08:44:44.384 INFO apply.rs:922: [region 26] 23576 execute admin command cmd_type: CompactLog compact_log {compact_index: 23895 compact_term: 37} at [term: 37, index: 23897] 2019/11/08 09:05:51.483 INFO apply.rs:922: [region 26] 23576 execute admin command cmd_type: CompactLog compact_log {compact_index: 23946 compact_term: 37} at [term: 37, index: 23948] 2019/11/08 09:26:50.284 INFO apply.rs:922: [region 26] 23576 execute admin command cmd_type: CompactLog compact_log {compact_index: 23997 compact_term: 37} at [term: 37, index: 23999] 2019/11/08 09:47:48.684 INFO apply.rs:922: [region 26] 23576 execute admin command cmd_type: CompactLog compact_log {compact_index: 24048 compact_term: 37} at [term: 37, index: 24050] 2019/11/08 10:08:45.885 INFO apply.rs:922: [region 26] 23576 execute admin command cmd_type: CompactLog compact_log {compact_index: 24099 compact_term: 37} at [term: 37, index: 24101] 2019/11/08 10:29:44.285 INFO apply.rs:922: [region 26] 23576 execute admin command cmd_type: CompactLog compact_log {compact_index: 24150 compact_term: 37} at [term: 37, index: 24152] 2019/11/08 10:50:52.084 INFO apply.rs:922: [region 26] 23576 execute admin command cmd_type: CompactLog compact_log {compact_index: 24201 compact_term: 37} at [term: 37, index: 24203]

tail -f tikv_stderr.log E1104 20:56:12.330419899 10304 chttp2_transport.c:1143] Received a GOAWAY with error code ENHANCE_YOUR_CALM and debug data equal to “too_many_pings” E1104 20:56:12.331440693 10304 chttp2_transport.c:1143] Received a GOAWAY with error code ENHANCE_YOUR_CALM and debug data equal to “too_many_pings” E1104 20:56:12.331423790 10305 chttp2_transport.c:1143] Received a GOAWAY with error code ENHANCE_YOUR_CALM and debug data equal to “too_many_pings” E1104 20:56:12.331457469 10306 chttp2_transport.c:1143] Received a GOAWAY with error code ENHANCE_YOUR_CALM and debug data equal to “too_many_pings” E1104 20:56:12.331451172 10307 chttp2_transport.c:1143] Received a GOAWAY with error code ENHANCE_YOUR_CALM and debug data equal to “too_many_pings” E1104 20:56:12.331510468 10307 chttp2_transport.c:1143] Received a GOAWAY with error code ENHANCE_YOUR_CALM and debug data equal to “too_many_pings” E1104 20:56:12.331623498 10304 chttp2_transport.c:1143] Received a GOAWAY with error code ENHANCE_YOUR_CALM and debug data equal to “too_many_pings” E1104 20:56:12.331625958 10306 chttp2_transport.c:1143] Received a GOAWAY with error code ENHANCE_YOUR_CALM and debug data equal to “too_many_pings” E1104 20:56:12.331663638 10306 chttp2_transport.c:1143] Received a GOAWAY with error code ENHANCE_YOUR_CALM and debug data equal to “too_many_pings” E1104 20:56:12.332365792 10306 chttp2_transport.c:1143] Received a GOAWAY with error code ENHANCE_YOUR_CALM and debug data equal to “too_many_pings”

您好: 当前你的集群有大量的读写吗? 麻烦把tikv完整日志上传下,多谢

集群是空闲状态,完整日志是停掉的那个节点的日志吗

你好: 是的,麻烦上传下日志,多谢

tikv.tar.gz (2.0 MB)

已上传完整日志, 444 2019-11-07 19:27:15 tidb ./pd-ctl -u “http://10.111.9.247:2379” -d store delete 11990,操作时间

你好:请参考这篇博客

请反馈pd调度监控,多谢