tidb 4.0.11版本tikv下线一直没反应

🪐 TiDB 技术问题
1

使用tiup全新部署tidb v4.0.11集群，使用tiup缩容下线一个tikv节点，一直没反应

QQ%E6%88%AA%E5%9B%BE20210423151553
QQ%E6%88%AA%E5%9B%BE202104231515531268×618 29.9 KB

2

可以通过 pd ctl 查看一下当前 store 情况，看一下是否 region 还没迁移完成。
https://docs.pingcap.com/zh/tidb/stable/tiup-cluster#集群控制工具-controllers

3

看起来是迁移完成了。以下是执行的结果输出。
tiup ctl pd -u http://127.0.0.1:2379 store
Starting component ctl: /root/.tiup/components/ctl/v4.0.11/ctl pd -u http://127.0.0.1:2379 store
{
“count”: 5,
“stores”: [
{
“store”: {
“id”: 1,
“address”: “10.3.10.248:20160”,
“labels”: [
{
“key”: “host”,
“value”: “logic-host-1”
}
],
“version”: “4.0.11”,
“status_address”: “10.3.10.248:20180”,
“git_hash”: “4ac5e7ea1839d63163e911e2e1164d663f49592b”,
“start_timestamp”: 1619142264,
“deploy_path”: “/home/tidb/tidb-deploy/tikv-20160/bin”,
“last_heartbeat”: 1619163377523712938,
“state_name”: “Up”
},
“status”: {
“capacity”: “440GiB”,
“available”: “279.8GiB”,
“used_size”: “85.88GiB”,
“leader_count”: 5262,
“leader_weight”: 1,
“leader_score”: 5262,
“leader_size”: 346393,
“region_count”: 10516,
“region_weight”: 1,
“region_score”: 685079,
“region_size”: 685079,
“start_ts”: “2021-04-23T09:44:24+08:00”,
“last_heartbeat_ts”: “2021-04-23T15:36:17.523712938+08:00”,
“uptime”: “5h51m53.523712938s”
}
},
{
“store”: {
“id”: 4,
“address”: “10.3.10.249:20161”,
“labels”: [
{
“key”: “host”,
“value”: “logic-host-3”
}
],
“version”: “4.0.11”,
“status_address”: “10.3.10.249:20181”,
“git_hash”: “4ac5e7ea1839d63163e911e2e1164d663f49592b”,
“start_timestamp”: 1619158955,
“deploy_path”: “/home/tidb/tidb-deploy/tikv-20161/bin”,
“last_heartbeat”: 1619163374825598644,
“state_name”: “Up”
},
“status”: {
“capacity”: “1.832TiB”,
“available”: “568.7GiB”,
“used_size”: “85.78GiB”,
“leader_count”: 5254,
“leader_weight”: 1,
“leader_score”: 5254,
“leader_size”: 338686,
“region_count”: 10516,
“region_weight”: 1,
“region_score”: 685079,
“region_size”: 685079,
“start_ts”: “2021-04-23T14:22:35+08:00”,
“last_heartbeat_ts”: “2021-04-23T15:36:14.825598644+08:00”,
“uptime”: “1h13m39.825598644s”
}
},
{
“store”: {
“id”: 5,
“address”: “10.3.10.249:20160”,
“state”: 1,
“labels”: [
{
“key”: “host”,
“value”: “logic-host-2”
}
],
“version”: “4.0.11”,
“status_address”: “10.3.10.249:20180”,
“git_hash”: “4ac5e7ea1839d63163e911e2e1164d663f49592b”,
“start_timestamp”: 1619163360,
“deploy_path”: “/home/tidb/tidb-deploy/tikv-20160/bin”,
“last_heartbeat”: 1619141409567036060,
“state_name”: “Offline”
},
“status”: {
“capacity”: “0B”,
“available”: “0B”,
“used_size”: “0B”,
“leader_count”: 0,
“leader_weight”: 1,
“leader_score”: 0,
“leader_size”: 0,
“region_count”: 10516,
“region_weight”: 1,
“region_score”: 685079,
“region_size”: 685079,
“start_ts”: “2021-04-23T15:36:00+08:00”,
“last_heartbeat_ts”: “2021-04-23T09:30:09.56703606+08:00”
}
},
{
“store”: {
“id”: 46,
“address”: “10.3.10.248:3930”,
“labels”: [
{
“key”: “engine”,
“value”: “tiflash”
}
],
“version”: “v4.0.11”,
“peer_address”: “10.3.10.248:20170”,
“status_address”: “10.3.10.248:20292”,
“git_hash”: “c17af8a365ca145c18ee0e38945a1f1956799acb”,
“start_timestamp”: 1619142288,
“deploy_path”: “/home/tidb/tidb-deploy/tiflash-9000/bin/tiflash”,
“last_heartbeat”: 1619163380682715230,
“state_name”: “Up”
},
“status”: {
“capacity”: “440GiB”,
“available”: “429.9GiB”,
“used_size”: “10.06GiB”,
“leader_count”: 0,
“leader_weight”: 1,
“leader_score”: 0,
“leader_size”: 0,
“region_count”: 587,
“region_weight”: 1,
“region_score”: 55818,
“region_size”: 55818,
“start_ts”: “2021-04-23T09:44:48+08:00”,
“last_heartbeat_ts”: “2021-04-23T15:36:20.68271523+08:00”,
“uptime”: “5h51m32.68271523s”
}
},
{
“store”: {
“id”: 47,
“address”: “10.3.10.249:3930”,
“labels”: [
{
“key”: “engine”,
“value”: “tiflash”
}
],
“version”: “v4.0.11”,
“peer_address”: “10.3.10.249:20170”,
“status_address”: “10.3.10.249:20292”,
“git_hash”: “c17af8a365ca145c18ee0e38945a1f1956799acb”,
“start_timestamp”: 1619142283,
“deploy_path”: “/home/tidb/tidb-deploy/tiflash-9000/bin/tiflash”,
“last_heartbeat”: 1619163380514785630,
“state_name”: “Up”
},
“status”: {
“capacity”: “1.832TiB”,
“available”: “1.813TiB”,
“used_size”: “20.25GiB”,
“leader_count”: 0,
“leader_weight”: 1,
“leader_score”: 0,
“leader_size”: 0,
“region_count”: 608,
“region_weight”: 1,
“region_score”: 64669,
“region_size”: 64669,
“start_ts”: “2021-04-23T09:44:43+08:00”,
“last_heartbeat_ts”: “2021-04-23T15:36:20.51478563+08:00”,
“uptime”: “5h51m37.51478563s”
}
}
]
}

4

tikv 进程还存活着么

5

进程不在了，但是日志还在不停的写入

6

把进程重新拉起来一下，下线过程中需要 tikv 进程存活着的

7

进程起不来，一直报错
[2021/04/23 15:53:41.869 +08:00] [INFO] [mod.rs:335] [“starting working thread”] [worker=consistency-check]
[2021/04/23 15:53:41.869 +08:00] [WARN] [store.rs:1272] [“set thread priority for raftstore failed”] [error=“Os { code: 13, kind: PermissionDenied, message: “Permission denied” }”]
[2021/04/23 15:53:41.870 +08:00] [INFO] [node.rs:173] [“put store to PD”] [store=“id: 5 address: “10.3.10.249:20160” labels { key: “host” value: “logic-host-2” } version: “4.0.11” status_address: “10.3.10.249:20180” git_hash: “4ac5e7ea1839d63163e911e2e1164d663f49592b” start_timestamp: 1619164421 deploy_path: “/home/tidb/tidb-deploy/tikv-20160/bin””]
[2021/04/23 15:53:41.880 +08:00] [INFO] [mod.rs:335] [“starting working thread”] [worker=cdc]
[2021/04/23 15:53:41.881 +08:00] [INFO] [future.rs:136] [“starting working thread”] [worker=waiter-manager]
[2021/04/23 15:53:41.882 +08:00] [INFO] [future.rs:136] [“starting working thread”] [worker=deadlock-detector]
[2021/04/23 15:53:41.882 +08:00] [INFO] [mod.rs:335] [“starting working thread”] [worker=backup-endpoint]
[2021/04/23 15:53:41.882 +08:00] [INFO] [mod.rs:335] [“starting working thread”] [worker=snap-handler]
[2021/04/23 15:53:41.883 +08:00] [INFO] [server.rs:224] [“listening on addr”] [addr=0.0.0.0:20160]
[2021/04/23 15:53:41.897 +08:00] [INFO] [kv.rs:577] [“batch_raft RPC is called, new gRPC stream established”]
[2021/04/23 15:53:41.898 +08:00] [INFO] [transport.rs:144] [“resolve store address ok”] [addr=10.3.10.248:20160] [store_id=1]
[2021/04/23 15:53:41.898 +08:00] [INFO] [raft_client.rs:48] [“server: new connection with tikv endpoint”] [addr=10.3.10.248:20160]
[2021/04/23 15:53:41.899 +08:00] [INFO] [transport.rs:144] [“resolve store address ok”] [addr=10.3.10.248:20160] [store_id=1]
[2021/04/23 15:53:41.899 +08:00] [INFO] [] [“New connected subchannel at 0x7efbe9c073c0 for subchannel 0x7efc570681c0”]
[2021/04/23 15:53:41.900 +08:00] [INFO] [server.rs:261] [“TiKV is ready to serve”]
[2021/04/23 15:53:41.901 +08:00] [WARN] [mod.rs:489] [“failed to register addr to pd”] [body=Body(Streaming)] [“status code”=400]
[2021/04/23 15:53:41.901 +08:00] [INFO] [util.rs:419] [“connecting to PD endpoint”] [endpoints=http://10.3.10.248:2379]
[2021/04/23 15:53:41.902 +08:00] [INFO] [util.rs:419] [“connecting to PD endpoint”] [endpoints=http://10.3.10.248:2379]
[2021/04/23 15:53:41.903 +08:00] [INFO] [util.rs:484] [“connected to PD leader”] [endpoints=http://10.3.10.248:2379]
[2021/04/23 15:53:41.903 +08:00] [INFO] [util.rs:190] [“heartbeat sender and receiver are stale, refreshing …”]
[2021/04/23 15:53:41.911 +08:00] [WARN] [util.rs:209] [“updating PD client done”] [spend=9.327303ms]
[2021/04/23 15:53:41.911 +08:00] [WARN] [mod.rs:489] [“failed to register addr to pd”] [body=Body(Streaming)] [“status code”=400]
[2021/04/23 15:53:41.911 +08:00] [WARN] [mod.rs:489] [“failed to register addr to pd”] [body=Body(Streaming)] [“status code”=400]
[2021/04/23 15:53:41.912 +08:00] [WARN] [mod.rs:489] [“failed to register addr to pd”] [body=Body(Streaming)] [“status code”=400]
[2021/04/23 15:53:41.912 +08:00] [WARN] [mod.rs:489] [“failed to register addr to pd”] [body=Body(Streaming)] [“status code”=400]
[2021/04/23 15:53:41.912 +08:00] [WARN] [mod.rs:499] [“failed to register addr to pd after 5 tries”]
[2021/04/23 15:53:42.057 +08:00] [INFO] [kv.rs:577] [“batch_raft RPC is called, new gRPC stream established”]
[2021/04/23 15:53:42.058 +08:00] [INFO] [transport.rs:144] [“resolve store address ok”] [addr=10.3.10.249:20161] [store_id=4]
[2021/04/23 15:53:42.058 +08:00] [INFO] [raft_client.rs:48] [“server: new connection with tikv endpoint”] [addr=10.3.10.249:20161]
[2021/04/23 15:53:42.058 +08:00] [INFO] [] [“New connected subchannel at 0x7efbe98192d0 for subchannel 0x7efc57068700”]
[2021/04/23 15:53:43.130 +08:00] [WARN] [apply.rs:546] ["[store 5] handle ready 4 committed entries"] [takes=1077]
[2021/04/23 15:53:43.131 +08:00] [WARN] [store.rs:645] ["[store 5] handle 1 pending peers include 1 ready, 4 entries, 1 messages and 0 snapshots"] [takes=1073]
[2021/04/23 15:53:43.131 +08:00] [WARN] [store.rs:645] ["[store 5] handle 1 pending peers include 1 ready, 2 entries, 1 messages and 0 snapshots"] [takes=1013]
[2021/04/23 15:53:43.131 +08:00] [ERROR] [sst_importer.rs:92] [“ingest failed”] [err_code=KV:Engine:Engine] [err=“Engine Engine(“Corruption: An external sst file with version 2 have global seqno property with value �x�l, while largest seqno in the file is 0”)”] [meta=“uuid: 0FD6EF08BF704F4B9548F3E9D53A7183 range { start: 7480000000000005FFA95F698000000000FF0000030146533138FF32373030FF303630FF3036310000FD0000FD end: 7480000000000005FFA95F698000000000FF0000030147583138FF31393349FF313034FF335F33315FFF3230FF000000000000F900FE } cf_name: “write” region_id: 47767 region_epoch { conf_ver: 5 version: 4415 }”]

8

在使用lighting工具导入数据的过程中，这个tikv节点挂了

9

tiup cluster scale-in --force 将这个节点下线吧

10

强制下线了，tiup cluster display已经不显示那个节点了，但是dashborad上面还有显示，这个要怎么清除掉

11

全新集群都是这个样子，无法下线tikv,感觉是版本问题

12

dashboard 页面信息是还有什么？一般会有一个垃圾桶的图标可以点删除