集群中一个kv节点下线时，出现其他有kv节点无法启动

abcd · 2021 年4 月 23 日 06:32

现在147和236已经up了，在此之前强制下线了38546296节点之后，按照文档在每个kv节点上执行了unsafe-recover 38546296

abcd · 2021 年4 月 23 日 06:36

打开tidb.log，执行会产生region is unavaible时的日志如下，由于字数限制，截取了部分

[2021/04/23 06:31:03.631 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:345.38219ms txnStartTS:424452921765658625 region_id:17258 store_addr:10.12.5.236:20160 kv_process_ms:344 scan_total_write:608672 scan_processed_write:608671 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:04.037 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:750.92044ms txnStartTS:424452921765658625 region_id:16104 store_addr:10.12.5.236:20160 kv_process_ms:748 scan_total_write:684810 scan_processed_write:684809 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:04.301 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:1.014316959s txnStartTS:424452921765658625 region_id:41298 store_addr:10.12.5.236:20160 kv_process_ms:1012 scan_total_write:1154902 scan_processed_write:1154901 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:04.386 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:1.099985818s txnStartTS:424452921765658625 region_id:10715 store_addr:10.12.5.239:20160 kv_process_ms:1056 scan_total_write:606255 scan_processed_write:606254 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:05.889 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:2.602951555s txnStartTS:424452921765658625 region_id:3953 store_addr:10.12.5.13:20160 kv_process_ms:2600 scan_total_write:497533 scan_processed_write:497532 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:06.670 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:2.369059348s txnStartTS:424452921765658625 region_id:11305 store_addr:10.12.5.13:20160 kv_process_ms:2367 scan_total_write:484696 scan_processed_write:484695 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:07.187 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:3.900343519s txnStartTS:424452921765658625 region_id:40651 store_addr:10.12.5.147:20160 kv_process_ms:3896 scan_total_write:1196679 scan_processed_write:1196678 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:07.466 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:4.180302073s txnStartTS:424452921765658625 region_id:17041 store_addr:10.12.5.240:20160 kv_process_ms:4180 scan_total_write:614075 scan_processed_write:614074 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:07.481 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:4.194580661s txnStartTS:424452921765658625 region_id:17093 store_addr:10.12.5.240:20160 kv_process_ms:4192 scan_total_write:623139 scan_processed_write:623138 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:07.511 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:4.225310388s txnStartTS:424452921765658625 region_id:1783 store_addr:10.12.5.13:20160 kv_process_ms:4223 scan_total_write:1006898 scan_processed_write:1006897 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:07.790 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:4.157903179s txnStartTS:424452921765658625 region_id:13654 store_addr:10.12.5.13:20160 kv_process_ms:4155 scan_total_write:992518 scan_processed_write:992517 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:07.813 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:4.527201304s txnStartTS:424452921765658625 region_id:13006 store_addr:10.12.5.147:20160 kv_process_ms:4524 scan_total_write:1113818 scan_processed_write:1113817 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:08.057 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:590.458756ms txnStartTS:424452921765658625 region_id:15284 store_addr:10.12.5.119:20160 kv_process_ms:584 scan_total_write:759566 scan_processed_write:759565 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:08.099 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:4.061518368s txnStartTS:424452921765658625 region_id:51800 store_addr:10.12.5.13:20160 kv_process_ms:3859 scan_total_write:954995 scan_processed_write:954994 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:08.188 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:4.902064087s txnStartTS:424452921765658625 region_id:13671 store_addr:10.12.5.240:20160 kv_process_ms:4900 scan_total_write:1000177 scan_processed_write:1000176 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:08.282 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:4.995192877s txnStartTS:424452921765658625 region_id:37249 store_addr:10.12.5.147:20160 kv_process_ms:4996 scan_total_write:1221367 scan_processed_write:1221366 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:08.349 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:5.063297619s txnStartTS:424452921765658625 region_id:40072 store_addr:10.12.5.147:20160 kv_process_ms:5064 scan_total_write:1202867 scan_processed_write:1202866 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:08.421 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:363.486715ms txnStartTS:424452921765658625 region_id:18511 store_addr:10.12.5.119:20160 kv_process_ms:360 scan_total_write:570219 scan_processed_write:570218 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:08.573 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:4.186804994s txnStartTS:424452921765658625 region_id:55684 store_addr:10.12.5.147:20160 kv_process_ms:4188 scan_total_write:1009696 scan_processed_write:1009695 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:09.295 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:1.481520243s txnStartTS:424452921765658625 region_id:49825 store_addr:10.12.5.13:20160 kv_process_ms:1480 scan_total_write:1122827 scan_processed_write:1122826 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:09.671 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:6.38508773s txnStartTS:424452921765658625 region_id:57623 store_addr:10.12.5.240:20160 kv_process_ms:6384 scan_total_write:959923 scan_processed_write:959922 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:10.067 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:3.397001619s txnStartTS:424452921765658625 region_id:59083 store_addr:10.12.5.239:20160 kv_process_ms:3392 scan_total_write:974521 scan_processed_write:974520 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:10.108 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:437.162054ms txnStartTS:424452921765658625 region_id:15887 store_addr:10.12.5.119:20160 kv_process_ms:436 scan_total_write:699583 scan_processed_write:699582 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:10.684 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:616.92997ms txnStartTS:424452921765658625 region_id:12906 store_addr:10.12.5.236:20160 kv_process_ms:616 scan_total_write:1111528 scan_processed_write:1111527 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:10.778 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:3.590591535s txnStartTS:424452921765658625 region_id:20895 store_addr:10.12.5.240:20160 kv_process_ms:3592 scan_total_write:489390 scan_processed_write:489389 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:11.581 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:5.407787626s txnStartTS:424452921765658625 region_id:3460 store_addr:10.12.5.240:20160 kv_process_ms:5400 scan_total_write:787149 scan_processed_write:787148 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:11.848 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:2.552111662s txnStartTS:424452921765658625 region_id:21803 store_addr:10.12.5.13:20160 kv_process_ms:2532 scan_total_write:1358881 scan_processed_write:1358880 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:12.019 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:3.73753809s txnStartTS:424452921765658625 region_id:18627 store_addr:10.12.5.147:20160 kv_process_ms:3735 scan_total_write:570423 scan_processed_write:570422 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:12.170 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:8.883615792s txnStartTS:424452921765658625 region_id:22118 store_addr:10.12.5.147:20160 kv_process_ms:8884 scan_total_write:1328778 scan_processed_write:1328777 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:12.564 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:3.990546932s txnStartTS:424452921765658625 region_id:3206 store_addr:10.12.5.147:20160 kv_process_ms:3988 scan_total_write:468324 scan_processed_write:468323 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:12.846 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:4.747618745s txnStartTS:424452921765658625 region_id:1790 store_addr:10.12.5.147:20160 kv_process_ms:4748 scan_total_write:869582 scan_processed_write:869581 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:13.272 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:4.922567739s txnStartTS:424452921765658625 region_id:14113 store_addr:10.12.5.240:20160 kv_process_ms:4920 scan_total_write:963612 scan_processed_write:963611 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:13.984 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:3.29975722s txnStartTS:424452921765658625 region_id:10781 store_addr:10.12.5.240:20160 kv_process_ms:3296 scan_total_write:552599 scan_processed_write:552598 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:14.441 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:6.651367869s txnStartTS:424452921765658625 region_id:10566 store_addr:10.12.5.147:20160 kv_process_ms:6652 scan_total_write:595259 scan_processed_write:595258 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:14.886 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:3.305261713s txnStartTS:424452921765658625 region_id:19074 store_addr:10.12.5.239:20160 kv_process_ms:3304 scan_total_write:539413 scan_processed_write:539412 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:15.045 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:7.563632299s txnStartTS:424452921765658625 region_id:12577 store_addr:10.12.5.147:20160 kv_process_ms:7560 scan_total_write:1114898 scan_processed_write:1114897 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:15.108 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:6.687459881s txnStartTS:424452921765658625 region_id:65895 store_addr:10.12.5.147:20160 kv_process_ms:6684 scan_total_write:914280 scan_processed_write:914279 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:15.673 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:786.756043ms txnStartTS:424452921765658625 region_id:11993 store_addr:10.12.5.236:20160 kv_process_ms:784 scan_total_write:1325590 scan_processed_write:1325589 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:15.847 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:7.658955248s txnStartTS:424452921765658625 region_id:51360 store_addr:10.12.5.147:20160 kv_process_ms:7656 scan_total_write:1105193 scan_processed_write:1105192 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:16.178 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:4.158638503s txnStartTS:424452921765658625 region_id:56713 store_addr:10.12.5.13:20160 kv_process_ms:4156 scan_total_write:1024328 scan_processed_write:1024327 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:16.200 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:5.422677877s txnStartTS:424452921765658625 region_id:55311 store_addr:10.12.5.13:20160 kv_process_ms:5416 scan_total_write:1056606 scan_processed_write:1056605 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:16.852 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:651.9901ms txnStartTS:424452921765658625 region_id:12509 store_addr:10.12.5.236:20160 kv_process_ms:652 scan_total_write:1197162 scan_processed_write:1197161 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:17.350 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:3.365945868s txnStartTS:424452921765658625 region_id:19157 store_addr:10.12.5.239:20160 kv_process_ms:3364 scan_total_write:542061 scan_processed_write:542060 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:17.740 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:2.63183731s txnStartTS:424452921765658625 region_id:3080 store_addr:10.12.5.147:20160 kv_process_ms:2628 scan_total_write:650512 scan_processed_write:650511 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:17.779 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:10.267572509s txnStartTS:424452921765658625 region_id:41043 store_addr:10.12.5.240:20160 kv_process_ms:10268 scan_total_write:1169612 scan_processed_write:1169611 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:18.164 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:5.993633985s txnStartTS:424452921765658625 region_id:19331 store_addr:10.12.5.240:20160 kv_process_ms:5992 scan_total_write:540620 scan_processed_write:540619 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:18.203 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:6.355087888s txnStartTS:424452921765658625 region_id:1946 store_addr:10.12.5.239:20160 kv_process_ms:6352 scan_total_write:812809 scan_processed_write:812808 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:18.603 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:5.330352278s txnStartTS:424452921765658625 region_id:56074 store_addr:10.12.5.147:20160 kv_process_ms:5328 scan_total_write:1044721 scan_processed_write:1044720 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:18.753 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:589.308826ms txnStartTS:424452921765658625 region_id:1736 store_addr:10.12.5.119:20160 kv_process_ms:584 scan_total_write:892872 scan_processed_write:892871 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:19.118 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:6.271323935s txnStartTS:424452921765658625 region_id:8083 store_addr:10.12.5.147:20160 kv_process_ms:6272 scan_total_write:1323843 scan_processed_write:1323842 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:19.346 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:1.995292387s txnStartTS:424452921765658625 region_id:10521 store_addr:10.12.5.239:20160 kv_process_ms:1996 scan_total_write:634481 scan_processed_write:634480 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:19.432 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:1.652434968s txnStartTS:424452921765658625 region_id:18121 store_addr:10.12.5.147:20160 kv_process_ms:1652 scan_total_write:607833 scan_processed_write:607832 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:20.405 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:1.059524102s txnStartTS:424452921765658625 region_id:3971 store_addr:10.12.5.239:20160 kv_process_ms:1040 scan_total_write:482174 scan_processed_write:482173 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:20.554 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:10.445205298s txnStartTS:424452921765658625 region_id:54657 store_addr:10.12.5.240:20160 kv_process_ms:10432 scan_total_write:1092342 scan_processed_write:1092341 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:20.848 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:8.28442003s txnStartTS:424452921765658625 region_id:14791 store_addr:10.12.5.240:20160 kv_process_ms:8280 scan_total_write:802694 scan_processed_write:802693 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:21.251 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:3.048020729s txnStartTS:424452921765658625 region_id:45761 store_addr:10.12.5.147:20160 kv_process_ms:3044 scan_total_write:1109334 scan_processed_write:1109333 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:21.403 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:3.663043504s txnStartTS:424452921765658625 region_id:41460 store_addr:10.12.5.147:20160 kv_process_ms:3660 scan_total_write:1145948 scan_processed_write:1145947 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:21.661 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:5.813969454s txnStartTS:424452921765658625 region_id:21054 store_addr:10.12.5.240:20160 kv_process_ms:5812 scan_total_write:487422 scan_processed_write:487421 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:22.533 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:1.979062278s txnStartTS:424452921765658625 region_id:14270 store_addr:10.12.5.147:20160 kv_process_ms:1976 scan_total_write:888301 scan_processed_write:888300 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:22.552 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:2.146451519s txnStartTS:424452921765658625 region_id:18499 store_addr:10.12.5.239:20160 kv_process_ms:2144 scan_total_write:571956 scan_processed_write:571955 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:22.998 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:6.819194412s txnStartTS:424452921765658625 region_id:15443 store_addr:10.12.5.240:20160 kv_process_ms:6816 scan_total_write:734631 scan_processed_write:734630 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:23.409 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:8.967995766s txnStartTS:424452921765658625 region_id:14722 store_addr:10.12.5.240:20160 kv_process_ms:8964 scan_total_write:816178 scan_processed_write:816177 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:23.536 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:1.874220594s txnStartTS:424452921765658625 region_id:41399 store_addr:10.12.5.239:20160 kv_process_ms:1872 scan_total_write:1147949 scan_processed_write:1147948 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:24.017 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:3.168750912s txnStartTS:424452921765658625 region_id:11683 store_addr:10.12.5.147:20160 kv_process_ms:3168 scan_total_write:1326923 scan_processed_write:1326922 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:24.483 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:5.730316743s txnStartTS:424452921765658625 region_id:14407 store_addr:10.12.5.240:20160 kv_process_ms:5728 scan_total_write:849311 scan_processed_write:849310 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:24.617 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:2.083535114s txnStartTS:424452921765658625 region_id:18203 store_addr:10.12.5.147:20160 kv_process_ms:2080 scan_total_write:589457 scan_processed_write:589456 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:24.697 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:9.023328205s txnStartTS:424452921765658625 region_id:51843 store_addr:10.12.5.240:20160 kv_process_ms:9020 scan_total_write:954170 scan_processed_write:954169 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:24.950 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:466.586835ms txnStartTS:424452921765658625 region_id:16153 store_addr:10.12.5.119:20160 kv_process_ms:464 scan_total_write:693590 scan_processed_write:692909 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:25.021 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:3.770334721s txnStartTS:424452921765658625 region_id:21990 store_addr:10.12.5.147:20160 kv_process_ms:3768 scan_total_write:1339157 scan_processed_write:1339156 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:25.050 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:6.447588335s txnStartTS:424452921765658625 region_id:18615 store_addr:10.12.5.240:20160 kv_process_ms:6448 scan_total_write:565403 scan_processed_write:565402 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:25.359 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:10.314680019s txnStartTS:424452921765658625 region_id:25532 store_addr:10.12.5.240:20160 kv_process_ms:10308 scan_total_write:1285109 scan_processed_write:1285108 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:25.862 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:811.191025ms txnStartTS:424452921765658625 region_id:12640 store_addr:10.12.5.240:20160 kv_process_ms:808 scan_total_write:1114016 scan_processed_write:1114015 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:26.197 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:9.343972966s txnStartTS:424452921765658625 region_id:40911 store_addr:10.12.5.240:20160 kv_process_ms:9340 scan_total_write:1178853 scan_processed_write:1178852 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:26.801 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:604.022366ms txnStartTS:424452921765658625 region_id:43675 store_addr:10.12.5.236:20160 kv_process_ms:604 scan_total_write:1121094 scan_processed_write:1121093 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:27.027 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:1.667628992s txnStartTS:424452921765658625 region_id:54552 store_addr:10.12.5.13:20160 kv_process_ms:1664 scan_total_write:1098773 scan_processed_write:1098772 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:27.062 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:5.658586096s txnStartTS:424452921765658625 region_id:3512 store_addr:10.12.5.240:20160 kv_process_ms:5660 scan_total_write:675075 scan_processed_write:675074 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]



[2021/04/23 06:31:38.980 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:10.863742871s txnStartTS:424452921765658625 region_id:11811 store_addr:10.12.5.240:20160 kv_process_ms:10860 scan_total_write:1340474 scan_processed_write:1340473 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:39.310 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:9.626673697s txnStartTS:424452921765658625 region_id:42865 store_addr:10.12.5.240:20160 kv_process_ms:9624 scan_total_write:1137845 scan_processed_write:1137844 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:39.335 +00:00] [INFO] [region_cache.go:829] ["switch region leader to specific leader due to kv return NotLeader"] [regionID=49] [currIdx=1] [leaderStoreID=24590972]

[2021/04/23 06:31:39.351 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:500.945092ms txnStartTS:424452921765658625 region_id:3272 store_addr:10.12.5.119:20160 backoff_ms:7010 backoff_types:[regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss]"] [conn=44389]

[2021/04/23 06:31:39.443 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:11.586804183s txnStartTS:424452921765658625 region_id:51322 store_addr:10.12.5.240:20160 kv_process_ms:11584 scan_total_write:1107136 scan_processed_write:1107135 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:39.477 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:5.215823214s txnStartTS:424452921765658625 region_id:9149 store_addr:10.12.5.240:20160 kv_process_ms:5204 scan_total_write:782980 scan_processed_write:782979 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:39.667 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_WAIT] resp_time:332.87224ms txnStartTS:424452931268378625 region_id:49 store_addr:10.12.5.240:20160 kv_wait_ms:332"]

[2021/04/23 06:31:39.703 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:722.587915ms txnStartTS:424452921765658625 region_id:18350 store_addr:10.12.5.147:20160 kv_process_ms:720 scan_total_write:584665 scan_processed_write:584664 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:39.842 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:6.720782679s txnStartTS:424452921765658625 region_id:58024 store_addr:10.12.5.240:20160 kv_process_ms:6720 scan_total_write:1005145 scan_processed_write:1005144 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:39.881 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:2.965223796s txnStartTS:424452921765658625 region_id:58409 store_addr:10.12.5.239:20160 kv_process_ms:2964 scan_total_write:993251 scan_processed_write:993250 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:39.913 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:1.333964251s txnStartTS:424452921765658625 region_id:42882 store_addr:10.12.5.119:20160 kv_process_ms:1328 scan_total_write:1138017 scan_processed_write:1138016 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:39.925 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:4.229604915s txnStartTS:424452921765658625 region_id:15333 store_addr:10.12.5.240:20160 kv_process_ms:4228 scan_total_write:741911 scan_processed_write:741910 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]



[2021/04/23 06:31:43.175 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:1.073918999s txnStartTS:424452921765658625 region_id:10113 store_addr:10.12.5.240:20160 kv_process_ms:1072 scan_total_write:695217 scan_processed_write:695216 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:43.358 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:500.96325ms txnStartTS:424452921765658625 region_id:3272 store_addr:10.12.5.119:20160 backoff_ms:11010 backoff_types:[regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss]"] [conn=44389]

[2021/04/23 06:31:43.421 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:1.455123263s txnStartTS:424452921765658625 region_id:21982 store_addr:10.12.5.240:20160 kv_process_ms:1452 scan_total_write:1344640 scan_processed_write:1344639 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:31:44.138 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:4.694382644s txnStartTS:424452921765658625 region_id:43021 store_addr:10.12.5.239:20160 kv_process_ms:4692 scan_total_write:1122453 scan_processed_write:1122452 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]



[2021/04/23 06:31:51.374 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:500.929985ms txnStartTS:424452921765658625 region_id:3272 store_addr:10.12.5.119:20160 backoff_ms:19010 backoff_types:[regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss,regionMiss]"] [conn=44389]

[2021/04/23 06:31:51.607 +00:00] [INFO] [coprocessor.go:926] ["[TIME_COP_PROCESS] resp_time:9.129048794s txnStartTS:424452921765658625 region_id:38525 store_addr:10.12.5.147:20160 kv_process_ms:9128 scan_total_write:1130914 scan_processed_write:1130913 scan_total_data:0 scan_processed_data:0 scan_total_lock:1 scan_processed_lock:0"] [conn=44389]

[2021/04/23 06:32:12.427 +00:00] [INFO] [conn.go:732] ["command dispatched failed"] [conn=44389] [connInfo="id:44389, addr:10.223.157.105:57811 status:10, collation:utf8mb4_0900_ai_ci, user:root"] [command=Query] [status="inTxn:0, autocommit:1"] [sql="select count(*) from address_info"] [txn_mode=PESSIMISTIC] [err="[tikv:9005]Region is unavailable\
github.com/pingcap/errors.AddStack\
\t/home/jenkins/agent/workspace/tidb_v4.0.6/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20200902104258-eba4f1d8f6de/errors.go:174\
github.com/pingcap/errors.Trace\
\t/home/jenkins/agent/workspace/tidb_v4.0.6/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20200902104258-eba4f1d8f6de/juju_adaptor.go:15\
github.com/pingcap/tidb/store/tikv.(*copIteratorWorker).handleCopResponse\
\t/home/jenkins/agent/workspace/tidb_v4.0.6/go/src/github.com/pingcap/tidb/store/tikv/coprocessor.go:989\
github.com/pingcap/tidb/store/tikv.(*copIteratorWorker).handleTaskOnce\
\t/home/jenkins/agent/workspace/tidb_v4.0.6/go/src/github.com/pingcap/tidb/store/tikv/coprocessor.go:798\
github.com/pingcap/tidb/store/tikv.(*copIteratorWorker).handleTask\
\t/home/jenkins/agent/workspace/tidb_v4.0.6/go/src/github.com/pingcap/tidb/store/tikv/coprocessor.go:705\
github.com/pingcap/tidb/store/tikv.(*copIteratorWorker).run\
\t/home/jenkins/agent/workspace/tidb_v4.0.6/go/src/github.com/pingcap/tidb/store/tikv/coprocessor.go:496\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357"]

GangShen · 2021 年4 月 23 日 07:02

pd-ctl 执行下 region 38525 看下这个 region 的信息

abcd · 2021 年4 月 23 日 07:04

region 38525
{
“id”: 38525,
“start_key”: “7480000000000000FF4B5F698000000000FF0000010131413236FF33417077FF63316EFF4C54467A35FF4A34FF7A6B50356F76FF31FF75464B4446434AFFFF4C58000000000000FFF900000000000000F8”,
“end_key”: “7480000000000000FF4B5F698000000000FF000001013141356FFF7654344BFF72464EFF4468317666FF464DFF4A6E38704C45FF4DFF6E6D42644B7638FFFF3534000000000000FFF900000000000000F8”,
“epoch”: {
“conf_ver”: 206,
“version”: 71
},
“peers”: [
{
“id”: 247082228,
“store_id”: 38833310
},
{
“id”: 262570657,
“store_id”: 262397455
},
{
“id”: 276756289,
“store_id”: 268391998
}
],
“leader”: {
“id”: 247082228,
“store_id”: 38833310
},
“written_bytes”: 0,
“read_bytes”: 0,
“written_keys”: 0,
“read_keys”: 0,
“approximate_size”: 118,
“approximate_keys”: 1130913
}

GangShen · 2021 年4 月 23 日 07:49

tikv 节点的日志中有没有 region is unavaible 的信息？

abcd · 2021 年4 月 23 日 08:19

好像没有

GangShen · 2021 年4 月 23 日 08:35

所有 tikv 节点上都检查了么

abcd · 2021 年4 月 23 日 08:42

全部kv都查过一遍了，确实没有

abcd · 2021 年4 月 25 日 06:14

本帖主题的kv节点无法启动问题已经解决，关于region is unavailble的问题已经新开贴https://asktug.com/t/topic/69848?u=abcd

system · 2022 年10 月 31 日 19:08

此话题已在最后回复的 1 分钟后被自动关闭。不再允许新回复。