tidb pd节点全部down掉了，大佬们帮忙看看

[TiDBer_lN8bLF2C]

[2024/09/14 15:48:53.949 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539454] [retry-timeout=500ms]

[2024/09/14 15:48:54.449 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539454] [retry-timeout=500ms]

[2024/09/14 15:48:54.951 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539454] [retry-timeout=500ms]

[2024/09/14 15:48:55.451 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539454] [retry-timeout=500ms]

[2024/09/14 15:48:55.952 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539454] [retry-timeout=500ms]

[2024/09/14 15:48:56.453 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539454] [retry-timeout=500ms]

[2024/09/14 15:48:56.954 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539454] [retry-timeout=500ms]

[2024/09/14 15:48:57.455 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539454] [retry-timeout=500ms]

[2024/09/14 15:48:57.730 +08:00] [WARN] [util.go:163] [“apply request took too long”] [took=1.999837366s] [expected-duration=100ms] [prefix="read-only range "] [request="key:"/topology/tidb/" range_end:"/topology/tidb0" "] [response=] [error=“context canceled”]

[2024/09/14 15:48:57.730 +08:00] [INFO] [trace.go:152] [“trace[670569142] range”] [detail=“{range_begin:/topology/tidb/; range_end:/topology/tidb0; }”] [duration=2.000063884s] [start=2024/09/14 15:48:55.730 +08:00] [end=2024/09/14 15:48:57.730 +08:00] [steps=“["trace[670569142] ‘agreement among raft nodes before linearized reading’ (duration: 1.999895244s)"]”]

[2024/09/14 15:48:57.955 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539454] [retry-timeout=500ms]

[2024/09/14 15:48:58.035 +08:00] [WARN] [util.go:163] [“apply request took too long”] [took=9.999834678s] [expected-duration=100ms] [prefix="read-only range "] [request="key:"/pd/7271086761806099252/config" "] [response=] [error=“context deadline exceeded”]

[2024/09/14 15:48:58.035 +08:00] [INFO] [trace.go:152] [“trace[2071331084] range”] [detail=“{range_begin:/pd/7271086761806099252/config; range_end:; }”] [duration=10.0000895s] [start=2024/09/14 15:48:48.035 +08:00] [end=2024/09/14 15:48:58.035 +08:00] [steps=“["trace[2071331084] ‘agreement among raft nodes before linearized reading’ (duration: 9.999830793s)"]”]

[2024/09/14 15:48:58.035 +08:00] [WARN] [etcdutil.go:121] [“kv gets too slow”] [request-key=/pd/7271086761806099252/config] [cost=10.000598005s] [error=“context deadline exceeded”]

[2024/09/14 15:48:58.035 +08:00] [ERROR] [etcdutil.go:126] [“load from etcd meet error”] [key=/pd/7271086761806099252/config] [error=“[PD:etcd:ErrEtcdKVGet]context deadline exceeded: context deadline exceeded”]

[2024/09/14 15:48:58.035 +08:00] [WARN] [manager.go:101] [“failed to reload persist options”]

[2024/09/14 15:48:58.456 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539454] [retry-timeout=500ms]

[2024/09/14 15:48:58.941 +08:00] [WARN] [v3_server.go:830] [“timed out waiting for read index response (local node might have slow network)”] [timeout=11s]

[2024/09/14 15:48:59.354 +08:00] [WARN] [util.go:163] [“apply request took too long”] [took=5.00066359s] [expected-duration=100ms] [prefix="read-only range "] [request="key:"/tidb/store/gcworker/saved_safe_point" "] [response=] [error=“context canceled”]

[2024/09/14 15:48:59.355 +08:00] [INFO] [trace.go:152] [“trace[1510270020] range”] [detail=“{range_begin:/tidb/store/gcworker/saved_safe_point; range_end:; }”] [duration=5.000946151s] [start=2024/09/14 15:48:54.354 +08:00] [end=2024/09/14 15:48:59.355 +08:00] [steps=“["trace[1510270020] ‘agreement among raft nodes before linearized reading’ (duration: 5.000725556s)"]”]

[2024/09/14 15:48:59.357 +08:00] [INFO] [raft.go:850] [“3d1515a151334b90 [logterm: 334, index: 27807545, vote: 3d1515a151334b90] ignored MsgPreVote from ab636f920c92529f [logterm: 334, index: 27807507] at term 334: lease is not expired (remaining ticks: 4)”]

[2024/09/14 15:48:59.442 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539459] [retry-timeout=500ms]

[2024/09/14 15:48:59.539 +08:00] [WARN] [server.go:1115] [“failed to revoke lease”] [lease-id=4b9091e69b653626] [error=“etcdserver: request timed out”]

[2024/09/14 15:48:59.681 +08:00] [WARN] [util.go:163] [“apply request took too long”] [took=5.000049061s] [expected-duration=100ms] [prefix="read-only range "] [request="key:"/tidb/store/gcworker/saved_safe_point" "] [response=] [error=“context canceled”]

[2024/09/14 15:48:59.681 +08:00] [INFO] [trace.go:152] [“trace[582209688] range”] [detail=“{range_begin:/tidb/store/gcworker/saved_safe_point; range_end:; }”] [duration=5.000218265s] [start=2024/09/14 15:48:54.681 +08:00] [end=2024/09/14 15:48:59.681 +08:00] [steps=“["trace[582209688] ‘agreement among raft nodes before linearized reading’ (duration: 5.000091282s)"]”]

[2024/09/14 15:48:59.942 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539459] [retry-timeout=500ms]

[2024/09/14 15:49:00.443 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539459] [retry-timeout=500ms]

[2024/09/14 15:49:00.547 +08:00] [WARN] [etcdutil.go:121] [“kv gets too slow”] [request-key=/pd/7271086761806099252/leader] [cost=10.000750899s] [error=“context deadline exceeded”]

[2024/09/14 15:49:00.547 +08:00] [ERROR] [etcdutil.go:126] [“load from etcd meet error”] [key=/pd/7271086761806099252/leader] [error=“[PD:etcd:ErrEtcdKVGet]context deadline exceeded: context deadline exceeded”]

[2024/09/14 15:49:00.547 +08:00] [ERROR] [member.go:167] [“getting pd leader meets error”] [error=“[PD:etcd:ErrEtcdKVGet]context deadline exceeded: context deadline exceeded”]

[2024/09/14 15:49:00.547 +08:00] [WARN] [util.go:163] [“apply request took too long”] [took=9.999879635s] [expected-duration=100ms] [prefix="read-only range "] [request="key:"/pd/7271086761806099252/leader" "] [response=] [error=“context deadline exceeded”]

[2024/09/14 15:49:00.547 +08:00] [INFO] [trace.go:152] [“trace[807533148] range”] [detail=“{range_begin:/pd/7271086761806099252/leader; range_end:; }”] [duration=10.000308706s] [start=2024/09/14 15:48:50.547 +08:00] [end=2024/09/14 15:49:00.547 +08:00] [steps=“["trace[807533148] ‘agreement among raft nodes before linearized reading’ (duration: 9.999904702s)"]”]

[2024/09/14 15:49:00.702 +08:00] [WARN] [util.go:163] [“apply request took too long”] [took=4.999996688s] [expected-duration=100ms] [prefix="read-only range "] [request="key:"/tidb/store/gcworker/saved_safe_point" "] [response=] [error=“context deadline exceeded”]

[2024/09/14 15:49:00.702 +08:00] [INFO] [trace.go:152] [“trace[578486376] range”] [detail=“{range_begin:/tidb/store/gcworker/saved_safe_point; range_end:; }”] [duration=5.000209789s] [start=2024/09/14 15:48:55.702 +08:00] [end=2024/09/14 15:49:00.702 +08:00] [steps=“["trace[578486376] ‘agreement among raft nodes before linearized reading’ (duration: 5.000043533s)"]”]

[2024/09/14 15:49:00.943 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539459] [retry-timeout=500ms]

[2024/09/14 15:49:01.445 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539459] [retry-timeout=500ms]

[2024/09/14 15:49:01.945 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539459] [retry-timeout=500ms]

[2024/09/14 15:49:02.446 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539459] [retry-timeout=500ms]

[2024/09/14 15:49:02.947 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539459] [retry-timeout=500ms]

[2024/09/14 15:49:03.447 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539459] [retry-timeout=500ms]

[2024/09/14 15:49:03.948 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539459] [retry-timeout=500ms]

[2024/09/14 15:49:04.449 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539459] [retry-timeout=500ms]

[2024/09/14 15:49:04.858 +08:00] [INFO] [raft.go:850] [“3d1515a151334b90 [logterm: 334, index: 27807546, vote: 3d1515a151334b90] ignored MsgPreVote from ab636f920c92529f [logterm: 334, index: 27807507] at term 334: lease is not expired (remaining ticks: 4)”]

[2024/09/14 15:49:04.950 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539459] [retry-timeout=500ms]

[2024/09/14 15:49:05.451 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539459] [retry-timeout=500ms]

[2024/09/14 15:49:05.952 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539459] [retry-timeout=500ms]

[2024/09/14 15:49:06.453 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539459] [retry-timeout=500ms]

[2024/09/14 15:49:06.554 +08:00] [WARN] [util.go:163] [“apply request took too long”] [took=1.000072777s] [expected-duration=100ms] [prefix="read-only range "] [request="key:"/global/config/" range_end:"/global/config0" "] [response=] [error=“context deadline exceeded”]

[2024/09/14 15:49:06.555 +08:00] [INFO] [trace.go:152] [“trace[177053562] range”] [detail=“{range_begin:/global/config/; range_end:/global/config0; }”] [duration=1.000447593s] [start=2024/09/14 15:49:05.554 +08:00] [end=2024/09/14 15:49:06.555 +08:00] [steps=“["trace[177053562] ‘agreement among raft nodes before linearized reading’ (duration: 1.000124158s)"]”]

[2024/09/14 15:49:06.954 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539459] [retry-timeout=500ms]

[2024/09/14 15:49:07.455 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539459] [retry-timeout=500ms]

[2024/09/14 15:49:07.756 +08:00] [WARN] [util.go:163] [“apply request took too long”] [took=999.908031ms] [expected-duration=100ms] [prefix="read-only range "] [request="key:"/global/config/" range_end:"/global/config0" "] [response=] [error=“context deadline exceeded”]

[2024/09/14 15:49:07.756 +08:00] [INFO] [trace.go:152] [“trace[2139686559] range”] [detail=“{range_begin:/global/config/; range_end:/global/config0; }”] [duration=1.000103329s] [start=2024/09/14 15:49:06.756 +08:00] [end=2024/09/14 15:49:07.756 +08:00] [steps=“["trace[2139686559] ‘agreement among raft nodes before linearized reading’ (duration: 999.970834ms)"]”]

[2024/09/14 15:49:07.955 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539459] [retry-timeout=500ms]

[2024/09/14 15:49:08.036 +08:00] [WARN] [util.go:163] [“apply request took too long”] [took=10.000307533s] [expected-duration=100ms] [prefix="read-only range "] [request="key:"/pd/7271086761806099252/config" "] [response=] [error=“context deadline exceeded”]

[2024/09/14 15:49:08.036 +08:00] [WARN] [etcdutil.go:121] [“kv gets too slow”] [request-key=/pd/7271086761806099252/config] [cost=10.001049943s] [error=“context deadline exceeded”]

[2024/09/14 15:49:08.037 +08:00] [INFO] [trace.go:152] [“trace[1755547562] range”] [detail=“{range_begin:/pd/7271086761806099252/config; range_end:; }”] [duration=10.000558392s] [start=2024/09/14 15:48:58.036 +08:00] [end=2024/09/14 15:49:08.037 +08:00] [steps=“["trace[1755547562] ‘agreement among raft nodes before linearized reading’ (duration: 10.000364113s)"]”]

[2024/09/14 15:49:08.037 +08:00] [ERROR] [etcdutil.go:126] [“load from etcd meet error”] [key=/pd/7271086761806099252/config] [error=“[PD:etcd:ErrEtcdKVGet]context deadline exceeded: context deadline exceeded”]

[2024/09/14 15:49:08.037 +08:00] [WARN] [manager.go:101] [“failed to reload persist options”]

[2024/09/14 15:49:08.456 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539459] [retry-timeout=500ms]

[2024/09/14 15:49:08.956 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539459] [retry-timeout=500ms]

[2024/09/14 15:49:09.457 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539459] [retry-timeout=500ms]

[2024/09/14 15:49:09.942 +08:00] [WARN] [v3_server.go:830] [“timed out waiting for read index response (local node might have slow network)”] [timeout=11s]

[2024/09/14 15:49:10.159 +08:00] [WARN] [util.go:163] [“apply request took too long”] [took=999.987554ms] [expected-duration=100ms] [prefix="read-only range "] [request="key:"/global/config/" range_end:"/global/config0" "] [response=] [error=“context canceled”]

[2024/09/14 15:49:10.159 +08:00] [INFO] [trace.go:152] [“trace[1002440032] range”] [detail=“{range_begin:/global/config/; range_end:/global/config0; }”] [duration=1.000215719s] [start=2024/09/14 15:49:09.159 +08:00] [end=2024/09/14 15:49:10.159 +08:00] [steps=“["trace[1002440032] ‘agreement among raft nodes before linearized reading’ (duration: 1.000031797s)"]”]

[2024/09/14 15:49:10.357 +08:00] [INFO] [raft.go:850] [“3d1515a151334b90 [logterm: 334, index: 27807547, vote: 3d1515a151334b90] ignored MsgPreVote from ab636f920c92529f [logterm: 334, index: 27807507] at term 334: lease is not expired (remaining ticks: 4)”]

[2024/09/14 15:49:10.443 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539462] [retry-timeout=500ms]

[2024/09/14 15:49:10.554 +08:00] [WARN] [server.go:1115] [“failed to revoke lease”] [lease-id=4b9091e69b653626] [error=“etcdserver: request timed out”]

[2024/09/14 15:49:10.749 +08:00] [WARN] [util.go:163] [“apply request took too long”] [took=10.000020888s] [expected-duration=100ms] [prefix="read-only range "] [request="key:"/pd/7271086761806099252/leader" "] [response=] [error=“context deadline exceeded”]

[2024/09/14 15:49:10.749 +08:00] [INFO] [trace.go:152] [“trace[177301511] range”] [detail=“{range_begin:/pd/7271086761806099252/leader; range_end:; }”] [duration=10.000178881s] [start=2024/09/14 15:49:00.749 +08:00] [end=2024/09/14 15:49:10.749 +08:00] [steps=“["trace[177301511] ‘agreement among raft nodes before linearized reading’ (duration: 10.000017932s)"]”]

[2024/09/14 15:49:10.749 +08:00] [WARN] [etcdutil.go:121] [“kv gets too slow”] [request-key=/pd/7271086761806099252/leader] [cost=10.000827759s] [error=“context deadline exceeded”]

[2024/09/14 15:49:10.750 +08:00] [ERROR] [etcdutil.go:126] [“load from etcd meet error”] [key=/pd/7271086761806099252/leader] [error=“[PD:etcd:ErrEtcdKVGet]context deadline exceeded: context deadline exceeded”]

[2024/09/14 15:49:10.750 +08:00] [ERROR] [member.go:167] [“getting pd leader meets error”] [error=“[PD:etcd:ErrEtcdKVGet]context deadline exceeded: context deadline exceeded”]

[2024/09/14 15:49:10.943 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539462] [retry-timeout=500ms]

[2024/09/14 15:49:11.444 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539462] [retry-timeout=500ms]

[2024/09/14 15:49:11.681 +08:00] [WARN] [util.go:163] [“apply request took too long”] [took=4.999551394s] [expected-duration=100ms] [prefix="read-only range "] [request="key:"/tidb/store/gcworker/saved_safe_point" "] [response=] [error=“context canceled”]

[2024/09/14 15:49:11.682 +08:00] [INFO] [trace.go:152] [“trace[760058348] range”] [detail=“{range_begin:/tidb/store/gcworker/saved_safe_point; range_end:; }”] [duration=4.99978928s] [start=2024/09/14 15:49:06.682 +08:00] [end=2024/09/14 15:49:11.682 +08:00] [steps=“["trace[760058348] ‘agreement among raft nodes before linearized reading’ (duration: 4.999617481s)"]”]

[2024/09/14 15:49:11.945 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539462] [retry-timeout=500ms]

[2024/09/14 15:49:12.446 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539462] [retry-timeout=500ms]

[2024/09/14 15:49:12.791 +08:00] [WARN] [util.go:163] [“apply request took too long”] [took=1.997086397s] [expected-duration=100ms] [prefix="read-only range "] [request="key:"/topology/tidb/" range_end:"/topology/tidb0" "] [response=] [error=“context deadline exceeded”]

[2024/09/14 15:49:12.791 +08:00] [INFO] [trace.go:152] [“trace[923308079] range”] [detail=“{range_begin:/topology/tidb/; range_end:/topology/tidb0; }”] [duration=1.997415554s] [start=2024/09/14 15:49:10.793 +08:00] [end=2024/09/14 15:49:12.791 +08:00] [steps=“["trace[923308079] ‘agreement among raft nodes before linearized reading’ (duration: 1.99716787s)"]”]

[2024/09/14 15:49:12.946 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539462] [retry-timeout=500ms]

[2024/09/14 15:49:13.447 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539462] [retry-timeout=500ms]

[2024/09/14 15:49:13.948 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539462] [retry-timeout=500ms]

[2024/09/14 15:49:14.448 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539462] [retry-timeout=500ms]

[2024/09/14 15:49:14.948 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539462] [retry-timeout=500ms]

[2024/09/14 15:49:15.449 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539462] [retry-timeout=500ms]

[2024/09/14 15:49:15.858 +08:00] [INFO] [raft.go:850] [“3d1515a151334b90 [logterm: 334, index: 27807550, vote: 3d1515a151334b90] ignored MsgPreVote from ab636f920c92529f [logterm: 334, index: 27807507] at term 334: lease is not expired (remaining ticks: 4)”]

[2024/09/14 15:49:15.950 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539462] [retry-timeout=500ms]

[2024/09/14 15:49:16.450 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539462] [retry-timeout=500ms]

[2024/09/14 15:49:16.951 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539462] [retry-timeout=500ms]

[2024/09/14 15:49:17.358 +08:00] [WARN] [util.go:163] [“apply request took too long”] [took=5.000162815s] [expected-duration=100ms] [prefix="read-only range "] [request="key:"/tidb/store/gcworker/saved_safe_point" "] [response=] [error=“context canceled”]

[2024/09/14 15:49:17.358 +08:00] [INFO] [trace.go:152] [“trace[895303937] range”] [detail=“{range_begin:/tidb/store/gcworker/saved_safe_point; range_end:; }”] [duration=5.000415445s] [start=2024/09/14 15:49:12.358 +08:00] [end=2024/09/14 15:49:17.358 +08:00] [steps=“["trace[895303937] ‘agreement among raft nodes before linearized reading’ (duration: 5.000204477s)"]”]

[2024/09/14 15:49:17.451 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539462] [retry-timeout=500ms]

[2024/09/14 15:49:17.953 +08:00] [WARN] [v3_server.go:814] [“waiting for ReadIndex response took too long, retrying”] [sent-request-id=5445012369126539462] [retry-timeout=500ms]

[TiDBer_lN8bLF2C]

请大佬帮忙看看

[Kongdom]

看到好多超时，是不是现在集群服务器负载很高？

[TiDBer_lN8bLF2C]

内存还有很多，cpu使用top看了下加起来20多

[Kongdom]

错误日志里有这种报错信息么？no longer a leader because lease has expired
看着和这个帖子比较像

TiDB集群无法启动

【 TiDB 使用环境】生产环境 /测试/ Poc 【 TiDB 版本】6.5 【遇到的问题】tidb集群不知明原因挂掉 【复现路径】无操作 【问题现象及影响】 集群宕机后无法正常拉起，pd单独启动正常，tikv单独启动正常，过一段时间又整体down掉 pd日志： [2023/06/15 15:24:39.106 +08:00] [WARN] [v3_server.go:830] […

[TiDBer_lN8bLF2C]

有的 。

[TiDBer_jYQINSnf]

网络通么？感觉是失联了似的。你这环境发生了什么变化？

[kevinsna (Ti D Ber P O Zcnp Ja)]

检查所有主机的时间同步是否正常，如果同步，检查IO负载是否会很高

[TiDBer_ZxWlj6A1]

@TiDBer_lN8bLF2C

waiting for ReadIndex response took too long

问题应该是这个返回时间超时了，集群网络ok？

[residentevil (Ti D Ber Zux S Mt7 W)]

从pd机器上ping下tikv节点，同时三台pd相互ping下看看

[kevinsna (Ti D Ber P O Zcnp Ja)]

这个问题最终解决了么？解决了的话，最终是如何解决的？

[db_user]

检测下io的压力吧

[koby (Ti D Ber Bk7apg Dl)]

网络问题吧，看着好像

[liuis (心安是吾乡)]

etcdserver: request timed out 看各种超时，感觉集群网络有问题

[xiaohaozifeifeifei]

都是超时的报错，网络问题么？

[LittleWrong (Ti D Ber Ucx Oaz5s)]

这个错误消息表明，TiDB 在等待 Raft 组件的 ReadIndex 响应时遇到了超时问题，并正在重试操作。具体来看：

ReadIndex 是 Raft 协议中的一种操作，通常用于确保读取操作的线性一致性，即确保读取的数据是最新的并且不会出现过时数据的情况。
日志中的 [WARN] 表示这是一个警告级别的日志，不是致命错误。
waiting for ReadIndex response took too long 说明 TiDB 等待 ReadIndex 响应的时间超过了预期。
[retry-timeout=500ms] 说明系统在每次重试之前，会等待 500 毫秒。

可能的原因有：

（1）集群负载过高：如果 TiDB 或 PD（Placement Driver）负载较高，处理 Raft 请求的响应时间可能会延迟。
（2）网络延迟：如果 TiDB 与 TiKV 或 PD 之间的网络有延迟或不稳定，可能导致这种情况。
（3）Raft 网络分区或 leader 切换：如果 Raft 组的 leader 切换频繁或某些节点不可达，这可能会延迟 ReadIndex 操作的响应时间。

解决建议：

（1）检查集群的监控，看看是否有高负载的情况，特别是 TiKV 和 PD 的资源使用情况。
（2）查看 TiKV 和 PD 的日志，看看是否有网络问题或 leader 切换的问题。
（3）检查是一下各节点的文件描述符，是否正常ulimit -n.