999分位的延时忽然飙升,其中有一张表在几秒内忽然变慢到了10秒。大量pdRPC backoffer

【 TiDB 使用环境】生产环境
【 TiDB 版本】4.0.16
【复现路径】未操作
【遇到的问题】 999分位的延时忽然飙升,其中有一张表在几秒内忽然变慢到了10秒。引发系统抖动

引发了 pdRPC backoffer 这种错误一般什么情况造成的平时都是txnLock,txnNotFound锁3秒。可以提供思路查询么?

[2024/10/23 11:22:20.933 +08:00] [WARN] [backoff.go:329] ["pdRPC backoffer.maxSleep 10000ms is exceeded, errors:\nregion not found for key \"t\\x80\\x00\\x00\\x00\\x00\\x00\\x00\\xe2_r\\x80\\x00\\x00\\x04g4\\xf4\\n\" at 2024-10-23T11:22
:16.041525783+08:00\nregion not found for key \"t\\x80\\x00\\x00\\x00\\x00\\x00\\x00\\xe2_r\\x80\\x00\\x00\\x04g4\\xf4\\n\" at 2024-10-23T11:22:18.514284568+08:00\nregion not found for key \"t\\x80\\x00\\x00\\x00\\x00\\x00\\x00\\xe2_r\\
x80\\x00\\x00\\x04g4\\xf4\\n\" at 2024-10-23T11:22:20.933776491+08:00"]
[2024/10/23 11:22:20.935 +08:00] [ERROR] [distsql.go:1009] ["table reader fetch next chunk failed"] [conn=200732] [error="[tikv:9001]PD server timeout"]
[2024/10/23 11:22:20.936 +08:00] [INFO] [conn.go:864] ["command dispatched failed"] [conn=200732] [connInfo="id:200732, addr:10.204.168.71:31339 status:10, collation:utf8_general_ci, user:es_post"] [command=Query] [status="inTxn:0, auto
commit:1"] [sql="select\n     \n\n    id, post_id, express_id, ed_code, box_type, box_id, receiver_mobile, receiver_uic_id,\n    staff_mobile, staff_uic_id, staff_company_id, staff_company_name, post_tm, pay_serial_number,\n    pay_type
, pay_status, pay_tm, refund_tm, order_status, coupon_id, coupon_type, coupon_msg_id,\n    coupon_discount_money, order_money, pay_money, record_version, create_emp, update_emp,\n    create_sys_tm, update_sys_tm, delete_flag, charge_typ
e,session_token_id,receiver_mobile_tag,multi_post_type,express_type,box_status\n    ,station_type,station_id\n   \n    from pre_post_order\n    where  post_id = 'FC7552570720241023112146378320M'"] [txn_mode=PESSIMISTIC] [err="[tik
v:9001]PD server timeout\ngithub.com/pingcap/errors.AddStack\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/errors.go:174\ngithub.com/pingcap
/errors.Trace\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/juju_adaptor.go:15\ngithub.com/pingcap/tidb/store/tikv.(*RegionCache).loadRegion
\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/store/tikv/region_cache.go:996\ngithub.com/pingcap/tidb/store/tikv.(*RegionCache).findRegionByKey\n\t/home/jenkins/agent/workspace/optimi
zation-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/store/tikv/region_cache.go:575\ngithub.com/pingcap/tidb/store/tikv.(*RegionCache).LocateKey\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/
pingcap/tidb/store/tikv/region_cache.go:535\ngithub.com/pingcap/tidb/store/tikv.splitRanges\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/store/tikv/coprocessor.go:323\ngithub.com/ping
cap/tidb/store/tikv.buildCopTasks\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/store/tikv/coprocessor.go:280\ngithub.com/pingcap/tidb/store/tikv.(*CopClient).Send\n\t/home/jenkins/age
nt/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/store/tikv/coprocessor.go:69\ngithub.com/pingcap/tidb/distsql.Select\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/ping
cap/tidb/distsql/distsql.go:49\ngithub.com/pingcap/tidb/distsql.SelectWithRuntimeStats\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/distsql/distsql.go:97\ngithub.com/pingcap/tidb/exec
utor.selectResultHook.SelectResult\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/table_reader.go:51\ngithub.com/pingcap/tidb/executor.(*dataReaderBuilder).buildTableReaderFrom
Handles\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/builder.go:2796\ngithub.com/pingcap/tidb/executor.(*IndexLookUpExecutor).buildTableReader\n\t/home/jenkins/agent/workspac
e/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/distsql.go:551\ngithub.com/pingcap/tidb/executor.(*tableWorker).executeTask\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com
/pingcap/tidb/executor/distsql.go:988\ngithub.com/pingcap/tidb/executor.(*tableWorker).pickAndExecTask\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/distsql.go:871\ngithub.com
/pingcap/tidb/executor.(*IndexLookUpExecutor).startTableWorker.func1\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/distsql.go:527\nruntime.goexit\n\t/usr/local/go/src/runtime/
asm_amd64.s:1357"]
[2024/10/23 11:22:21.005 +08:00] [WARN] [backoff.go:329] ["pdRPC backoffer.maxSleep 10000ms is exceeded, errors:\nregion not found for key \"t\\x80\\x00\\x00\\x00\\x00\\x00\\x00\\xe2_r\\x80\\x00\\x00\\x04g5B\\xb1\" at 2024-10-23T11:22:1
6.894600342+08:00\nregion not found for key \"t\\x80\\x00\\x00\\x00\\x00\\x00\\x00\\xe2_r\\x80\\x00\\x00\\x04g5B\\xb1\" at 2024-10-23T11:22:19.404118112+08:00\nregion not found for key \"t\\x80\\x00\\x00\\x00\\x00\\x00\\x00\\xe2_r\\x80\
\x00\\x00\\x04g5B\\xb1\" at 2024-10-23T11:22:21.00560486+08:00"]
[2024/10/23 11:22:21.005 +08:00] [ERROR] [distsql.go:1009] ["table reader fetch next chunk failed"] [conn=200539] [error="[tikv:9001]PD server timeout"]
[2024/10/23 11:22:21.006 +08:00] [INFO] [conn.go:864] ["command dispatched failed"] [conn=200539] [connInfo="id:200539, addr:10.204.168.71:18503 status:10, collation:utf8_general_ci, user:es_post"] [command=Query] [status="inTxn:0, auto
commit:1"] [sql="select\n     \n\n    id, post_id, express_id, ed_code, box_type, box_id, receiver_mobile, receiver_uic_id,\n    staff_mobile, staff_uic_id, staff_company_id, staff_company_name, post_tm, pay_serial_number,\n    pay_type
, pay_status, pay_tm, refund_tm, order_status, coupon_id, coupon_type, coupon_msg_id,\n    coupon_discount_money, order_money, pay_money, record_version, create_emp, update_emp,\n    create_sys_tm, update_sys_tm, delete_flag, charge_typ
e,session_token_id,receiver_mobile_tag,multi_post_type,express_type,box_status\n    ,station_type,station_id\n   \n    from pre_post_order\n    where  post_id = 'FC532782920241023112200785893M'"] [txn_mode=PESSIMISTIC] [err="[tikv
:9001]PD server timeout\ngithub.com/pingcap/errors.AddStack\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/errors.go:174\ngithub.com/pingcap/
errors.Trace\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/juju_adaptor.go:15\ngithub.com/pingcap/tidb/store/tikv.(*RegionCache).loadRegion\
n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/store/tikv/region_cache.go:996\ngithub.com/pingcap/tidb/store/tikv.(*RegionCache).findRegionByKey\n\t/home/jenkins/agent/workspace/optimiz
ation-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/store/tikv/region_cache.go:575\ngithub.com/pingcap/tidb/store/tikv.(*RegionCache).LocateKey\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/p
ingcap/tidb/store/tikv/region_cache.go:535\ngithub.com/pingcap/tidb/store/tikv.splitRanges\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/store/tikv/coprocessor.go:323\ngithub.com/pingc
ap/tidb/store/tikv.buildCopTasks\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/store/tikv/coprocessor.go:280\ngithub.com/pingcap/tidb/store/tikv.(*CopClient).Send\n\t/home/jenkins/agen
t/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/store/tikv/coprocessor.go:69\ngithub.com/pingcap/tidb/distsql.Select\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingc
ap/tidb/distsql/distsql.go:49\ngithub.com/pingcap/tidb/distsql.SelectWithRuntimeStats\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/distsql/distsql.go:97\ngithub.com/pingcap/tidb/execu
tor.selectResultHook.SelectResult\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/table_reader.go:51\ngithub.com/pingcap/tidb/executor.(*dataReaderBuilder).buildTableReaderFromH
andles\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/builder.go:2796\ngithub.com/pingcap/tidb/executor.(*IndexLookUpExecutor).buildTableReader\n\t/home/jenkins/agent/workspace
/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/distsql.go:551\ngithub.com/pingcap/tidb/executor.(*tableWorker).executeTask\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/
pingcap/tidb/executor/distsql.go:988\ngithub.com/pingcap/tidb/executor.(*tableWorker).pickAndExecTask\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/distsql.go:871\ngithub.com/
pingcap/tidb/executor.(*IndexLookUpExecutor).startTableWorker.func1\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/distsql.go:527\nruntime.goexit\n\t/usr/local/go/src/runtime/a
sm_amd64.s:1357"]
[2024/10/23 11:22:21.058 +08:00] [WARN] [backoff.go:329] ["pdRPC backoffer.maxSleep 10000ms is exceeded, errors:\nregion not found for key \"t\\x80\\x00\\x00\\x00\\x00\\x00\\x00\\xe2_r\\x80\\x00\\x00\\x04g5C\\n\" at 2024-10-23T11:22:16.
265580406+08:00\nregion not found for key \"t\\x80\\x00\\x00\\x00\\x00\\x00\\x00\\xe2_r\\x80\\x00\\x00\\x04g5C\\n\" at 2024-10-23T11:22:19.193921456+08:00\nregion not found for key \"t\\x80\\x00\\x00\\x00\\x00\\x00\\x00\\xe2_r\\x80\\x00
\\x00\\x04g5C\\n\" at 2024-10-23T11:22:21.05828939+08:00"]
[2024/10/23 11:22:21.058 +08:00] [ERROR] [distsql.go:1009] ["table reader fetch next chunk failed"] [conn=201144] [error="[tikv:9001]PD server timeout"]
[2024/10/23 11:22:21.058 +08:00] [INFO] [conn.go:864] ["command dispatched failed"] [conn=201144] [connInfo="id:201144, addr:10.204.168.71:59071 status:10, collation:utf8_general_ci, user:es_post"] [command=Query] [status="inTxn:0, auto
commit:1"] [sql="select\n     \n\n    id, post_id, express_id, ed_code, box_type, box_id, receiver_mobile, receiver_uic_id,\n    staff_mobile, staff_uic_id, staff_company_id, staff_company_name, post_tm, pay_serial_number,\n    pay_type
, pay_status, pay_tm, refund_tm, order_status, coupon_id, coupon_type, coupon_msg_id,\n    coupon_discount_money, order_money, pay_money, record_version, create_emp, update_emp,\n    create_sys_tm, update_sys_tm, delete_flag, charge_typ
e,session_token_id,receiver_mobile_tag,multi_post_type,express_type,box_status\n    ,station_type,station_id\n   \n    from pre_post_order\n    where  post_id = 'FC0201467320241023112203617669M'"] [txn_mode=PESSIMISTIC] [err="[tik
v:9001]PD server timeout\ngithub.com/pingcap/errors.AddStack\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/errors.go:174\ngithub.com/pingcap
/errors.Trace\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/juju_adaptor.go:15\ngithub.com/pingcap/tidb/store/tikv.(*RegionCache).loadRegion
\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/store/tikv/region_cache.go:996\ngithub.com/pingcap/tidb/store/tikv.(*RegionCache).findRegionByKey\n\t/home/jenkins/agent/workspace/optimi
zation-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/store/tikv/region_cache.go:575\ngithub.com/pingcap/tidb/store/tikv.(*RegionCache).LocateKey\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/
pingcap/tidb/store/tikv/region_cache.go:535\ngithub.com/pingcap/tidb/store/tikv.splitRanges\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/store/tikv/coprocessor.go:323\ngithub.com/ping
cap/tidb/store/tikv.buildCopTasks\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/store/tikv/coprocessor.go:280\ngithub.com/pingcap/tidb/store/tikv.(*CopClient).Send\n\t/home/jenkins/age
nt/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/store/tikv/coprocessor.go:69\ngithub.com/pingcap/tidb/distsql.Select\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/ping
cap/tidb/distsql/distsql.go:49\ngithub.com/pingcap/tidb/distsql.SelectWithRuntimeStats\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/distsql/distsql.go:97\ngithub.com/pingcap/tidb/exec
utor.selectResultHook.SelectResult\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/table_reader.go:51\ngithub.com/pingcap/tidb/executor.(*dataReaderBuilder).buildTableReaderFrom
Handles\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/builder.go:2796\ngithub.com/pingcap/tidb/executor.(*IndexLookUpExecutor).buildTableReader\n\t/home/jenkins/agent/workspac
e/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/distsql.go:551\ngithub.com/pingcap/tidb/executor.(*tableWorker).executeTask\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com
/pingcap/tidb/executor/distsql.go:988\ngithub.com/pingcap/tidb/executor.(*tableWorker).pickAndExecTask\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/distsql.go:871\ngithub.com
/pingcap/tidb/executor.(*IndexLookUpExecutor).startTableWorker.func1\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/distsql.go:527\nruntime.goexit\n\t/usr/local/go/src/runtime/
asm_amd64.s:1357"]
[2024/10/23 11:22:21.069 +08:00] [WARN] [backoff.go:329] ["pdRPC backoffer.maxSleep 10000ms is exceeded, errors:\nregion not found for key \"t\\x80\\x00\\x00\\x00\\x00\\x00\\x00\\xe2_r\\x80\\x00\\x00\\x04g4\\xfe{\" at 2024-10-23T11:22:1
6.146744182+08:00\nregion not found for key \"t\\x80\\x00\\x00\\x00\\x00\\x00\\x00\\xe2_r\\x80\\x00\\x00\\x04g4\\xfe{\" at 2024-10-23T11:22:18.743122928+08:00\nregion not found for key \"t\\x80\\x00\\x00\\x00\\x00\\x00\\x00\\xe2_r\\x80\
\x00\\x00\\x04g4\\xfe{\" at 2024-10-23T11:22:21.069488675+08:00"]
[2024/10/23 11:22:21.069 +08:00] [ERROR] [distsql.go:1009] ["table reader fetch next chunk failed"] [conn=200361] [error="[tikv:9001]PD server timeout"]
[2024/10/23 11:22:21.070 +08:00] [INFO] [conn.go:864] ["command dispatched failed"] [conn=200361] [connInfo="id:200361, addr:10.204.168.71:60952 status:10, collation:utf8_general_ci, user:es_post"] [command=Query] [status="inTxn:0, auto
commit:1"] [sql="select\n     \n\n    id, post_id, express_id, ed_code, box_type, box_id, receiver_mobile, receiver_uic_id,\n    staff_mobile, staff_uic_id, staff_company_id, staff_company_name, post_tm, pay_serial_number,\n    pay_type
, pay_status, pay_tm, refund_tm, order_status, coupon_id, coupon_type, coupon_msg_id,\n    coupon_discount_money, order_money, pay_money, record_version, create_emp, update_emp,\n    create_sys_tm, update_sys_tm, delete_flag, charge_typ
e,session_token_id,receiver_mobile_tag,multi_post_type,express_type,box_status\n    ,station_type,station_id\n   \n    from pre_post_order\n    where  post_id = 'FC0202600420241023112203143183M'"] [txn_mode=PESSIMISTIC] [err="[tik
v:9001]PD server timeout\ngithub.com/pingcap/errors.AddStack\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/errors.go:174\ngithub.com/pingcap
/errors.Trace\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/juju_adaptor.go:15\ngithub.com/pingcap/tidb/store/tikv.(*RegionCache).loadRegion
\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/store/tikv/region_cache.go:996\ngithub.com/pingcap/tidb/store/tikv.(*RegionCache).findRegionByKey\n\t/home/jenkins/agent/workspace/optimi
zation-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/store/tikv/region_cache.go:575\ngithub.com/pingcap/tidb/store/tikv.(*RegionCache).LocateKey\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/
pingcap/tidb/store/tikv/region_cache.go:535\ngithub.com/pingcap/tidb/store/tikv.splitRanges\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/store/tikv/coprocessor.go:323\ngithub.com/ping
cap/tidb/store/tikv.buildCopTasks\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/store/tikv/coprocessor.go:280\ngithub.com/pingcap/tidb/store/tikv.(*CopClient).Send\n\t/home/jenkins/age
nt/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/store/tikv/coprocessor.go:69\ngithub.com/pingcap/tidb/distsql.Select\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/ping
cap/tidb/distsql/distsql.go:49\ngithub.com/pingcap/tidb/distsql.SelectWithRuntimeStats\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/distsql/distsql.go:97\ngithub.com/pingcap/tidb/exec
utor.selectResultHook.SelectResult\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/table_reader.go:51\ngithub.com/pingcap/tidb/executor.(*dataReaderBuilder).buildTableReaderFrom
Handles\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/builder.go:2796\ngithub.com/pingcap/tidb/executor.(*IndexLookUpExecutor).buildTableReader\n\t/home/jenkins/agent/workspac
e/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/distsql.go:551\ngithub.com/pingcap/tidb/executor.(*tableWorker).executeTask\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com
/pingcap/tidb/executor/distsql.go:988\ngithub.com/pingcap/tidb/executor.(*tableWorker).pickAndExecTask\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/distsql.go:871\ngithub.com
/pingcap/tidb/executor.(*IndexLookUpExecutor).startTableWorker.func1\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/distsql.go:527\nruntime.goexit\n\t/usr/local/go/src/runtime/
asm_amd64.s:1357"]
[2024/10/23 11:22:21.192 +08:00] [WARN] [backoff.go:329] ["pdRPC backoffer.maxSleep 10000ms is exceeded, errors:\nregion not found for key \"t\\x80\\x00\\x00\\x00\\x00\\x00\\x00\\xe2_r\\x80\\x00\\x00\\x04g4\\xfd\\xa1\" at 2024-10-23T11:
22:15.956137022+08:00\nregion not found for key \"t\\x80\\x00\\x00\\x00\\x00\\x00\\x00\\xe2_r\\x80\\x00\\x00\\x04g4\\xfd\\xa1\" at 2024-10-23T11:22:18.351513964+08:00\nregion not found for key \"t\\x80\\x00\\x00\\x00\\x00\\x00\\x00\\xe2
_r\\x80\\x00\\x00\\x04g4\\xfd\\xa1\" at 2024-10-23T11:22:21.192103657+08:00"]
[2024/10/23 11:22:21.192 +08:00] [ERROR] [distsql.go:1009] ["table reader fetch next chunk failed"] [conn=200424] [error="[tikv:9001]PD server timeout"]

兄弟,看我的,你还突然飙升,我这就是常态。 :joy:


你这貌似都是]PD server timeout"] 这个,是不是和连接PD的网络有关系?

只有这张表,看着是锁冲突导致的。

拿着这个表,去慢查询和sql分析里面 搜一搜对应的数据,看看有没有对应sql执行慢的信息。

都是慢SQL,没有因为跑错索引或者消耗高的SQL

那如果是因为突然并发高锁冲突导致,就只能找开发,代码层面去解决了。

解决慢sql,基本所有指标就都快正常了 :sweat_smile:

该升级了,4.0已经EOL了,以后也不会有小版本升级了。

检查 TiKV 节点的监控指标,确认是否存在资源瓶颈。如果是,可以考虑扩展集群,增加节点数量,或优化查询和数据结构。

查过丢包率吗?

查出来了,是因为region split导致的抖动,但是为啥会报这个错没看出来。

升级吧

升级吧,4.0都结束生命周期了。