err="other error: Coprocessor task terminated due to exceeding the deadline

【 TiDB 使用环境】线上、测试、调研
线上
【 TiDB 版本】
V5.4.0
【遇到的问题】
【复现路径】做过哪些操作出现的问题
sqoop导数据导tidb失败,某几张表不能读了
【问题现象及影响】
表不可读
select count(*) 查询报错err="other error: Coprocessor task terminated due to exceeding the deadline
tikv一直报错:
[2022/08/10 10:20:25.116 +08:00] [WARN] [endpoint.rs:606] [error-response] [err=“Coprocessor task terminated due to exceeding the deadline”]
[2022/08/10 10:20:25.126 +08:00] [WARN] [endpoint.rs:606] [error-response] [err=“Coprocessor task terminated due to exceeding the deadline”]
[2022/08/10 10:20:25.136 +08:00] [WARN] [endpoint.rs:606] [error-response] [err=“Coprocessor task terminated due to exceeding the deadline”]
[2022/08/10 10:20:25.137 +08:00] [WARN] [endpoint.rs:606] [error-response] [err=“Coprocessor task terminated due to exceeding the deadline”]
[2022/08/10 10:20:25.138 +08:00] [WARN] [endpoint.rs:606] [error-response] [err=“Coprocessor task terminated due to exceeding the deadline”]
[2022/08/10 10:20:25.139 +08:00] [WARN] [endpoint.rs:606] [error-response] [err=“Coprocessor task terminated due to exceeding the deadline”]
[2022/08/10 10:20:25.139 +08:00] [WARN] [endpoint.rs:606] [error-response] [err=“Coprocessor task terminated due to exceeding the deadline”]
[2022/08/10 10:20:25.140 +08:00] [WARN] [endpoint.rs:606] [error-response] [err=“Coprocessor task terminated due to exceeding the deadline”]
[2022/08/10 10:20:25.141 +08:00] [WARN] [endpoint.rs:606] [error-response] [err=“Coprocessor task terminated due to exceeding the deadline”]
[2022/08/10 10:20:25.162 +08:00] [WARN] [endpoint.rs:606] [error-response] [err=“Coprocessor task terminated due to exceeding the deadline”]
[2022/08/10 10:20:25.170 +08:00] [WARN] [endpoint.rs:606] [error-response] [err=“Coprocessor task terminated due to exceeding the deadline”]
[2022/08/10 10:20:25.173 +08:00] [WARN] [endpoint.rs:606] [error-response] [err=“Coprocessor task terminated due to exceeding the deadline”]
[2022/08/10 10:20:25.283 +08:00] [WARN] [endpoint.rs:606] [error-response] [err=“Coprocessor task terminated due to exceeding the deadline”]
[2022/08/10 10:20:25.289 +08:00] [WARN] [endpoint.rs:606] [error-response] [err=“Coprocessor task terminated due to exceeding the deadline”]
[2022/08/10 10:20:25.290 +08:00] [WARN] [endpoint.rs:606] [error-response] [err=“Coprocessor task terminated due to exceeding the deadline”]
[2022/08/10 10:20:25.291 +08:00] [WARN] [endpoint.rs:606] [error-response] [err=“Coprocessor task terminated due to exceeding the deadline”]

【附件】

请提供各个组件的 version 信息,如 cdc/tikv,可通过执行 cdc version/tikv-server --version 获取。

现在已经正常,很奇怪
解决步骤
1、迁移某张表region的leader,迁移后查还是异常(应该无效)
2、重启tidb日志看到,某个一直出现的tikv,[wait_max_addr=xxx:20162](重启后查询正常)
有点奇怪,所以tikv都狂刷Coprocessor task terminated due to exceeding the deadline

重启一个后,都不刷了

  1. Coprocessor task terminated due to exceeding the deadline 这是 tikv 反给 tidb 的 error;
  2. 在 tidb select 数据的时候,他们之间会 build 一个 stream,如果中间出现报错就会在 tidb 里刷这种日志。

重启之后正常的预期行为,因为重启后,该 grpc stream 不存在了,查询请求也不存在了。

还是要查一下为什么超时,结合监控,和日志,是超过了事务最大之行时间? 还是 coprocessor 打满? 等问题…

2 个赞

此话题已在最后回复的 60 天后被自动关闭。不再允许新回复。