Information schema is out of date: schema failed to update in 1 lease, please make sure TiDB can connect to TiKV"]

tidb日志里频繁报
[2022/03/29 16:40:46.697 +08:00] [INFO] [2pc.go:1630] [“Schema changed for async commit txn”] [error="[domain:8027]Information schema is out of date: schema failed to update in 1 lease, please make sure TiDB can connect to TiKV"] [startTS=432155719544340481]
[2022/03/29 16:40:46.697 +08:00] [WARN] [session.go:722] [“can not retry txn”] [conn=21907] [label=general] [error="[domain:8027]Information schema is out of date: schema failed to update in 1 lease, please make sure TiDB can connect to TiKV"] [IsBatchInsert=false] [IsPessimistic=false] [InRestrictedSQL=false] [tidb_retry_limit=10] [tidb_disable_txn_auto_retry=true]
[2022/03/29 16:40:46.697 +08:00] [WARN] [session.go:738] [“commit failed”] [conn=21907] [“finished txn”=“Txn{state=invalid}”] [error="[domain:8027]Information schema is out of date: schema failed to update in 1 lease, please make sure TiDB can connect to TiKV"]
[2022/03/29 16:40:46.697 +08:00] [WARN] [session.go:1686] [“run statement failed”] [conn=21907] [schemaVersion=2263] [error="[domain:8027]Information schema is out of date: schema failed to update in 1 lease, please make sure TiDB can connect to TiKV"] [session="{\ “currDBName”: “ps_accounting”,\ “id”: 21907,\ “status”: 2,\ “strictMode”: true,\ “user”: {\ “Username”: “ps_accounting_admin”,\ “Hostname”: “10.18.48.181”,\ “CurrentUser”: false,\ “AuthUsername”: “ps_accounting_admin”,\ “AuthHostname”: “%”\ }\ }"]

应用日志里报
“exceptionMessage”:"\ ### Error updating database. Cause: java.sql.SQLException: Information schema is out of date: schema failed to update in 1 lease, please make sure TiDB can connect to TiKV\ ### The error may involve com.gz.eim.ac.base.mapper.AcBaseOperationLogMapper.updateById-Inline\ ### The error occurred while setting parameters\ ### SQL: UPDATE

tikv日志
[2022/03/29 16:22:40.393 +08:00] [WARN] [prewrite.rs:490] [“commit_ts is too large, fallback to normal 2PC”] [lock=“Lock { lock_type: Put, primary_key: 748000000000000CB85F698000000000000001038000000000000009038000017FD4C508BE, start_ts: TimeStamp(432155435197005829), ttl: 20008, short_value: 80000E00000002030405060708090A0B0C0D0E1008001000140015001D001E001F002000340035003600370037003700000000A4857AAC19000000A4857AAC195BB12E0009BE08C5D47F01000000000069647474625F7461736B5F74696D65725F6A6F62000000, for_update_ts: TimeStamp(432155435197005829), txn_size: 1, min_commit_ts: TimeStamp(432155435210113026), use_async_commit: true, secondaries: [], rollback_ts: [] }”] [max_commit_ts=432155435728109573] [min_commit_ts=432155436075188250] [start_ts=432155435197005829] [key=748000000000000CFFB85F728000000000FF2F26DE0000000000FA]

业务是周日晚上开始使用的, 周一正常使用,今天凌晨三点多有个数仓的拉数据任务,将集群拉的负载比较高,开始频繁报 schema is out of date ,tikv机器内存被打满, 11点左右恢复后, 下午偶尔还是报这个 schema is out of date 错误。

可以参考下官方文档
https://docs.pingcap.com/zh/tidb/stable/sql-faq#触发-information-schema-is-out-of-date-错误的原因

@啦啦啦啦啦 这个看过的,

  • TiDB 在执行此 DML 时,有一段时间内连不上 PD 或者 TiKV,导致 TiDB 超过一个 DDL lease 时间没有 load schema,或者导致 TiDB 断开与 PD 之间带 keep alive 设置的连接。

看起来跟这个情况近一点,但但在tidb \ tikv \pd压力现在都不大。

我们会有比较大的查询,原来业务在2.1集群,没有什么问题,迁移到5.2.3上才出现的这个情况。这个如何优化。

如果tikv压力不大的话检查下TiDB,PD和TiKV 的网络吧

感觉是大查询导致的性能问题。dashboard里看下慢查询,分析下执行计划。 不知道你说的大的查询是不是AP类的。AP类可以考虑走tilfash。

该主题在最后一个回复创建后60天后自动关闭。不再允许新的回复。