tidb无法建立新连接,已建立连接报错: Information schema is out of date

为提高效率,提问时请提供以下信息,问题描述清晰可优先响应。

  • 【TiDB 版本】:v4.0.7
  • 【问题描述】:
    uncategorized SQLException for SQL []; SQL state [HY000]; error code [8027]; Information schema is out of date: schema failed to update in 1 lease, please make sure TiDB can connect to TiKV;

拓扑:
3个tidb,3个pd,3个tikv,2个tiflash
tiflash是2副本

操作:
业务正常增删改查,有olap的sql请求到tiflash
同时load data

现象:
tikv2 cpu被打满
tidb无法接受新连接

其中一条错误日志:
[2020/12/02 11:11:41.496 +00:00] [INFO] [conn.go:780] [“command dispatched failed”] [conn=967] [connInfo=“id:967, addr:10.211.41.105:52666 status:10, collation:utf8_general_ci, user:jcloud_billing_rw”] [command=Query] [status=“inTxn:0, autocommit:1”] [sql=“insert into tidb_accounting_sub_bill\ ( bill_id,\ \ \ bill_time,\ \ \ data_source,\ \ \ source_id,\ \ \ pin,\ \ \ start_time,\ \ \ end_time,\ \ \ site,\ \ \ region,\ \ \ app_code,\ \ \ service_code,\ \ \ resource_id,\ \ \ billing_type,\ \ \ formula,\ \ \ bill_fee,\ \ \ bill_fee2,\ \ \ discount_fee,\ \ \ coupon_id,\ \ \ coupon_fee,\ \ \ actual_fee,\ \ \ cash_coupon_fee,\ \ \ \ \ balance_pay_fee,\ \ \ cash_pay_fee,\ \ \ \ transaction_no,\ \ \ \ org,\ \ \ pay_state,\ \ \ pay_time,\ \ \ trade_type,\ \ \ formula_desc,\ \ \ is_deleted,\ \ \ create_time,\ \ \ update_time ) \ values ( 1226649735,\ \ \ ‘2020-12-02 19:11:36’,\ \ \ 1,\ \ \ ‘1226649719’,\ \ \ ‘iaas_jke_admin’,\ \ \ ‘2020-12-02 18:00:00’,\ \ \ ‘2020-12-02 18:59:59’,\ \ \ 0,\ \ \ ‘cn-north-1’,\ \ \ ‘jcloud’,\ \ \ ‘ip’,\ \ \ ‘fip-e5b1fp3vhr’,\ \ \ 1,\ \ \ '[{“key”:“bw”,“unit”:”",“value”:10.0}]’,\ \ \ 0.9200,\ \ \ 0.92,\ \ \ 0.92,\ \ \ ‘’,\ \ \ 0.00,\ \ \ 0.00,\ \ \ 0,\ \ \ \ \ 0,\ \ \ 0,\ \ \ \ ‘12096942324452304’,\ \ \ \ ‘950’,\ \ \ 1,\ \ \ ‘2020-12-02 19:11:36’,\ \ \ 1,\ \ \ ‘带宽:10M’,\ \ \ 0,\ \ \ ‘2020-12-02 19:11:36’,\ \ \ ‘2020-12-02 19:11:36’ )"] [txn_mode=PESSIMISTIC] [err="[domain:8027]Information schema is out of date: schema failed to update in 1 lease, please make sure TiDB can connect to TiKV\ngithub.com/pingcap/errors.AddStack\ \t/home/jenkins/agent/workspace/tidb_v4.0.7/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20200917111840-a15ef68f753d/errors.go:174\ github.com/pingcap/errors.Trace\ \t/home/jenkins/agent/workspace/tidb_v4.0.7/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20200917111840-a15ef68f753d/juju_adaptor.go:15\ github.com/pingcap/tidb/store/tikv.(*twoPhaseCommitter).checkSchemaValid\ \t/home/jenkins/agent/workspace/tidb_v4.0.7/go/src/github.com/pingcap/tidb/store/tikv/2pc.go:1531\ngithub.com/pingcap/tidb/store/tikv.(*twoPhaseCommitter).execute\ \t/home/jenkins/agent/workspace/tidb_v4.0.7/go/src/github.com/pingcap/tidb/store/tikv/2pc.go:1349\ngithub.com/pingcap/tidb/store/tikv.(*tikvTxn).Commit\ \t/home/jenkins/agent/workspace/tidb_v4.0.7/go/src/github.com/pingcap/tidb/store/tikv/txn.go:307\ngithub.com/pingcap/tidb/session.(*TxnState).Commit\ \t/home/jenkins/agent/workspace/tidb_v4.0.7/go/src/github.com/pingcap/tidb/session/txn.go:279\ngithub.com/pingcap/tidb/session.(*session).doCommit\ \t/home/jenkins/agent/workspace/tidb_v4.0.7/go/src/github.com/pingcap/tidb/session/session.go:432\ngithub.com/pingcap/tidb/session.(*session).doCommitWithRetry\ \t/home/jenkins/agent/workspace/tidb_v4.0.7/go/src/github.com/pingcap/tidb/session/session.go:452\ngithub.com/pingcap/tidb/session.(*session).CommitTxn\ \t/home/jenkins/agent/workspace/tidb_v4.0.7/go/src/github.com/pingcap/tidb/session/session.go:510\ngithub.com/pingcap/tidb/session.autoCommitAfterStmt\ \t/home/jenkins/agent/workspace/tidb_v4.0.7/go/src/github.com/pingcap/tidb/session/tidb.go:230\ github.com/pingcap/tidb/session.finishStmt\ \t/home/jenkins/agent/workspace/tidb_v4.0.7/go/src/github.com/pingcap/tidb/session/tidb.go:196\ github.com/pingcap/tidb/session.runStmt\ \t/home/jenkins/agent/workspace/tidb_v4.0.7/go/src/github.com/pingcap/tidb/session/tidb.go:316\ github.com/pingcap/tidb/session.(*session).executeStatement\ \t/home/jenkins/agent/workspace/tidb_v4.0.7/go/src/github.com/pingcap/tidb/session/session.go:1061\ngithub.com/pingcap/tidb/session.(*session).execute\ \t/home/jenkins/agent/workspace/tidb_v4.0.7/go/src/github.com/pingcap/tidb/session/session.go:1173\ngithub.com/pingcap/tidb/session.(*session).Execute\ \t/home/jenkins/agent/workspace/tidb_v4.0.7/go/src/github.com/pingcap/tidb/session/session.go:1104\ngithub.com/pingcap/tidb/server.(*TiDBContext).Execute\ \t/home/jenkins/agent/workspace/tidb_v4.0.7/go/src/github.com/pingcap/tidb/server/driver_tidb.go:248\ngithub.com/pingcap/tidb/server.(*clientConn).handleQuery\ \t/home/jenkins/agent/workspace/tidb_v4.0.7/go/src/github.com/pingcap/tidb/server/conn.go:1335\ngithub.com/pingcap/tidb/server.(*clientConn).dispatch\ \t/home/jenkins/agent/workspace/tidb_v4.0.7/go/src/github.com/pingcap/tidb/server/conn.go:966\ngithub.com/pingcap/tidb/server.(*clientConn).Run\ \t/home/jenkins/agent/workspace/tidb_v4.0.7/go/src/github.com/pingcap/tidb/server/conn.go:765\ngithub.com/pingcap/tidb/server.(*Server).onConn\ \t/home/jenkins/agent/workspace/tidb_v4.0.7/go/src/github.com/pingcap/tidb/server/server.go:421\ runtime.goexit\ \t/usr/local/go/src/runtime/asm_amd64.s:1357"]

其他日志可以继续补充。谢谢。

tidb 每隔一定周期都需要去 tikv 加载 schema (表结构等)信息,以确保不同的 tidb 节点之间,看到的元信息没有落后。如果 tidb 不能加载到 schema ,它就不知道其它 tidb 节点有没有执行 DDL 修改 schema,它就不能确保它用缓存的 schema 信息去操作数据库对不对

这里出现 information schema out of data 问题的根本原因是 tikv 压力太大了,tikv 忙不过来,tidb 加载 schema 收不到响应,于是就不能服务了

建议的解决方案是,扩容tikv; 从慢日志中找出并优化掉影响 TP 类的 AP 请求;load data 类请求找业务低峰期,错峰执行;

1 个赞

麻烦问下:tidb加载schema收不到响应的错误日志关键字是什么?我在我们的日志中搜索下。

你提供的日志就是: Information schema is out of date

information schema is out of date 是使用时检查报错吧,有没有获取失败的关键字,我看下报错的确切时间?或者代码层面的,更新的过程在哪里?谢谢。

1、是使用时检查报错,但是这个报错是指超时报错(非获取动作的报错)
2、获取动作的报错具体是没有的,但日志中会有其他报错信息:比如 tikv-server timeout
3、对于为什么这么设计及报错的影响原理什么的,可以看看:如下几篇文章(内容较多)
https://docs.pingcap.com/zh/tidb/stable/tidb-computing#元信息管理
https://github.com/ngaut/builddatabase/blob/master/f1/schema-change.md

好的,感谢。

好的:ok_hand:

这个问题你最后是怎么解决的?我也遇到了相同的问题

降低tikv的压力,等下一波tidb能获取到schema后就行了。