tidb service 组件一直重启

【 TiDB 使用环境】生产环境
【 TiDB 版本】v6.5.1
【复现路径】
【遇到的问题:问题现象及影响】
【资源配置】进入到 TiDB Dashboard -集群信息 (Cluster Info) -主机(Hosts) 截图此页面
【附件:截图/日志/监控】


tidb service 组件一直重启,以下是重启的报错日志 这个是为啥。
[2024/07/05 11:06:38.058 +08:00] [FATAL] [prewrite.go:345] [“one pc happened multiple times”] [startTS=450926686506057747] [stack=“github.com/tikv/client-go/v2/txnkv/transaction.actionPrewrite.handleSingleBatch\n\t/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.4-0.20230131081004-cd83d1507d70/txnkv/transaction/prewrite.go:345\ngithub.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnBatches\n\t/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.4-0.20230131081004-cd83d1507d70/txnkv/transaction/2pc.go:1035\ngithub.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnGroupMutations\n\t/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.4-0.20230131081004-cd83d1507d70/txnkv/transaction/2pc.go:1016\ngithub.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnMutations\n\t/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.4-0.20230131081004-cd83d1507d70/txnkv/transaction/2pc.go:787\ngithub.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).prewriteMutations\n\t/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.4-0.20230131081004-cd83d1507d70/txnkv/transaction/prewrite.go:438\ngithub.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).execute\n\t/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.4-0.20230131081004-cd83d1507d70/txnkv/transaction/2pc.go:1480\ngithub.com/tikv/client-go/v2/txnkv/transaction.(*KVTxn).Commit\n\t/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.4-0.20230131081004-cd83d1507d70/txnkv/transaction/txn.go:502\ngithub.com/pingcap/tidb/store/driver/txn.(*tikvTxn).Commit\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/store/driver/txn/txn_driver.go:86\ngithub.com/pingcap/tidb/session.(*LazyTxn).Commit\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/txn.go:408\ngithub.com/pingcap/tidb/session.(*session).commitTxnWithTemporaryData\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:840\ngithub.com/pingcap/tidb/session.(*session).doCommit\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:726\ngithub.com/pingcap/tidb/session.(*session).doCommitWithRetry\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:972\ngithub.com/pingcap/tidb/session.(*session).CommitTxn\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:1102\ngithub.com/pingcap/tidb/session.autoCommitAfterStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/tidb.go:282\ngithub.com/pingcap/tidb/session.finishStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/tidb.go:248\ngithub.com/pingcap/tidb/session.(*execStmtResult).Close\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:2412\ngithub.com/pingcap/tidb/parser/terror.Call\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:306\ngithub.com/pingcap/tidb/statistics/handle.(*Handle).initTopNCountSum\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/handle/bootstrap.go:354\ngithub.com/pingcap/tidb/statistics/handle.(*Handle).initStatsHistograms4Chunk\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/handle/bootstrap.go:144\ngithub.com/pingcap/tidb/statistics/handle.(*Handle).initStatsHistograms\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/handle/bootstrap.go:185\ngithub.com/pingcap/tidb/statistics/handle.(*Handle).InitStats\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/handle/bootstrap.go:415\ngithub.com/pingcap/tidb/domain.(*Domain).loadStatsWorker\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/domain/domain.go:1882\ngithub.com/pingcap/tidb/util.(*WaitGroupWrapper).Run.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/wait_group_wrapper.go:33”]

看下是否和这个有关 run `bindinfo/bind` test will cause the tidb-server to crash · Issue #49189 · pingcap/tidb · GitHub

可以升级到 v6.5 最新的小版本,感觉大概率能解决

研究了下 貌似不是一个问题

堆栈不一样,但是你这个肯定也是 bug,只是没搜索到而已,建议升级,或者你试试只启动一个 tidb server 实例呢

集群中的节点时钟不同步可能导致时间戳混乱,影响事务处理逻辑。
看一下是不是这个原因

出现这个情况之前做过哪些操作呢?

1 个赞

看看升级版本能否解决,当前版本的确不高。

可以考虑升级一下版本

版本太低了,6.5都到10以上了

问下楼主,问题解决了吗?是通过升级解决的吗?