场景:
对同一个Key的大量Read-Modify-Write操作
问题:
发现当使用事务KV接口进行操作时,Get API会返回TiKV server timeout错误。堆栈跟踪如下:
ts=2020-08-11T15:07:38+08:00 level=error msg=“Failed to execute HSET” key=0a28044dd2c8af463926 field=FingerprintsTable value="\ \u000e\ \t461253038\u0010\ufffd\u001b" error="[tikv:9002]TiKV server timeout" errorVerbose="[tikv:9002]TiKV server timeout
github.com/pingcap/errors.AddStack
/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20190809092503-95897b64e011/errors.go:174github.com/pingcap/errors.Trace
/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20190809092503-95897b64e011/juju_adaptor.go:15github.com/pingcap/tidb/store/tikv.(*RegionRequestSender).onSendFail
/root/go/pkg/mod/github.com/pingcap/tidb@v1.1.0-beta.0.20200604055950-efc1c154d098/store/tikv/region_request.go:315github.com/pingcap/tidb/store/tikv.(*RegionRequestSender).sendReqToRegion
/root/go/pkg/mod/github.com/pingcap/tidb@v1.1.0-beta.0.20200604055950-efc1c154d098/store/tikv/region_request.go:256github.com/pingcap/tidb/store/tikv.(*RegionRequestSender).SendReqCtx
/root/go/pkg/mod/github.com/pingcap/tidb@v1.1.0-beta.0.20200604055950-efc1c154d098/store/tikv/region_request.go:216github.com/pingcap/tidb/store/tikv.(*RegionRequestSender).SendReq
/root/go/pkg/mod/github.com/pingcap/tidb@v1.1.0-beta.0.20200604055950-efc1c154d098/store/tikv/region_request.go:126github.com/pingcap/tidb/store/tikv.(*tikvStore).SendReq
/root/go/pkg/mod/github.com/pingcap/tidb@v1.1.0-beta.0.20200604055950-efc1c154d098/store/tikv/kv.go:401github.com/pingcap/tidb/store/tikv.(*LockResolver).getTxnStatus
/root/go/pkg/mod/github.com/pingcap/tidb@v1.1.0-beta.0.20200604055950-efc1c154d098/store/tikv/lock_resolver.go:531github.com/pingcap/tidb/store/tikv.(*LockResolver).getTxnStatusFromLock
/root/go/pkg/mod/github.com/pingcap/tidb@v1.1.0-beta.0.20200604055950-efc1c154d098/store/tikv/lock_resolver.go:450github.com/pingcap/tidb/store/tikv.(*LockResolver).resolveLocks
/root/go/pkg/mod/github.com/pingcap/tidb@v1.1.0-beta.0.20200604055950-efc1c154d098/store/tikv/lock_resolver.go:321github.com/pingcap/tidb/store/tikv.(*LockResolver).resolveLocksLite
/root/go/pkg/mod/github.com/pingcap/tidb@v1.1.0-beta.0.20200604055950-efc1c154d098/store/tikv/lock_resolver.go:295github.com/pingcap/tidb/store/tikv.(*clientHelper).ResolveLocks
/root/go/pkg/mod/github.com/pingcap/tidb@v1.1.0-beta.0.20200604055950-efc1c154d098/store/tikv/coprocessor.go:823github.com/pingcap/tidb/store/tikv.(*tikvSnapshot).get
/root/go/pkg/mod/github.com/pingcap/tidb@v1.1.0-beta.0.20200604055950-efc1c154d098/store/tikv/snapshot.go:388github.com/pingcap/tidb/store/tikv.(*tikvSnapshot).Get
/root/go/pkg/mod/github.com/pingcap/tidb@v1.1.0-beta.0.20200604055950-efc1c154d098/store/tikv/snapshot.go:309github.com/pingcap/tidb/kv.(*unionStore).Get
/root/go/pkg/mod/github.com/pingcap/tidb@v1.1.0-beta.0.20200604055950-efc1c154d098/kv/union_store.go:113github.com/pingcap/tidb/store/tikv.(*tikvTxn).Get
/root/go/pkg/mod/github.com/pingcap/tidb@v1.1.0-beta.0.20200604055950-efc1c154d098/store/tikv/txn.go:144gitlab.vmic.xyz/daas/tula/store/txnkv.(*database).HSet.func1
/root/tula/store/txnkv/hash.go:54gitlab.vmic.xyz/daas/tula/kv.RunInNewTxn
/root/tula/kv/txn.go:43gitlab.vmic.xyz/daas/tula/store/txnkv.(*database).HSet
/root/tula/store/txnkv/hash.go:43gitlab.vmic.xyz/daas/tula/command.HSet
/root/tula/command/hash.go:18gitlab.vmic.xyz/daas/tula/command.(*Executor).Execute
/root/tula/command/command.go:166gitlab.vmic.xyz/daas/tula.(*connection).execute
/root/tula/conn.go:143gitlab.vmic.xyz/daas/tula.(*connection).serve
/root/tula/conn.go:112gitlab.vmic.xyz/daas/tula.(*Server).serve.func1
/root/tula/tula.go:128
我们发现在TiKV SDK中调用发送CheckTxnStatus Request时,会返回超时错误,但是服务器的连通性是可以保证的。
TiKV 版本:3.0.12
请教一下该问题应该如何排查,谢谢!