Jjuicefs使用 tikv，kv_batch_rollback 有极高的QPS并且调度时间极其长

TiDBer_gQ17uyYE · 2022 年5 月 25 日 03:29

【 TiDB 使用环境】生产环境【 TiDB 版本】5.3.0 【遇到的问题】 Jjuicefs使用 tikv，kv_batch_rollback 有极高的QPS 【复现路径】做过哪些操作出现的问题【问题现象及影响】 JFS 客户端一直报错： tikv reportsServerIsBusy` retry later

pingyu · 2022 年5 月 25 日 03:41

请采集并上传诊断数据。请参考【SOP 系列 22】TiDB 集群诊断信息收集 Clinic 使用指南&资料大全

TiDBer_gQ17uyYE · 2022 年5 月 25 日 03:58

diag-fTmQz5rzY9L.zip (54.1 MB)
当前的诊断结果

TiDBer_gQ17uyYE · 2022 年5 月 25 日 04:01

我们将JFS的回收站延长到30天后，目前看 tikv 监控已经不提示server is busy ,
但是JFS 客户端还在提示 tikv reports ServerIsBusy retry later， deadline is exceeded

TiDBer_gQ17uyYE · 2022 年5 月 25 日 04:39

tikv.log (20.1 MB)

上传其中一个tikv 最新的日志信息

TiDBer_gQ17uyYE · 2022 年5 月 25 日 04:41

JFS 层的报错均是：
May 25 04:07:02 bd-cpu029 juicefs: [2022/05/25 04:07:02.423 +08:00] [WARN] [region_request.go:1405] ["tikv reports `=“region ID: 2260800, meta: id:2260800 start_key:“aidi-store-data\375K\000\000\000\000\035\235aH\000\002\000\000” end_key:“aidi-stor” region_epoch:<conf_ver:89615 version:252 > peers:<id:2292867 store_id:12 > peers:<id:2297457 store_id:1079297 > peers:<id:2298845 store_id:1079pu.cc:20161, idx: 0, reqStoreType: TiKvOnly, runStoreType: tikv”]

pingyu · 2022 年5 月 25 日 04:47

diag 版本太老，请升级 diag，然后重新采集和上传

请参考
升级：https://docs.pingcap.com/zh/tidb/v6.0/clinic-user-guide-for-tiup#准备工作
上传：https://docs.pingcap.com/zh/tidb/v6.0/clinic-user-guide-for-tiup#方式-1直接上传

diag 报错：
Error: input is not a diag package, plaease use diag v0.7.0 or newer version to package and upload

TiDBer_gQ17uyYE · 2022 年5 月 25 日 06:44

diag-fTn2CrhyPLS.zip (52.3 MB)

基于diag 0.7.0 版本采集的数据

TiDBer_gQ17uyYE · 2022 年5 月 25 日 12:17

日志报错还有：
tikv 还有一个问题：
get snapshot failed
[“get snapshot failed”] [err=“Error(Request(message: “region 2260661 is missing” region_not_found { region_id: 2260661 }))”] [cid=20109212]

TiDBer_gQ17uyYE · 2022 年5 月 25 日 12:20

JFS端相关的日志堆栈分析：
juicefs.goroutine.txt.zip (3.1 MB)

帮忙看下这个堆栈吧，起了两万多个类似的 goroutine：

goroutine 17861423 [select]:
github.com/tikv/client-go/v2/internal/retry.newBackoffFn.func1({0x2d48690, 0xc0967cea00}, 0xffffffffffffffff)
/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/internal/retry/config.go:181 +0x596
github.com/tikv/client-go/v2/internal/retry.(*Backoffer).BackoffWithCfgAndMaxSleep(0xc02acb9c20, 0xc000248d20, 0x18, {0x2cfb540, 0xc620a273e0})
/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/internal/retry/backoff.go:166 +0x64c
github.com/tikv/client-go/v2/internal/retry.(*Backoffer).Backoff(0xc02acb9c20, 0xc000248d20, {0x2cfb540, 0xc620a273e0})
/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/internal/retry/backoff.go:120 +0x247
github.com/tikv/client-go/v2/internal/loca

pingyu · 2022 年5 月 25 日 16:51

堆栈看着有点异常，反复走到 cleanupMutations。也许处理上不是太合理，我们再分析下
这可能是导致 kv_batch_rollback QPS 很高的原因

goroutine 17514450 [select]:
github.com/tikv/client-go/v2/internal/retry.newBackoffFn.func1({0x2d48690, 0xc10dc95e80}, 0xffffffffffffffff)
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/internal/retry/config.go:181 +0x596
github.com/tikv/client-go/v2/internal/retry.(*Backoffer).BackoffWithCfgAndMaxSleep(0xc00470e510, 0xc000248cc0, 0x100000000000028, {0x2cfb540, 0xc0412b6348})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/internal/retry/backoff.go:166 +0x64c
github.com/tikv/client-go/v2/internal/retry.(*Backoffer).Backoff(0xc00470e510, 0xc000248cc0, {0x2cfb540, 0xc0412b6348})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/internal/retry/backoff.go:120 +0x247
github.com/tikv/client-go/v2/txnkv/transaction.actionCleanup.handleSingleBatch({}, 0xc4c8c2ef00, 0xc1bdbd0d78, {{0x227eb9, 0x160e5, 0xfc}, {0x2d70970, 0xc26979d520}, 0x0})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/cleanup.go:79 +0x745
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnBatches(0xc4c8c2ef00, 0xc00470e510, {0x2d22900, 0x4893b48}, {0xc373603ef0, 0xc26979d320, 0xc1bdbd0f88})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:824 +0x31b
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnGroupMutations(0xc4c8c2ef00, 0xc00470e510, {0x2d22900, 0x4893b48}, {0xc373603e90, 0x1, 0x18})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:805 +0xcaf
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnMutations(0xc4c8c2ef00, 0xc000248cc0, {0x2d22900, 0x4893b48}, {0x2d70970, 0xc45f67ccc0})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:588 +0xc7
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).cleanupMutations(...)
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/cleanup.go:96
github.com/tikv/client-go/v2/txnkv/transaction.actionCleanup.handleSingleBatch({}, 0xc4c8c2ef00, 0xc1bdbd1588, {{0x227eb9, 0x160e5, 0xfc}, {0x2d70970, 0xc45f67ccc0}, 0x0})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/cleanup.go:83 +0x78d
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnBatches(0xc4c8c2ef00, 0xc00470e510, {0x2d22900, 0x4893b48}, {0xc2ed7e3d10, 0xc45f67c900, 0xc1bdbd1798})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:824 +0x31b
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnGroupMutations(0xc4c8c2ef00, 0xc00470e510, {0x2d22900, 0x4893b48}, {0xc2ed7e3a40, 0x1, 0x18})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:805 +0xcaf
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnMutations(0xc4c8c2ef00, 0xc000248cc0, {0x2d22900, 0x4893b48}, {0x2d70970, 0xc358095c80})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:588 +0xc7
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).cleanupMutations(...)
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/cleanup.go:96
github.com/tikv/client-go/v2/txnkv/transaction.actionCleanup.handleSingleBatch({}, 0xc4c8c2ef00, 0xc1bdbd1d98, {{0x227eb9, 0x160e5, 0xfc}, {0x2d70970, 0xc358095c80}, 0x0})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/cleanup.go:83 +0x78d
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnBatches(0xc4c8c2ef00, 0xc00470e510, {0x2d22900, 0x4893b48}, {0xc41ecc9380, 0xc358095c20, 0xc1bdbd1fa8})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:824 +0x31b
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnGroupMutations(0xc4c8c2ef00, 0xc00470e510, {0x2d22900, 0x4893b48}, {0xc41ecc9350, 0x1, 0x18})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:805 +0xcaf
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnMutations(0xc4c8c2ef00, 0xc000248cc0, {0x2d22900, 0x4893b48}, {0x2d70970, 0xc46797a400})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:588 +0xc7
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).cleanupMutations(...)
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/cleanup.go:96
github.com/tikv/client-go/v2/txnkv/transaction.actionCleanup.handleSingleBatch({}, 0xc4c8c2ef00, 0xc1bdbd25a8, {{0x227eb9, 0x160e5, 0xfc}, {0x2d70970, 0xc46797a400}, 0x0})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/cleanup.go:83 +0x78d
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnBatches(0xc4c8c2ef00, 0xc00470e510, {0x2d22900, 0x4893b48}, {0xc3c8feb350, 0xc46797a3e0, 0xc1bdbd27b8})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:824 +0x31b
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnGroupMutations(0xc4c8c2ef00, 0xc00470e510, {0x2d22900, 0x4893b48}, {0xc3c8feb2f0, 0x1, 0x18})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:805 +0xcaf
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnMutations(0xc4c8c2ef00, 0xc000248cc0, {0x2d22900, 0x4893b48}, {0x2d70970, 0xc09b007ca0})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:588 +0xc7
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).cleanupMutations(...)
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/cleanup.go:96
github.com/tikv/client-go/v2/txnkv/transaction.actionCleanup.handleSingleBatch({}, 0xc4c8c2ef00, 0xc1bdbd2db8, {{0x227eb9, 0x160e5, 0xfc}, {0x2d70970, 0xc09b007ca0}, 0x0})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/cleanup.go:83 +0x78d
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnBatches(0xc4c8c2ef00, 0xc00470e510, {0x2d22900, 0x4893b48}, {0xc4b9ffab70, 0xc09b007c80, 0xc1bdbd2fc8})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:824 +0x31b
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnGroupMutations(0xc4c8c2ef00, 0xc00470e510, {0x2d22900, 0x4893b48}, {0xc4b9ffab40, 0x1, 0x18})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:805 +0xcaf
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnMutations(0xc4c8c2ef00, 0xc000248cc0, {0x2d22900, 0x4893b48}, {0x2d70970, 0xc1201a9440})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:588 +0xc7
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).cleanupMutations(...)
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/cleanup.go:96
github.com/tikv/client-go/v2/txnkv/transaction.actionCleanup.handleSingleBatch({}, 0xc4c8c2ef00, 0xc1bdbd35c8, {{0x227eb9, 0x160e5, 0xfc}, {0x2d70970, 0xc1201a9440}, 0x0})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/cleanup.go:83 +0x78d
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnBatches(0xc4c8c2ef00, 0xc00470e510, {0x2d22900, 0x4893b48}, {0xc1394d6a80, 0xc1201a8e80, 0xc1bdbd37d8})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:824 +0x31b
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnGroupMutations(0xc4c8c2ef00, 0xc00470e510, {0x2d22900, 0x4893b48}, {0xc1394d6a20, 0x1, 0x18})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:805 +0xcaf
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnMutations(0xc4c8c2ef00, 0xc000248cc0, {0x2d22900, 0x4893b48}, {0x2d70970, 0xc4453bb140})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:588 +0xc7
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).cleanupMutations(...)
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/cleanup.go:96
github.com/tikv/client-go/v2/txnkv/transaction.actionCleanup.handleSingleBatch({}, 0xc4c8c2ef00, 0xc1bdbd3dd8, {{0x227eb9, 0x160e5, 0xfc}, {0x2d70970, 0xc4453bb140}, 0x0})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/cleanup.go:83 +0x78d
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnBatches(0xc4c8c2ef00, 0xc00470e510, {0x2d22900, 0x4893b48}, {0xc04ceb7f20, 0xc4453baf20, 0xc1bdbd3fe8})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:824 +0x31b
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnGroupMutations(0xc4c8c2ef00, 0xc00470e510, {0x2d22900, 0x4893b48}, {0xc04ceb7ec0, 0x1, 0x18})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:805 +0xcaf
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnMutations(0xc4c8c2ef00, 0xc000248cc0, {0x2d22900, 0x4893b48}, {0x2d70970, 0xc5b678f820})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:588 +0xc7
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).cleanupMutations(...)
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/cleanup.go:96
github.com/tikv/client-go/v2/txnkv/transaction.actionCleanup.handleSingleBatch({}, 0xc4c8c2ef00, 0xc1bdbd45e8, {{0x227eb9, 0x160e5, 0xfc}, {0x2d70970, 0xc5b678f820}, 0x0})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/cleanup.go:83 +0x78d
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnBatches(0xc4c8c2ef00, 0xc00470e510, {0x2d22900, 0x4893b48}, {0xc532760a80, 0xc5b678f7c0, 0xc1bdbd47f8})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:824 +0x31b
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnGroupMutations(0xc4c8c2ef00, 0xc00470e510, {0x2d22900, 0x4893b48}, {0xc532760990, 0x1, 0x18})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:805 +0xcaf
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnMutations(0xc4c8c2ef00, 0xc000248cc0, {0x2d22900, 0x4893b48}, {0x2d70970, 0xc0aaeed720})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:588 +0xc7
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).cleanupMutations(...)
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/cleanup.go:96
github.com/tikv/client-go/v2/txnkv/transaction.actionCleanup.handleSingleBatch({}, 0xc4c8c2ef00, 0xc1bdbd4df8, {{0x227eb9, 0x160e5, 0xfc}, {0x2d70970, 0xc0aaeed720}, 0x0})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/cleanup.go:83 +0x78d
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnBatches(0xc4c8c2ef00, 0xc00470e510, {0x2d22900, 0x4893b48}, {0xc46a9eae10, 0xc0aaeecb80, 0xc1bdbd5008})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:824 +0x31b
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnGroupMutations(0xc4c8c2ef00, 0xc00470e510, {0x2d22900, 0x4893b48}, {0xc46a9eacf0, 0x1, 0x18})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:805 +0xcaf
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnMutations(0xc4c8c2ef00, 0xc000248cc0, {0x2d22900, 0x4893b48}, {0x2d70970, 0xc061858240})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:588 +0xc7
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).cleanupMutations(...)
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/cleanup.go:96
github.com/tikv/client-go/v2/txnkv/transaction.actionCleanup.handleSingleBatch({}, 0xc4c8c2ef00, 0xc1bdbd5608, {{0x227eb9, 0x160e5, 0xfc}, {0x2d70970, 0xc061858240}, 0x0})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/cleanup.go:83 +0x78d
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnBatches(0xc4c8c2ef00, 0xc00470e510, {0x2d22900, 0x4893b48}, {0xc33a6c6330, 0xc061858100, 0xc1bdbd5818})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:824 +0x31b
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnGroupMutations(0xc4c8c2ef00, 0xc00470e510, {0x2d22900, 0x4893b48}, {0xc33a6c61e0, 0x1, 0x18})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:805 +0xcaf
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnMutations(0xc4c8c2ef00, 0xc000248cc0, {0x2d22900, 0x4893b48}, {0x2d70970, 0xc4e0b0b380})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:588 +0xc7
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).cleanupMutations(...)
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/cleanup.go:96
github.com/tikv/client-go/v2/txnkv/transaction.actionCleanup.handleSingleBatch({}, 0xc4c8c2ef00, 0xc1bdbd5e18, {{0x227eb9, 0x160e5, 0xfc}, {0x2d70970, 0xc4e0b0b380}, 0x0})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/cleanup.go:83 +0x78d
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnBatches(0xc4c8c2ef00, 0xc00470e510, {0x2d22900, 0x4893b48}, {0xc0b78e1ad0, 0xc4e0b0b340, 0xc1bdbd6028})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:824 +0x31b
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnGroupMutations(0xc4c8c2ef00, 0xc00470e510, {0x2d22900, 0x4893b48}, {0xc0b78e1950, 0x1, 0x18})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:805 +0xcaf
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnMutations(0xc4c8c2ef00, 0xc000248cc0, {0x2d22900, 0x4893b48}, {0x2d70970, 0xc3134f0620})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:588 +0xc7
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).cleanupMutations(...)
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/cleanup.go:96
github.com/tikv/client-go/v2/txnkv/transaction.actionCleanup.handleSingleBatch({}, 0xc4c8c2ef00, 0x7fc758dc6e30, {{0x227eb9, 0x160e5, 0xfc}, {0x2d70970, 0xc3134f0620}, 0x0})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/cleanup.go:83 +0x78d
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnBatches(0xc4c8c2ef00, 0xc00470e510, {0x2d22900, 0x4893b48}, {0xc4f6ee2a80, 0xc48e9d2f00, 0xc1bdbd6838})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:824 +0x31b
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnGroupMutations(0xc4c8c2ef00, 0xc00470e510, {0x2d22900, 0x4893b48}, {0xc4f6ee2990, 0x1, 0x203006})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:805 +0xcaf
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnMutations(0xc4c8c2ef00, 0xc000248cc0, {0x2d22900, 0x4893b48}, {0x2d70970, 0xc0d5fb65a0})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:588 +0xc7
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).cleanupMutations(...)
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/cleanup.go:96
github.com/tikv/client-go/v2/txnkv/transaction.actionCleanup.handleSingleBatch({}, 0xc4c8c2ef00, 0xc1bdbd6e38, {{0x227eb9, 0x160e5, 0xfc}, {0x2d70970, 0xc0d5fb65a0}, 0x0})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/cleanup.go:83 +0x78d
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnBatches(0xc4c8c2ef00, 0xc00470e510, {0x2d22900, 0x4893b48}, {0xc205a06960, 0xc0d5fb6560, 0xc1bdbd7048})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:824 +0x31b
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnGroupMutations(0xc4c8c2ef00, 0xc00470e510, {0x2d22900, 0x4893b48}, {0xc205a06780, 0x1, 0x18})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:805 +0xcaf
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnMutations(0xc4c8c2ef00, 0xc000248cc0, {0x2d22900, 0x4893b48}, {0x2d70970, 0xc29be694c0})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:588 +0xc7
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).cleanupMutations(...)
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/cleanup.go:96
github.com/tikv/client-go/v2/txnkv/transaction.actionCleanup.handleSingleBatch({}, 0xc4c8c2ef00, 0xc1bdbd7648, {{0x227eb9, 0x160e5, 0xfc}, {0x2d70970, 0xc29be694c0}, 0x0})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/cleanup.go:83 +0x78d
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnBatches(0xc4c8c2ef00, 0xc00470e510, {0x2d22900, 0x4893b48}, {0xc4c399a6f0, 0xc29be692a0, 0xc1bdbd7858})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:824 +0x31b
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnGroupMutations(0xc4c8c2ef00, 0xc00470e510, {0x2d22900, 0x4893b48}, {0xc4c399a600, 0x1, 0x18})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:805 +0xcaf
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnMutations(0xc4c8c2ef00, 0xc000248cc0, {0x2d22900, 0x4893b48}, {0x2d70970, 0xc064f44640})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:588 +0xc7
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).cleanupMutations(...)
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/cleanup.go:96
github.com/tikv/client-go/v2/txnkv/transaction.actionCleanup.handleSingleBatch({}, 0xc4c8c2ef00, 0xc1bdbd7e58, {{0x227eb9, 0x160e5, 0xfc}, {0x2d70970, 0xc064f44640}, 0x0})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/cleanup.go:83 +0x78d
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnBatches(0xc4c8c2ef00, 0xc00470e510, {0x2d22900, 0x4893b48}, {0xc1f66f6e10, 0xc064f444c0, 0xc1bdbd8068})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:824 +0x31b
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnGroupMutations(0xc4c8c2ef00, 0xc00470e510, {0x2d22900, 0x4893b48}, {0xc1f66f6d50, 0x1, 0x18})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:805 +0xcaf
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnMutations(0xc4c8c2ef00, 0xc000248cc0, {0x2d22900, 0x4893b48}, {0x2d70970, 0xc1db1e8460})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:588 +0xc7
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).cleanupMutations(...)
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/cleanup.go:96
github.com/tikv/client-go/v2/txnkv/transaction.actionCleanup.handleSingleBatch({}, 0xc4c8c2ef00, 0xc1bdbd8668, {{0x227eb9, 0x160e5, 0xfc}, {0x2d70970, 0xc1db1e8460}, 0x0})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/cleanup.go:83 +0x78d
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnBatches(0xc4c8c2ef00, 0xc00470e510, {0x2d22900, 0x4893b48}, {0xc04c167080, 0xc1db1e8120, 0xc1bdbd8878})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:824 +0x31b
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnGroupMutations(0xc4c8c2ef00, 0xc00470e510, {0x2d22900, 0x4893b48}, {0xc04c166ff0, 0x1, 0x18})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:805 +0xcaf
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnMutations(0xc4c8c2ef00, 0xc000248cc0, {0x2d22900, 0x4893b48}, {0x2d70970, 0xc51893c7c0})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:588 +0xc7
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).cleanupMutations(...)
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/cleanup.go:96
github.com/tikv/client-go/v2/txnkv/transaction.actionCleanup.handleSingleBatch({}, 0xc4c8c2ef00, 0xc1bdbd8e78, {{0x227eb9, 0x160e5, 0xfc}, {0x2d70970, 0xc51893c7c0}, 0x0})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/cleanup.go:83 +0x78d
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnBatches(0xc4c8c2ef00, 0xc00470e510, {0x2d22900, 0x4893b48}, {0xc2ae445110, 0xc51893c660, 0xc1bdbd9088})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:824 +0x31b
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnGroupMutations(0xc4c8c2ef00, 0xc00470e510, {0x2d22900, 0x4893b48}, {0xc2ae4450b0, 0x1, 0x18})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:805 +0xcaf
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnMutations(0xc4c8c2ef00, 0xc000248cc0, {0x2d22900, 0x4893b48}, {0x2d70970, 0xc3134f0e60})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:588 +0xc7
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).cleanupMutations(...)
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/cleanup.go:96
github.com/tikv/client-go/v2/txnkv/transaction.actionCleanup.handleSingleBatch({}, 0xc4c8c2ef00, 0xc1bdbd9688, {{0x227eb9, 0x160e5, 0xfc}, {0x2d70970, 0xc3134f0e60}, 0x0})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/cleanup.go:83 +0x78d
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnBatches(0xc4c8c2ef00, 0xc00470e510, {0x2d22900, 0x4893b48}, {0xc2bb542060, 0xc3134f0e00, 0xc1bdbd9898})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:824 +0x31b
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnGroupMutations(0xc4c8c2ef00, 0xc00470e510, {0x2d22900, 0x4893b48}, {0xc2bb542030, 0x1, 0x18})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:805 +0xcaf
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnMutations(0xc4c8c2ef00, 0xc000248cc0, {0x2d22900, 0x4893b48}, {0x2d70970, 0xc36bbe2de0})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:588 +0xc7
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).cleanupMutations(...)
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/cleanup.go:96
github.com/tikv/client-go/v2/txnkv/transaction.actionCleanup.handleSingleBatch({}, 0xc4c8c2ef00, 0x7fc756e30668, {{0x227eb9, 0x160e5, 0xfc}, {0x2d70970, 0xc36bbe2de0}, 0x0})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/cleanup.go:83 +0x78d
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnBatches(0xc4c8c2ef00, 0xc00470e510, {0x2d22900, 0x4893b48}, {0xc56b2072f0, 0xc093355f00, 0xc1bdbda0a8})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:824 +0x31b
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnGroupMutations(0xc4c8c2ef00, 0xc00470e510, {0x2d22900, 0x4893b48}, {0xc56b2072c0, 0x1, 0x18})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:805 +0xcaf
github.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnMutations(0xc4c8c2ef00, 0xc000248cc0, {0x2d22900, 0x4893b48}, {0x2d70970, 0xc319289440})
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:588 +0xc7
created by github.com/tikv/client-go/v2/txnkv/transaction.(*batchExecutor).startWorker
	/var/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.0-rc/txnkv/transaction/2pc.go:1787 +0x1dc

pingyu · 2022 年5 月 25 日 16:52

gRPC

Thread CPU

TiDBer_gQ17uyYE · 2022 年5 月 26 日 03:08

新提供一个情况： tikv 一直有报错提示
[2022/05/26 11:07:34.459 +08:00] [INFO] [scheduler.rs:515] [“get snapshot failed”] [err=“Error(Request(message: “peer is not leader for region 2260808, leader may Some(id: 2337963 store_id: 23)” not_leader {region_id: 2260808 leader { id: 2337963 store_id: 23 } }))”] [cid=862603068]

TiDBer_gQ17uyYE · 2022 年5 月 26 日 03:19

tikv 报错： peer id: 2337963 不是 region_id: 2260808的leader ，但是我们从数据库中读出来的两者是有从属关系的

[2022/05/26 11:07:34.526 +08:00] [INFO] [scheduler.rs:515] [“get snapshot failed”] [err=“Error(Request(message: "peer is not leader for region 2260808, leader may Some(id: 2337963 store_id: 23)" not_leader {region_id: 2260808 leader { id: 2337963 store_id: 23 } }))”] [cid=862604682]

pingyu · 2022 年5 月 26 日 06:43

看起来是系统繁忙导致的结果，并不是原因。系统繁忙后 raft 消息不能及时响应，状态也不能及时更新

pingyu · 2022 年5 月 26 日 06:43

导致系统繁忙的原因，我觉得是有两方面

其一，是事务冲突比较严重。建议考虑使用悲观事务模型来解决。可以请 JuiceFS 的同学帮忙看看

其二，client-go 在遇到事务冲突时，处理上可能有点问题。昨天的堆栈里看到 cleanup 反复多次递归的情况，可能是导致进一步加剧事务冲突的原因

如果系统当前持续繁忙，建议先通过扩容来缓解
此外，如果系统 CPU 还有余量，可以试试调高 Scheduler 的线程数
参考这里 https://docs.pingcap.com/zh/tidb/stable/tune-tikv-thread-performance

TiDBer_gQ17uyYE · 2022 年5 月 27 日 02:15

JFS 使用悲观锁可能比较时候事务冲突的场景，这个需要JFS评估影响，怀疑会导致JFS性能出现下降
tikv 的client-go ，有预计修复计划吗？

另外 region_id 和 peer_id 所属关系不正确这个有有办法解决吗？

pingyu · 2022 年5 月 27 日 10:43

client-go 这里确认了一下，并不是问题。目前是用递归实现的重试，所以 goroutine 看到了反复多次递归。见 https://github.com/tikv/client-go/blob/v2.0.0-rc/txnkv/transaction/cleanup.go#L83
导致重试的原因，结合前边的监控，应该是 ServerIsBusy
也许用循环实现重试会更好吧，避免爆堆栈，但暂时不会去优化。重试之间有指数级 backoff，所以递归的次数应该是有限的

pingyu · 2022 年5 月 27 日 10:52

“所属关系不正确”指的是什么？如果是前边的 peer is not leader for region 2260808，这句话并没有说所属关系不对啊。只是说这个 peer 不是 region 的 leader

peer is not leader 的原因，前边有说过，看起来是系统繁忙的结果。系统繁忙后 raft 消息不能及时响应，状态也不能及时更新。所以请求端以为这个 peer 是 leader，但实际上已经不是了

lidaohang · 2022 年5 月 28 日 10:53

@pingyu@PingCAP 我们发现tikv 在获取快照的时候，一致在刷not leader 但是我分别不同时间段导出region 发现leader没有变化，epoch也没变化. 你这里说的不是这个场景是不存在的，我们在出问题前后分别导出来三四次region 对应的leader就是这个，而日志一直说region leader不是这个IP