在进行BR恢复的时候恢复到97%的时候其中一台TiKV被打挂了

zhengjunbo · 2020 年5 月 28 日 15:46

为提高效率，提问时请提供以下信息，问题描述清晰可优先响应。

【TiDB 版本】：4.0.0.RC2
【问题描述】：恢复到97%的时候TiKV挂了，该TiKV和BR是放在一起的。

在BR显示

[2020/05/28 19:16:22.539 +08:00] [ERROR] [import.go:309] ["ingest file failed"] [file="name:\"6_416805_1346_fba04601ea882fe8ddfba24421a14dc94e15f8876bf5fe41ccdd5868d0c13d8d_default.sst\" sha256:\"\\021\\023f\\022\\353-47\\307\\215\\001pS\\374k\\264\\237\\345\\355\\267\\247\\245\\260\\312\\270a\\177\\345O\\372\\254\\302\" start_key:\"t\\200\\000\\000\\000\\000\\000\\007\\355_r\\206\\310\\212\\373Y\\014\\240\\000\" end_key:\"t\\200\\000\\000\\000\\000\\000\\007\\355_r\\206\\310\\220E\\023\\215\\260\\000\" end_version:416974625298710529 crc64xor:12303497799414833925 total_kvs:352896 total_bytes:102903596 cf:\"default\" size:45683144 "] [range="start:\"t\\200\\000\\000\\000\\000\\000\\013\\377\\354_r\\206\\310\\212\\373Y\\377\\014\\240\\000\\000\\000\\000\\000\\000\\372\" end:\"t\\200\\000\\000\\000\\000\\000\\013\\377\\354_r\\206\\310\\220E\\023\\377L\\200\\000\\000\\000\\000\\000\\000\\372\" "] [region="id:644437 start_key:\"t\\200\\000\\000\\000\\000\\000\\013\\377\\354_r\\206\\310\\212\\373Y\\377\\014\\240\\000\\000\\000\\000\\000\\000\\372\" end_key:\"t\\200\\000\\000\\000\\000\\000\\013\\377\\354_r\\206\\310\\220E\\023\\377\\215\\260\\000\\000\\000\\000\\000\\000\\372\" region_epoch:<conf_ver:6 version:12795 > peers:<id:644438 store_id:1 > peers:<id:644439 store_id:4 > peers:<id:644440 store_id:6 > "] [error="rpc error: code = Unavailable desc = transport is closing"] [errorVerbose="rpc error: code = Unavailable desc = transport is closing\
github.com/pingcap/errors.AddStack\
\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20190809092503-95897b64e011/errors.go:174\
github.com/pingcap/errors.Trace\
\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20190809092503-95897b64e011/juju_adaptor.go:15\
github.com/pingcap/br/pkg/restore.(*FileImporter).ingestSST\
\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/restore/import.go:456\
github.com/pingcap/br/pkg/restore.(*FileImporter).Import.func1\
\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/restore/import.go:258\
github.com/pingcap/br/pkg/utils.WithRetry\
\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/utils/retry.go:29\
github.com/pingcap/br/pkg/restore.(*FileImporter).Import\
\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/restore/import.go:203\
github.com/pingcap/br/pkg/restore.(*Client).RestoreFiles.func2\
\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/restore/client.go:493\
github.com/pingcap/br/pkg/utils.(*WorkerPool).Apply.func1\
\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/utils/worker.go:47\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [stack="github.com/pingcap/log.Error\
\t/go/pkg/mod/github.com/pingcap/log@v0.0.0-20200117041106-d28c14d3b1cd/global.go:42\
github.com/pingcap/br/pkg/restore.(*FileImporter).Import.func1\
\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/restore/import.go:309\
github.com/pingcap/br/pkg/utils.WithRetry\
\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/utils/retry.go:29\
github.com/pingcap/br/pkg/restore.(*FileImporter).Import\
\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/restore/import.go:203\
github.com/pingcap/br/pkg/restore.(*Client).RestoreFiles.func2\
\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/restore/client.go:493\
github.com/pingcap/br/pkg/utils.(*WorkerPool).Apply.func1\
\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/utils/worker.go:47"]
[2020/05/28 19:16:22.540 +08:00] [WARN] [backoff.go:74] ["unexcepted error, stop to retry"] [error="rpc error: code = Unavailable desc = transport is closing"] [errorVerbose="rpc error: code = Unavailable desc = transport is closing\
github.com/pingcap/errors.AddStack\
\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20190809092503-95897b64e011/errors.go:174\
github.com/pingcap/errors.Trace\
\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20190809092503-95897b64e011/juju_adaptor.go:15\
github.com/pingcap/br/pkg/restore.(*FileImporter).ingestSST\
\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/restore/import.go:456\
github.com/pingcap/br/pkg/restore.(*FileImporter).Import.func1\
\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/restore/import.go:258\
github.com/pingcap/br/pkg/utils.WithRetry\
\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/utils/retry.go:29\
github.com/pingcap/br/pkg/restore.(*FileImporter).Import\
\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/restore/import.go:203\
github.com/pingcap/br/pkg/restore.(*Client).RestoreFiles.func2\
\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/restore/client.go:493\
github.com/pingcap/br/pkg/utils.(*WorkerPool).Apply.func1\
\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/utils/worker.go:47\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357"]
[2020/05/28 19:16:22.540 +08:00] [ERROR] [client.go:504] ["restore files failed"] [error="rpc error: code = Unavailable desc = transport is closing"] [errorVerbose="rpc error: code = Unavailable desc = transport is closing\
github.com/pingcap/errors.AddStack\
\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20190809092503-95897b64e011/errors.go:174\
github.com/pingcap/errors.Trace\
\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20190809092503-95897b64e011/juju_adaptor.go:15\
github.com/pingcap/br/pkg/restore.(*FileImporter).ingestSST\
\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/restore/import.go:456\
github.com/pingcap/br/pkg/restore.(*FileImporter).Import.func1\
\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/restore/import.go:258\
github.com/pingcap/br/pkg/utils.WithRetry\
\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/utils/retry.go:29\
github.com/pingcap/br/pkg/restore.(*FileImporter).Import\
\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/restore/import.go:203\
github.com/pingcap/br/pkg/restore.(*Client).RestoreFiles.func2\
\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/restore/client.go:493\
github.com/pingcap/br/pkg/utils.(*WorkerPool).Apply.func1\
\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/utils/worker.go:47\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [stack="github.com/pingcap/log.Error\
\t/go/pkg/mod/github.com/pingcap/log@v0.0.0-20200117041106-d28c14d3b1cd/global.go:42\
github.com/pingcap/br/pkg/restore.(*Client).RestoreFiles\
\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/restore/client.go:504\
github.com/pingcap/br/pkg/task.RunRestore\
\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/task/restore.go:270\
github.com/pingcap/br/cmd.runRestoreCommand\
\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/cmd/restore.go:21\
github.com/pingcap/br/cmd.newDbRestoreCommand.func1\
\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/cmd/restore.go:93\
github.com/spf13/cobra.(*Command).execute\
\t/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:826\
github.com/spf13/cobra.(*Command).ExecuteC\
\t/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914\
github.com/spf13/cobra.(*Command).Execute\
\t/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864\
main.main\
\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/main.go:54\
runtime.main\
\t/usr/local/go/src/runtime/proc.go:203"]
[2020/05/28 19:16:23.712 +08:00] [INFO] [client.go:134] ["Restore client closed"]
[2020/05/28 19:16:23.713 +08:00] [INFO] [ddl_worker.go:124] ["[ddl] DDL worker closed"] [worker="worker 1, tp general"] ["take time"=9.178µs]
[2020/05/28 19:16:23.713 +08:00] [INFO] [ddl_worker.go:124] ["[ddl] DDL worker closed"] [worker="worker 2, tp add index"] ["take time"=3.55µs]
[2020/05/28 19:16:23.713 +08:00] [INFO] [delete_range.go:123] ["[ddl] closing delRange"]
[2020/05/28 19:16:23.713 +08:00] [INFO] [session_pool.go:85] ["[ddl] closing sessionPool"]
[2020/05/28 19:16:23.714 +08:00] [INFO] [ddl.go:407] ["[ddl] DDL closed"] [ID=c0a8b238-53f8-40db-bf96-bd1e4661669a] ["take time"=2.023493ms]
[2020/05/28 19:16:23.714 +08:00] [INFO] [ddl.go:301] ["[ddl] stop DDL"] [ID=c0a8b238-53f8-40db-bf96-bd1e4661669a]
[2020/05/28 19:16:23.714 +08:00] [WARN] [domain.go:378] ["loading schema takes a long time"] ["take time"=1.08233189s]
[2020/05/28 19:16:23.714 +08:00] [INFO] [manager.go:267] ["failed to campaign"] ["owner info"="[ddl] /tidb/ddl/fg/owner ownerManager c0a8b238-53f8-40db-bf96-bd1e4661669a"] [error="context canceled"]
[2020/05/28 19:16:23.714 +08:00] [INFO] [manager.go:248] ["break campaign loop, context is done"] ["owner info"="[ddl] /tidb/ddl/fg/owner ownerManager c0a8b238-53f8-40db-bf96-bd1e4661669a"]
[2020/05/28 19:16:23.715 +08:00] [INFO] [manager.go:292] ["revoke session"] ["owner info"="[ddl] /tidb/ddl/fg/owner ownerManager c0a8b238-53f8-40db-bf96-bd1e4661669a"] []
[2020/05/28 19:16:23.723 +08:00] [INFO] [domain.go:512] ["reload schema in loop, schema syncer need restart"]
[2020/05/28 19:16:23.723 +08:00] [INFO] [schema_validator.go:96] ["the schema validator stops"]
[2020/05/28 19:16:24.965 +08:00] [WARN] [manager.go:170] ["failed to new session to etcd"] [ownerInfo="[ddl-syncer] /tidb/ddl/all_schema_versions/c0a8b238-53f8-40db-bf96-bd1e4661669a"] [error="rpc error: code = Canceled desc = grpc: the client connection is closing"]
[2020/05/28 19:16:25.165 +08:00] [INFO] [domain.go:582] ["domain is closed"]
[2020/05/28 19:16:25.165 +08:00] [ERROR] [domain.go:522] ["reload schema in loop, schema syncer restart failed"] [error="rpc error: code = Canceled desc = grpc: the client connection is closing"] [errorVerbose="rpc error: code = Canceled desc = grpc: the client connection is closing\
github.com/pingcap/errors.AddStack\
\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20190809092503-95897b64e011/errors.go:174\
github.com/pingcap/errors.Trace\
\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20190809092503-95897b64e011/juju_adaptor.go:15\
github.com/pingcap/tidb/owner.contextDone\
\t/go/pkg/mod/github.com/pingcap/tidb@v0.0.0-20200401141416-959eca8f3a39/owner/manager.go:380\
github.com/pingcap/tidb/owner.NewSession\
\t/go/pkg/mod/github.com/pingcap/tidb@v0.0.0-20200401141416-959eca8f3a39/owner/manager.go:142\
github.com/pingcap/tidb/ddl/util.(*schemaVersionSyncer).Restart\
\t/go/pkg/mod/github.com/pingcap/tidb@v0.0.0-20200401141416-959eca8f3a39/ddl/util/syncer.go:216\
github.com/pingcap/tidb/domain.(*Domain).mustRestartSyncer\
\t/go/pkg/mod/github.com/pingcap/tidb@v0.0.0-20200401141416-959eca8f3a39/domain/domain.go:547\
github.com/pingcap/tidb/domain.(*Domain).loadSchemaInLoop\
\t/go/pkg/mod/github.com/pingcap/tidb@v0.0.0-20200401141416-959eca8f3a39/domain/domain.go:520\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [stack="github.com/pingcap/tidb/domain.(*Domain).loadSchemaInLoop\
\t/go/pkg/mod/github.com/pingcap/tidb@v0.0.0-20200401141416-959eca8f3a39/domain/domain.go:522"]
[2020/05/28 19:16:25.165 +08:00] [INFO] [domain.go:512] ["reload schema in loop, schema syncer need restart"]
[2020/05/28 19:16:25.165 +08:00] [INFO] [schema_validator.go:96] ["the schema validator stops"]
[2020/05/28 19:16:26.394 +08:00] [WARN] [manager.go:170] ["failed to new session to etcd"] [ownerInfo="[ddl-syncer] /tidb/ddl/all_schema_versions/c0a8b238-53f8-40db-bf96-bd1e4661669a"] [error="rpc error: code = Canceled desc = grpc: the client connection is closing"]
[2020/05/28 19:16:26.594 +08:00] [INFO] [domain.go:582] ["domain is closed"]
[2020/05/28 19:16:26.594 +08:00] [ERROR] [domain.go:522] ["reload schema in loop, schema syncer restart failed"] [error="rpc error: code = Canceled desc = grpc: the client connection is closing"] [errorVerbose="rpc error: code = Canceled desc = grpc: the client connection is closing\
github.com/pingcap/errors.AddStack\
\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20190809092503-95897b64e011/errors.go:174\
github.com/pingcap/errors.Trace\
\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20190809092503-95897b64e011/juju_adaptor.go:15\
github.com/pingcap/tidb/owner.contextDone\
\t/go/pkg/mod/github.com/pingcap/tidb@v0.0.0-20200401141416-959eca8f3a39/owner/manager.go:380\
github.com/pingcap/tidb/owner.NewSession\
\t/go/pkg/mod/github.com/pingcap/tidb@v0.0.0-20200401141416-959eca8f3a39/owner/manager.go:142\
github.com/pingcap/tidb/ddl/util.(*schemaVersionSyncer).Restart\
\t/go/pkg/mod/github.com/pingcap/tidb@v0.0.0-20200401141416-959eca8f3a39/ddl/util/syncer.go:216\
github.com/pingcap/tidb/domain.(*Domain).mustRestartSyncer\
\t/go/pkg/mod/github.com/pingcap/tidb@v0.0.0-20200401141416-959eca8f3a39/domain/domain.go:547\
github.com/pingcap/tidb/domain.(*Domain).loadSchemaInLoop\
\t/go/pkg/mod/github.com/pingcap/tidb@v0.0.0-20200401141416-959eca8f3a39/domain/domain.go:520\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [stack="github.com/pingcap/tidb/domain.(*Domain).loadSchemaInLoop\
\t/go/pkg/mod/github.com/pingcap/tidb@v0.0.0-20200401141416-959eca8f3a39/domain/domain.go:522"]
[2020/05/28 19:16:26.595 +08:00] [INFO] [domain.go:607] ["domain closed"] ["take time"=2.882981392s]
[2020/05/28 19:16:26.596 +08:00] [INFO] [collector.go:180] ["Database restore Failed summary : total restore files: 153, total success: 152, total failed: 1"] ["split region"=21m24.626908186s] ["restore ranges"=46980] [unitName=file:255] [error="rpc error: code = Unavailable desc = transport is closing"] [errorVerbose="rpc error: code = Unavailable desc = transport is closing\
github.com/pingcap/errors.AddStack\
\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20190809092503-95897b64e011/errors.go:174\
github.com/pingcap/errors.Trace\
\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20190809092503-95897b64e011/juju_adaptor.go:15\
github.com/pingcap/br/pkg/restore.(*FileImporter).ingestSST\
\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/restore/import.go:456\
github.com/pingcap/br/pkg/restore.(*FileImporter).Import.func1\
\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/restore/import.go:258\
github.com/pingcap/br/pkg/utils.WithRetry\
\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/utils/retry.go:29\
github.com/pingcap/br/pkg/restore.(*FileImporter).Import\
\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/restore/import.go:203\
github.com/pingcap/br/pkg/restore.(*Client).RestoreFiles.func2\
\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/restore/client.go:493\
github.com/pingcap/br/pkg/utils.(*WorkerPool).Apply.func1\
\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/utils/worker.go:47\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357"]

目前在TiKV的error上显示

[2020/05/28 19:18:52.110 +08:00] [INFO] [apply.rs:2637] ["remove delegate from apply delegates"] [peer_id=514067] [region_id=514065]
[2020/05/28 19:18:52.110 +08:00] [2020/05/28 19:29:17.487 +08:00[2020/05/28 19:34:20.158 +08:00] [WARN] [store.rs:595] ["[store 4] handle 137 pending peers include 137 ready, 0 entries, 137 messages and 0 snapshots[2020/05/28 19:35:17.765 +08:00] [INFO] [peer.rs:502] ["peer destroy itself"] [takes=259.05µs] [peer_id=616719] [2020/05/28 19:36:12.124 +08:00] [WARN] [store.rs:595] ["[store 4] handle 4 pending peers include 4 ready, 0 entries, 4 messages and 0 snapshots[2020/05/28 19:47:06.688 +08:00] [INFO] [peer_storage.rs:1413] ["finish clear peer meta"] [takes=33.122µs] [raft_logs=3] [raft_key=1] [apply_key[2020/05/28 19:50:43.346 +08:00] [WARN] [store.rs:595] [[2020/05/28 20:01:11.278 +08:00] [WARN[[2020/05/28 20:05:41.924 +08:00] [WARN] [store.rs:595] ["[store 4] handle 1 pending peers include 1 ready, 0 entries, 1 messages and 0 snapshots"] [takes=1[[[2020/05/28 20:49:54.891 +08:00] [WARN] [store.rs:595] ["[store 4] handle 4 pending peers include 4 ready, 0 entries, 4 messages and 0 snapshots"[2020/05/28 21:09:17.917 +08:00] [WARN] [mod.rs:89] ["handle task RaftStoreEvent(RoleChange { region: id: 332385 start_key: 7480000000000007FFED5F72864CD3517CFFCDB0000000000000FA end_key: 7480000000000007FFED5F72864CD5D2F6FF43E0000000000000FA region_epoch { conf_ver: 5 version: 1336 } peers { id: 332386 store_id: 1 } peers { id: 332387 store_id: 4 } peers { id: 332388 store_id: 6 }, role: Follower })"] [takes=[2020/05/28 21:11:05.963 +08:00] [WARN] [2020/05/28 21:55:10.451 +08:00] [WARN] [store.rs:595] ["[store 4] handle 1 pending peers include 1 ready, 0 entries, 1 messages and 0 snapshots"] [takes=1]
[2020/05/28 21:59:11.358 +08:00] [WARN] [store.rs:595] [2020/05/28 22:05:26.409 +08:00] [2020/05/28 22:34:52.711 +08:00] [WARN] [store.rs:595] ["[store 4] handle 4 pending peers include 4 ready, 0 entries, 4 messages and 0 snapshots"] [takes=1]
[2020/05/28 22:37:32.636 +08:00] [WARN] [store.rs:595] ["[store 4] handle 1 pending peers include 1 ready, 0 entries, 1 messages and 0 snapshots"] [takes=1]
[[2020/05/28 22:50:54.916 +08:00] [WARN][2020/05/28 23:03:24.973 +08:00] [WARN] [store.rs:595] ["[store 4] handle 2 pending peers include 2 ready, 0 entries, 2 messages and 0 snapshots"[2020/05/28 23:16:18.455 +08:00] [WARN] [store.rs:595] ["[store 4] handle 3 pending peers include 3 ready, 0 entries, 3 messages and 0 snapshots"] [takes=1]

在TiKV的日志上反复刷,tikv没看到进程，但是日志会往里刷

2020/05/28 23:45:49.791 +08:00] [INFO] [raw_node.rs:222] ["RawNode created with id 657347."] [id=657347] [raft_id=657347] [region_id=657345]
[2020/05/28 23:45:49.791 +08:00] [INFO] [peer.rs:147] ["create peer"] [peer_id=657351] [region_id=657349]
[2020/05/28 23:45:49.791 +08:00] [INFO] [raft.rs:809] ["became follower at term 6"] [term=6] [raft_id=657351] [region_id=657349]
[2020/05/28 23:45:49.791 +08:00] [INFO] [raft.rs:288] [newRaft] [peers="[(657352, Progress { matched: 0, next_idx: 7, state: Probe, paused: false, pending_snapshot: 0, pending_request_snapshot: 0, recent_active: false, ins: Inflights { start: 0, count: 0, buffer: [] } }), (657350, Progress { matched: 0, next_idx: 7, state: Probe, paused: false, pending_snapshot: 0, pending_request_snapshot: 0, recent_active: false, ins: Inflights { start: 0, count: 0, buffer: [] } }), (657351, Progress { matched: 6, next_idx: 7, state: Probe, paused: false, pending_snapshot: 0, pending_request_snapshot: 0, recent_active: false, ins: Inflights { start: 0, count: 0, buffer: [] } })]"] ["last term"=6] ["last index"=6] [applied=6] [commit=6] [term=6] [raft_id=657351] [region_id=657349]
[2020/05/28 23:45:49.791 +08:00] [INFO] [raw_node.rs:222] ["RawNode created with id 657351."] [id=657351] [raft_id=657351] [region_id=657349]
[2020/05/28 23:45:49.791 +08:00] [INFO] [store.rs:863] ["start store"] [takes=5.102686031s] [merge_count=0] [applying_count=0] [tombstone_count=68867] [region_count=160353] [store_id=4]
[2020/05/28 23:45:49.791 +08:00] [INFO] [store.rs:914] ["cleans up garbage data"] [takes=41.620069ms] [garbage_range_count=91487] [store_id=4]
[2020/05/28 23:45:50.651 +08:00] [INFO] [mod.rs:335] ["starting working thread"] [worker=snapshot-worker]
[2020/05/28 23:45:50.651 +08:00] [INFO] [mod.rs:335] ["starting working thread"] [worker=raft-gc-worker]
[2020/05/28 23:45:50.651 +08:00] [INFO] [mod.rs:335] ["starting working thread"] [worker=cleanup-worker]
[2020/05/28 23:45:50.651 +08:00] [INFO] [future.rs:136] ["starting working thread"] [worker=pd-worker]
[2020/05/28 23:45:50.651 +08:00] [INFO] [mod.rs:335] ["starting working thread"] [worker=consistency-check]
[2020/05/28 23:45:50.651 +08:00] [WARN] [store.rs:1200] ["set thread priority for raftstore failed"] [error="Os { code: 13, kind: PermissionDenied, message: \"Permission denied\" }"]
[2020/05/28 23:45:50.651 +08:00] [INFO] [node.rs:170] ["put store to PD"] [store="id: 4 address: \"10.204.9.86:20160\" labels { key: \"host\" value: \"tikv2\" } version: \"4.0.0-rc.2\" status_address: \"10.204.9.86:20180\" git_hash: \"2fdb2804bf8ffaab4b18c4996970e19906296497\" start_timestamp: 1590680744 deploy_path: \"/data31/tidb-deploy/tikv-20160/bin\""]
[2020/05/28 23:45:50.655 +08:00] [INFO] [mod.rs:335] ["starting working thread"] [worker=cdc]
[2020/05/28 23:45:50.723 +08:00] [INFO] [future.rs:136] ["starting working thread"] [worker=waiter-manager]
[2020/05/28 23:45:50.723 +08:00] [INFO] [future.rs:136] ["starting working thread"] [worker=deadlock-detector]
[2020/05/28 23:45:50.723 +08:00] [INFO] [mod.rs:335] ["starting working thread"] [worker=backup-endpoint]
[2020/05/28 23:45:50.738 +08:00] [INFO] [<unknown>] ["Failed to add :: listener, the environment may not support IPv6: {\"created\":\"@1590680750.738553338\",\"description\":\"Address family not supported by protocol\",\"errno\":97,\"file\":\"/rust/registry/src/github.com-1ecc6299db9ec823/grpcio-sys-0.5.3/grpc/src/core/lib/iomgr/socket_utils_common_posix.cc\",\"file_line\":406,\"os_error\":\"Address family not supported by protocol\",\"syscall\":\"socket\",\"target_address\":\"[::]:20160\"}"]
[2020/05/28 23:45:50.738 +08:00] [INFO] [mod.rs:335] ["starting working thread"] [worker=snap-handler]
[2020/05/28 23:45:50.739 +08:00] [INFO] [server.rs:223] ["listening on addr"] [addr=0.0.0.0:20160]
[2020/05/28 23:45:50.739 +08:00] [INFO] [kv.rs:587] ["batch_raft RPC is called, new gRPC stream established"]
[2020/05/28 23:45:50.739 +08:00] [INFO] [kv.rs:587] ["batch_raft RPC is called, new gRPC stream established"]
[2020/05/28 23:45:50.744 +08:00] [INFO] [transport.rs:144] ["resolve store address ok"] [addr=10.204.9.85:20160] [store_id=6]
[2020/05/28 23:45:50.745 +08:00] [INFO] [raft_client.rs:48] ["server: new connection with tikv endpoint"] [addr=10.204.9.85:20160]
[2020/05/28 23:45:50.745 +08:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f7472af9540 for subchannel 0x7f74ee4691c0"]
[2020/05/28 23:45:50.746 +08:00] [INFO] [server.rs:248] ["TiKV is ready to serve"]
[2020/05/28 23:45:50.749 +08:00] [INFO] [transport.rs:144] ["resolve store address ok"] [addr=10.204.9.87:20160] [store_id=1]
[2020/05/28 23:45:50.749 +08:00] [INFO] [raft_client.rs:48] ["server: new connection with tikv endpoint"] [addr=10.204.9.87:20160]
[2020/05/28 23:45:50.750 +08:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f7472448150 for subchannel 0x7f74ee469700"]
[2020/05/28 23:45:50.756 +08:00] [ERROR] [status_server.rs:577] ["failed to register addr to pd"] [response="Response { url: \"http://10.204.9.131:2379/pd/api/v1/component\", status: 400, headers: {\"access-control-allow-headers\": \"accept, content-type, authorization\", \"access-control-allow-methods\": \"POST, GET, OPTIONS, PUT, DELETE\", \"access-control-allow-origin\": \"*\", \"content-type\": \"application/json; charset=UTF-8\", \"date\": \"Thu, 28 May 2020 15:45:23 GMT\", \"content-length\": \"71\"} }"]
[2020/05/28 23:45:50.756 +08:00] [INFO] [util.rs:398] ["connecting to PD endpoint"] [endpoints=http://10.204.9.133:2379]
[2020/05/28 23:45:50.757 +08:00] [INFO] [<unknown>] ["New connected subchannel at 0x7f74f103a540 for subchannel 0x7f750181b7c0"]
[2020/05/28 23:45:50.758 +08:00] [INFO] [util.rs:398] ["connecting to PD endpoint"] [endpoints=http://10.204.9.131:2379]
[2020/05/28 23:45:50.759 +08:00] [INFO] [util.rs:457] ["connected to PD leader"] [endpoints=http://10.204.9.131:2379]
[2020/05/28 23:45:50.759 +08:00] [INFO] [util.rs:175] ["heartbeat sender and receiver are stale, refreshing ..."]
[2020/05/28 23:45:50.808 +08:00] [ERROR] [sst_importer.rs:86] ["ingest failed"] [err="Engine Engine(\"Corruption: An external sst file with version 2 have global seqno property with value �\\u{7}G�\\u{1a}, while largest seqno in the file is 0\")"] [meta="uuid: 8EFA894A72F342728B5B5E8D55A9D556 range { start: 748000000000000BFFEC5F7286C88AFB59FF0CA0000000000000FA end: 748000000000000BFFEC5F7286C8904513FF4C80000000000000FA } cf_name: \"default\" region_id: 644437 region_epoch { conf_ver: 6 version: 12795 }"]
[2020/05/28 23:45:50.830 +08:00] [WARN] [util.rs:194] ["updating PD client done"] [spend=73.66611ms]
[2020/05/28 23:45:50.831 +08:00] [ERROR] [status_server.rs:577] ["failed to register addr to pd"] [response="Response { url: \"http://10.204.9.131:2379/pd/api/v1/component\", status: 400, headers: {\"access-control-allow-headers\": \"accept, content-type, authorization\", \"access-control-allow-methods\": \"POST, GET, OPTIONS, PUT, DELETE\", \"access-control-allow-origin\": \"*\", \"content-type\": \"application/json; charset=UTF-8\", \"date\": \"Thu, 28 May 2020 15:45:23 GMT\", \"content-length\": \"71\"} }"]
[2020/05/28 23:45:50.831 +08:00] [ERROR] [status_server.rs:577] ["failed to register addr to pd"] [response="Response { url: \"http://10.204.9.131:2379/pd/api/v1/component\", status: 400, headers: {\"access-control-allow-headers\": \"accept, content-type, authorization\", \"access-control-allow-methods\": \"POST, GET, OPTIONS, PUT, DELETE\", \"access-control-allow-origin\": \"*\", \"content-type\": \"application/json; charset=UTF-8\", \"date\": \"Thu, 28 May 2020 15:45:23 GMT\", \"content-length\": \"71\"} }"]
[2020/05/28 23:45:50.832 +08:00] [ERROR] [status_server.rs:577] ["failed to register addr to pd"] [response="Response { url: \"http://10.204.9.131:2379/pd/api/v1/component\", status: 400, headers: {\"access-control-allow-headers\": \"accept, content-type, authorization\", \"access-control-allow-methods\": \"POST, GET, OPTIONS, PUT, DELETE\", \"access-control-allow-origin\": \"*\", \"content-type\": \"application/json; charset=UTF-8\", \"date\": \"Thu, 28 May 2020 15:45:23 GMT\", \"content-length\": \"71\"} }"]
[2020/05/28 23:45:50.832 +08:00] [ERROR] [status_server.rs:577] ["failed to register addr to pd"] [response="Response { url: \"http://10.204.9.131:2379/pd/api/v1/component\", status: 400, headers: {\"access-control-allow-headers\": \"accept, content-type, authorization\", \"access-control-allow-methods\": \"POST, GET, OPTIONS, PUT, DELETE\", \"access-control-allow-origin\": \"*\", \"content-type\": \"application/json; charset=UTF-8\", \"date\": \"Thu, 28 May 2020 15:45:23 GMT\", \"content-length\": \"71\"} }"]
[2020/05/28 23:45:50.832 +08:00] [ERROR] [status_server.rs:586] ["failed to register addr to pd after 5 tries"]
[2020/05/28 23:45:51.076 +08:00] [FATAL] [lib.rs:480] ["[region 644437] 644439 ingest uuid: 8EFA894A72F342728B5B5E8D55A9D556 range { start: 748000000000000BFFEC5F7286C88AFB59FF0CA0000000000000FA end: 748000000000000BFFEC5F7286C8904513FF4C80000000000000FA } cf_name: \"default\" region_id: 644437 region_epoch { conf_ver: 6 version: 12795 }: EngineTraits(Engine(\"Corruption: An external sst file with version 2 have global seqno property with value �\\u{7}G�\\u{1a}, while largest seqno in the file is 0\"))"] [backtrace="stack backtrace:\
   0: tikv_util::set_panic_hook::{{closure}}\
             at components/tikv_util/src/lib.rs:479\
   1: std::panicking::rust_panic_with_hook\
             at src/libstd/panicking.rs:475\
   2: rust_begin_unwind\
             at src/libstd/panicking.rs:375\
   3: std::panicking::begin_panic_fmt\
             at src/libstd/panicking.rs:326\
   4: raftstore::store::fsm::apply::ApplyDelegate::handle_ingest_sst::{{closure}}\
             at components/raftstore/src/store/fsm/apply.rs:1431\
   5: core::result::Result<T,E>::unwrap_or_else\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libcore/result.rs:841\
      raftstore::store::fsm::apply::ApplyDelegate::handle_ingest_sst\
             at components/raftstore/src/store/fsm/apply.rs:1428\
   6: raftstore::store::fsm::apply::ApplyDelegate::exec_write_cmd\
             at /home/jenkins/agent/workspace/ld_tikv_multi_branch_v4.0.0-rc.2/tikv/components/raftstore/src/store/fsm/apply.rs:1204\
      raftstore::store::fsm::apply::ApplyDelegate::exec_raft_cmd\
             at /home/jenkins/agent/workspace/ld_tikv_multi_branch_v4.0.0-rc.2/tikv/components/raftstore/src/store/fsm/apply.rs:1130\
      raftstore::store::fsm::apply::ApplyDelegate::apply_raft_cmd\
             at /home/jenkins/agent/workspace/ld_tikv_multi_branch_v4.0.0-rc.2/tikv/components/raftstore/src/store/fsm/apply.rs:1019\
      raftstore::store::fsm::apply::ApplyDelegate::process_raft_cmd\
             at /home/jenkins/agent/workspace/ld_tikv_multi_branch_v4.0.0-rc.2/tikv/components/raftstore/src/store/fsm/apply.rs:971\
   7: raftstore::store::fsm::apply::ApplyDelegate::handle_raft_entry_normal\
             at /home/jenkins/agent/workspace/ld_tikv_multi_branch_v4.0.0-rc.2/tikv/components/raftstore/src/store/fsm/apply.rs:860\
      raftstore::store::fsm::apply::ApplyDelegate::handle_raft_committed_entries\
             at /home/jenkins/agent/workspace/ld_tikv_multi_branch_v4.0.0-rc.2/tikv/components/raftstore/src/store/fsm/apply.rs:778\
   8: raftstore::store::fsm::apply::ApplyFsm::handle_apply\
             at /home/jenkins/agent/workspace/ld_tikv_multi_branch_v4.0.0-rc.2/tikv/components/raftstore/src/store/fsm/apply.rs:2585\
      raftstore::store::fsm::apply::ApplyFsm::handle_tasks\
             at /home/jenkins/agent/workspace/ld_tikv_multi_branch_v4.0.0-rc.2/tikv/components/raftstore/src/store/fsm/apply.rs:2887\
   9: <raftstore::store::fsm::apply::ApplyPoller<W> as batch_system::batch::PollHandler<raftstore::store::fsm::apply::ApplyFsm,raftstore::store::fsm::apply::ControlFsm>>::handle_normal\
             at /home/jenkins/agent/workspace/ld_tikv_multi_branch_v4.0.0-rc.2/tikv/components/raftstore/src/store/fsm/apply.rs:3030\
  10: batch_system::batch::Poller<N,C,Handler>::poll\
             at /home/jenkins/agent/workspace/ld_tikv_multi_branch_v4.0.0-rc.2/tikv/components/batch-system/src/batch.rs:294\
      batch_system::batch::BatchSystem<N,C>::spawn::{{closure}}\
             at /home/jenkins/agent/workspace/ld_tikv_multi_branch_v4.0.0-rc.2/tikv/components/batch-system/src/batch.rs:398\
      std::sys_common::backtrace::__rust_begin_short_backtrace\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/sys_common/backtrace.rs:136\
  11: std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}}\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/thread/mod.rs:469\
      <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/panic.rs:318\
      std::panicking::try::do_call\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/panicking.rs:292\
      std::panicking::try\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8//src/libpanic_unwind/lib.rs:78\
      std::panic::catch_unwind\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/panic.rs:394\
      std::thread::Builder::spawn_unchecked::{{closure}}\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/thread/mod.rs:468\
      core::ops::function::FnOnce::call_once{{vtable.shim}}\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libcore/ops/function.rs:232\
  12: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/liballoc/boxed.rs:1022\
  13: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/liballoc/boxed.rs:1022\
      std::sys_common::thread::start_thread\
             at src/libstd/sys_common/thread.rs:13\
      std::sys::unix::thread::Thread::new::thread_start\
             at src/libstd/sys/unix/thread.rs:80\
  14: start_thread\
  15: __clone\
"] [location=components/raftstore/src/store/fsm/apply.rs:1431] [thread_name=apply-0]

yilong · 2020 年5 月 29 日 02:30

问题已经反馈，会尽快答复，多谢

luancheng-PingCAP · 2020 年5 月 29 日 02:46

使用 br 执行 br validate checksum -s storage 看下，备份文件是否正常？

zhengjunbo · 2020 年5 月 29 日 08:07

还有其他问题可以查么。这个命令我都跑了三四个小时了。

luancheng-PingCAP · 2020 年5 月 29 日 08:10

大概是多大的数据量？备份下来多少文件？这个命令会计算 sha256 看文件是否有损坏看下日志是否正常

zhengjunbo · 2020 年5 月 29 日 08:12

大概是1T左右，日志正常，只是中间断了ssh上去的时候session断开了，重新用nohup跑，我以为一二十分钟就搞定了

luancheng-PingCAP · 2020 年5 月 29 日 09:53

大概有多少 SST 文件呢？可以发下日志和命令，我这边看下

zhengjunbo · 2020 年5 月 29 日 10:01

6万4个SST文件
具体什么日志和命令呢

luancheng-PingCAP · 2020 年5 月 29 日 10:40

确认下版本 BR 是 4.0.0-rc.2 吗？另外，6_416805_1346_fba04601ea882fe8ddfba24421a14dc94e15f8876bf5fe41ccdd5868d0c13d8d_default.sst 这个报错的文件能发一下，我们看下是不是这个文件有问题。

如果可以的话，提供一下，所有 tikv 节点 19:16 到 19:18 的报错日志。我们排查一下

zhengjunbo · 2020 年5 月 29 日 12:00

文件太大在网盘这里链接: https://pan.baidu.com/s/10L4AstA1R8IVHEgd9a9olA 提取码: bp3c

不懂就问 · 2020 年5 月 29 日 13:03

收到我们先看下日志，有信息会更新。

Connor1996 · 2020 年6 月 9 日 02:35

[2020/05/28 19:16:22.539 +08:00] [ERROR] [import.go:309] ["ingest file failed"] [file="name:\"6_416805_1346_fba04601ea882fe8ddfba24421a14dc94e15f8876bf5fe41ccdd5868d0c13d8d_default.sst\" sha256:\"\\021\\023f\\022\\353-47\\307\\215\\001pS\\374k\\264\\237\\345\\355\\267\\247\\245\\260\\312\\270a\\177\\345O\\372\\254\\302\" start_key:\"t\\200\\000\\000\\000\\000\\000\\007\\355_r\\206\\310\\212\\373Y\\014\\240\\000\" end_key:\"t\\200\\000\\000\\000\\000\\000\\007\\355_r\\206\\310\\220E\\023\\215\\260\\000\" end_version:416974625298710529 crc64xor:12303497799414833925 total_kvs:352896 total_bytes:102903596 cf:\"default\" size:45683144 "] [range="start:\"t\\200\\000\\000\\000\\000\\000\\013\\377\\354_r\\206\\310\\212\\373Y\\377\\014\\240\\000\\000\\000\\000\\000\\000\\372\" end:\"t\\200\\000\\000\\000\\000\\000\\013\\377\\354_r\\206\\310\\220E\\023\\377L\\200\\000\\000\\000\\000\\000\\000\\372\" "] [region="id:644437 start_key:\"t\\200\\000\\000\\000\\000\\000\\013\\377\\354_r\\206\\310\\212\\373Y\\377\\014\\240\\000\\000\\000\\000\\000\\000\\372\" end_key:\"t\\200\\000\\000\\000\\000\\000\\013\\377\\354_r\\206\\310\\220E\\023\\377\\215\\260\\000\\000\\000\\000\\000\\000\\372\" region_epoch:<conf_ver:6 version:12795 > peers:<id:644438 store_id:1 > peers:<id:644439 store_id:4 > peers:<id:644440 store_id:6 > "] [error="rpc error: code = Unavailable desc = transport is closing"] [errorVerbose="rpc error: code = Unavailable desc = transport is closing\
github.com/pingcap/errors.AddStack\
\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20190809092503-95897b64e011/errors.go:174\
github.com/pingcap/errors.Trace\
\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20190809092503-95897b64e011/juju_adaptor.go:15\
github.com/pingcap/br/pkg/restore.(*FileImporter).ingestSST\
\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/restore/import.go:456\
github.com/pingcap/br/pkg/restore.(*FileImporter).Import.func1\
\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/restore/import.go:258\
github.com/pingcap/br/pkg/utils.WithRetry\
\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/utils/retry.go:29\
github.com/pingcap/br/pkg/restore.(*FileImporter).Import\
\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/restore/import.go:203\
github.com/pingcap/br/pkg/restore.(*Client).RestoreFiles.func2\
\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/restore/client.go:493\
github.com/pingcap/br/pkg/utils.(*WorkerPool).Apply.func1\
\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/utils/worker.go:47\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [stack="github.com/pingcap/log.Error\
\t/go/pkg/mod/github.com/pingcap/log@v0.0.0-20200117041106-d28c14d3b1cd/global.go:42\
github.com/pingcap/br/pkg/restore.(*FileImporter).Import.func1\
\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/restore/import.go:309\
github.com/pingcap/br/pkg/utils.WithRetry\
\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/utils/retry.go:29\
github.com/pingcap/br/pkg/restore.(*FileImporter).Import\
\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/restore/import.go:203\
github.com/pingcap/br/pkg/restore.(*Client).RestoreFiles.func2\
\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/restore/client.go:493\
github.com/pingcap/br/pkg/utils.(*WorkerPool).Apply.func1\
\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/utils/worker.go:47"]

可以看到 BR 是 05/28 19:16 的时候报错的，tikv 这个是 05/28 23:45 的信息，应该已经是多次重启后的报错。怀疑第一次出问题的时候报错不是这样的，麻烦找一下 tikv 在 br 报错时间的报错。

[2020/05/28 23:45:51.076 +08:00] [FATAL] [lib.rs:480] ["[region 644437] 644439 ingest uuid: 8EFA894A72F342728B5B5E8D55A9D556 range { start: 748000000000000BFFEC5F7286C88AFB59FF0CA0000000000000FA end: 748000000000000BFFEC5F7286C8904513FF4C80000000000000FA } cf_name: \"default\" region_id: 644437 region_epoch { conf_ver: 6 version: 12795 }: EngineTraits(Engine(\"Corruption: An external sst file with version 2 have global seqno property with value �\\u{7}G�\\u{1a}, while largest seqno in the file is 0\"))"] [backtrace="stack backtrace:\
   0: tikv_util::set_panic_hook::{{closure}}\
             at components/tikv_util/src/lib.rs:479\
   1: std::panicking::rust_panic_with_hook\
             at src/libstd/panicking.rs:475\
   2: rust_begin_unwind\
             at src/libstd/panicking.rs:375\
   3: std::panicking::begin_panic_fmt\
             at src/libstd/panicking.rs:326\
   4: raftstore::store::fsm::apply::ApplyDelegate::handle_ingest_sst::{{closure}}\
             at components/raftstore/src/store/fsm/apply.rs:1431\
   5: core::result::Result<T,E>::unwrap_or_else\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libcore/result.rs:841\
      raftstore::store::fsm::apply::ApplyDelegate::handle_ingest_sst\
             at components/raftstore/src/store/fsm/apply.rs:1428\
   6: raftstore::store::fsm::apply::ApplyDelegate::exec_write_cmd\
             at /home/jenkins/agent/workspace/ld_tikv_multi_branch_v4.0.0-rc.2/tikv/components/raftstore/src/store/fsm/apply.rs:1204\
      raftstore::store::fsm::apply::ApplyDelegate::exec_raft_cmd\
             at /home/jenkins/agent/workspace/ld_tikv_multi_branch_v4.0.0-rc.2/tikv/components/raftstore/src/store/fsm/apply.rs:1130\
      raftstore::store::fsm::apply::ApplyDelegate::apply_raft_cmd\
             at /home/jenkins/agent/workspace/ld_tikv_multi_branch_v4.0.0-rc.2/tikv/components/raftstore/src/store/fsm/apply.rs:1019\
      raftstore::store::fsm::apply::ApplyDelegate::process_raft_cmd\
             at /home/jenkins/agent/workspace/ld_tikv_multi_branch_v4.0.0-rc.2/tikv/components/raftstore/src/store/fsm/apply.rs:971\
   7: raftstore::store::fsm::apply::ApplyDelegate::handle_raft_entry_normal\
             at /home/jenkins/agent/workspace/ld_tikv_multi_branch_v4.0.0-rc.2/tikv/components/raftstore/src/store/fsm/apply.rs:860\
      raftstore::store::fsm::apply::ApplyDelegate::handle_raft_committed_entries\
             at /home/jenkins/agent/workspace/ld_tikv_multi_branch_v4.0.0-rc.2/tikv/components/raftstore/src/store/fsm/apply.rs:778\
   8: raftstore::store::fsm::apply::ApplyFsm::handle_apply\
             at /home/jenkins/agent/workspace/ld_tikv_multi_branch_v4.0.0-rc.2/tikv/components/raftstore/src/store/fsm/apply.rs:2585\
      raftstore::store::fsm::apply::ApplyFsm::handle_tasks\
             at /home/jenkins/agent/workspace/ld_tikv_multi_branch_v4.0.0-rc.2/tikv/components/raftstore/src/store/fsm/apply.rs:2887\
   9: <raftstore::store::fsm::apply::ApplyPoller<W> as batch_system::batch::PollHandler<raftstore::store::fsm::apply::ApplyFsm,raftstore::store::fsm::apply::ControlFsm>>::handle_normal\
             at /home/jenkins/agent/workspace/ld_tikv_multi_branch_v4.0.0-rc.2/tikv/components/raftstore/src/store/fsm/apply.rs:3030\
  10: batch_system::batch::Poller<N,C,Handler>::poll\
             at /home/jenkins/agent/workspace/ld_tikv_multi_branch_v4.0.0-rc.2/tikv/components/batch-system/src/batch.rs:294\
      batch_system::batch::BatchSystem<N,C>::spawn::{{closure}}\
             at /home/jenkins/agent/workspace/ld_tikv_multi_branch_v4.0.0-rc.2/tikv/components/batch-system/src/batch.rs:398\
      std::sys_common::backtrace::__rust_begin_short_backtrace\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/sys_common/backtrace.rs:136\
  11: std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}}\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/thread/mod.rs:469\
      <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/panic.rs:318\
      std::panicking::try::do_call\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/panicking.rs:292\
      std::panicking::try\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8//src/libpanic_unwind/lib.rs:78\
      std::panic::catch_unwind\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/panic.rs:394\
      std::thread::Builder::spawn_unchecked::{{closure}}\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/thread/mod.rs:468\
      core::ops::function::FnOnce::call_once{{vtable.shim}}\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libcore/ops/function.rs:232\
  12: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/liballoc/boxed.rs:1022\
  13: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once\
             at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/liballoc/boxed.rs:1022\
      std::sys_common::thread::start_thread\
             at src/libstd/sys_common/thread.rs:13\
      std::sys::unix::thread::Thread::new::thread_start\
             at src/libstd/sys/unix/thread.rs:80\
  14: start_thread\
  15: __clone\
"] [location=components/raftstore/src/store/fsm/apply.rs:1431] [thread_name=apply-0]