br备份失败 table checksum finished error="[PD:client:ErrClientGetMember]error:rpc error: code = Canceled desc = context canceled target:10.90.218.70:2379 status:READY"

error log

[2022/06/15 22:42:36.232 +08:00] [INFO] [worker.go:63] ["table checksum finished"] [db=bigdata] [table=aid_realname] [Crc64Xor=2061580306211648899] [TotalKvs=1421816203] [TotalBytes=132290140021] [take=20m31.065101983s]
[2022/06/15 22:42:36.232 +08:00] [INFO] [worker.go:63] ["table checksum finished"] [db=bigdata] [table=urs_realname] [Crc64Xor=10314382278231369152] [TotalKvs=1481185054] [TotalBytes=140450231475] [take=20m30.356531647s]
[2022/06/15 22:42:36.237 +08:00] [INFO] [client.go:510] ["[pd] exit tso dispatcher"] [dc-location=global]
[2022/06/15 22:42:36.237 +08:00] [INFO] [base_client.go:296] ["[pd] cannot update member from this address"] [address=http://10.90.218.70:2379] [error="[PD:client:ErrClientGetMember]error:rpc error: code = Canceled desc = context canceled target:10.90.218.70:2379 status:READY"]
[2022/06/15 22:42:36.237 +08:00] [ERROR] [base_client.go:166] ["[pd] failed updateMember"] [error="[PD:client:ErrClientGetMember]error:rpc error: code = Canceled desc = context canceled target:10.90.218.70:2379 status:READY"] [errorVerbose="[PD:client:ErrClientGetMember]error:rpc error: code = Canceled desc = context canceled target:10.90.218.70:2379 status:READY\
github.com/tikv/pd/client.(*baseClient).updateMember\
\tgithub.com/tikv/pd@v1.1.0-beta.0.20210323121136-78679e5e209d/client/base_client.go:301\
github.com/tikv/pd/client.(*baseClient).memberLoop\
\tgithub.com/tikv/pd@v1.1.0-beta.0.20210323121136-78679e5e209d/client/base_client.go:165\
runtime.goexit\
\truntime/asm_amd64.s:1371"] [stack="github.com/tikv/pd/client.(*baseClient).memberLoop\
\tgithub.com/tikv/pd@v1.1.0-beta.0.20210323121136-78679e5e209d/client/base_client.go:166"]
[2022/06/15 22:42:36.239 +08:00] [INFO] [client.go:510] ["[pd] exit tso dispatcher"] [dc-location=global]
[2022/06/15 22:42:36.239 +08:00] [INFO] [metafile.go:505] ["exit write metas by context done"]
[2022/06/15 22:42:36.239 +08:00] [INFO] [collector.go:67] ["Full backup failed summary"] [total-ranges=194075] [ranges-succeed=194075] [ranges-failed=0] [backup-total-ranges=5857]
[2022/06/15 22:42:36.239 +08:00] [ERROR] [backup.go:41] ["failed to backup"] [error="other error: Coprocessor task terminated due to exceeding the deadline"] [errorVerbose="other error: Coprocessor task terminated due to exceeding the deadline\
github.com/pingcap/tidb/store/copr.(*copIteratorWorker).handleCopResponse\
\tgithub.com/pingcap/tidb@v1.1.0-beta.0.20210611041035-25dc28ee80f4/store/copr/coprocessor.go:897\
github.com/pingcap/tidb/store/copr.(*copIteratorWorker).handleTaskOnce\
\tgithub.com/pingcap/tidb@v1.1.0-beta.0.20210611041035-25dc28ee80f4/store/copr/coprocessor.go:747\
github.com/pingcap/tidb/store/copr.(*copIteratorWorker).handleTask\
\tgithub.com/pingcap/tidb@v1.1.0-beta.0.20210611041035-25dc28ee80f4/store/copr/coprocessor.go:639\
github.com/pingcap/tidb/store/copr.(*copIteratorWorker).run\
\tgithub.com/pingcap/tidb@v1.1.0-beta.0.20210611041035-25dc28ee80f4/store/copr/coprocessor.go:376\
runtime.goexit\
\truntime/asm_amd64.s:1371"] [stack="main.runBackupCommand\
\tcommand-line-arguments/backup.go:41\
main.newFullBackupCommand.func1\
\tcommand-line-arguments/backup.go:109\
github.com/spf13/cobra.(*Command).execute\
\tgithub.com/spf13/cobra@v1.0.0/command.go:842\
github.com/spf13/cobra.(*Command).ExecuteC\
\tgithub.com/spf13/cobra@v1.0.0/command.go:950\
github.com/spf13/cobra.(*Command).Execute\
\tgithub.com/spf13/cobra@v1.0.0/command.go:887\
main.main\
\tcommand-line-arguments/main.go:56\
runtime.main\
\truntime/proc.go:225"]
[2022/06/15 22:42:36.239 +08:00] [ERROR] [main.go:58] ["br failed"] [error="other error: Coprocessor task terminated due to exceeding the deadline"] [errorVerbose="other error: Coprocessor task terminated due to exceeding the deadline\
github.com/pingcap/tidb/store/copr.(*copIteratorWorker).handleCopResponse\
\tgithub.com/pingcap/tidb@v1.1.0-beta.0.20210611041035-25dc28ee80f4/store/copr/coprocessor.go:897\
github.com/pingcap/tidb/store/copr.(*copIteratorWorker).handleTaskOnce\
\tgithub.com/pingcap/tidb@v1.1.0-beta.0.20210611041035-25dc28ee80f4/store/copr/coprocessor.go:747\
github.com/pingcap/tidb/store/copr.(*copIteratorWorker).handleTask\
\tgithub.com/pingcap/tidb@v1.1.0-beta.0.20210611041035-25dc28ee80f4/store/copr/coprocessor.go:639\
github.com/pingcap/tidb/store/copr.(*copIteratorWorker).run\
\tgithub.com/pingcap/tidb@v1.1.0-beta.0.20210611041035-25dc28ee80f4/store/copr/coprocessor.go:376\
runtime.goexit\
\truntime/asm_amd64.s:1371"] [stack="main.main\
\tcommand-line-arguments/main.go:58\
runtime.main\
\truntime/proc.go:225"]

看起来是 集群状态有问题

["[pd] cannot update member from this address"] 
[address=http://10.90.218.70:2379]
[error="[PD:client:ErrClientGetMember]error:rpc error: 
code = Canceled desc = context canceled 
target:10.90.218.70:2379 status:READY"]
1 个赞

没看出集群状态有问题,业务也是正常请求的