BR备份报Error: msg:"Io(Os { code: 22, kind: InvalidInput, message: \"Invalid argument\" })"

为提高效率,提问时请提供以下信息,问题描述清晰可优先响应。

  • 【TiDB 版本】:4.0.0-RC
  • 【问题描述】:在BR导出数据的时候报Error: msg:“Io(Os { code: 22, kind: InvalidInput, message: “Invalid argument” })”

执行的语句

./br backup full --pd "10.204.54.85:2379" --storage "local:///data_bak" --ratelimit 120 --log-file backupfull.log

日志中打印的错误

[2020/05/18 15:27:40.282 +08:00] [INFO] [client.go:402] ["backup started"] [StartKey="dIAAAAAAAABvX3IAAAAAAAAAAA=="] [EndKey="dIAAAAAAAABvX3L//////////wA="] [RateLimit=125829120] [Concurrency=4]
[2020/05/18 15:27:40.283 +08:00] [INFO] [push.go:29] ["new backup client"]
[2020/05/18 15:27:40.284 +08:00] [INFO] [client.go:721] ["try backup"] ["backup request"="{\"cluster_id\":6714244546456264833,\"start_key\":\"dIAAAAAAAABvX3IAAAAAAAAAAA==\",\"end_key\":\"dIAAAAAAAABvX3L//////////wA=\",\"end_version\":416753086478680158,\"rate_limit\":125829120,\"concurrency\":4,\"storage_backend\":{\"Backend\":{\"Local\":{\"path\":\"/data_bak\"}}}}"]
[2020/05/18 15:27:40.284 +08:00] [INFO] [client.go:721] ["try backup"] ["backup request"="{\"cluster_id\":6714244546456264833,\"start_key\":\"dIAAAAAAAABvX3IAAAAAAAAAAA==\",\"end_key\":\"dIAAAAAAAABvX3L//////////wA=\",\"end_version\":416753086478680158,\"rate_limit\":125829120,\"concurrency\":4,\"storage_backend\":{\"Backend\":{\"Local\":{\"path\":\"/data_bak\"}}}}"]
[2020/05/18 15:27:40.285 +08:00] [INFO] [client.go:721] ["try backup"] ["backup request"="{\"cluster_id\":6714244546456264833,\"start_key\":\"dIAAAAAAAABvX3IAAAAAAAAAAA==\",\"end_key\":\"dIAAAAAAAABvX3L//////////wA=\",\"end_version\":416753086478680158,\"rate_limit\":125829120,\"concurrency\":4,\"storage_backend\":{\"Backend\":{\"Local\":{\"path\":\"/data_bak\"}}}}"]
[2020/05/18 15:27:40.286 +08:00] [INFO] [client.go:733] ["backup streaming finish"] [StoreID=1]
[2020/05/18 15:27:40.286 +08:00] [INFO] [client.go:733] ["backup streaming finish"] [StoreID=5]
[2020/05/18 15:27:40.476 +08:00] [INFO] [client.go:740] ["range backuped"] [StartKey="dIAAAAAAAABvX3IAAAAAAAAAAA=="] [EndKey="dIAAAAAAAABvX3L//////////wA="]
[2020/05/18 15:27:40.476 +08:00] [INFO] [client.go:733] ["backup streaming finish"] [StoreID=4]
[2020/05/18 15:27:40.476 +08:00] [ERROR] [push.go:111] ["backup occur unknown error"] [error="Io(Os { code: 22, kind: InvalidInput, message: \"Invalid argument\" })"] [stack="github.com/pingcap/log.Error\n\t/go/pkg/mod/github.com/pingcap/log@v0.0.0-20200117041106-d28c14d3b1cd/global.go:42\ngithub.com/pingcap/br/pkg/backup.(*pushDown).pushBackup\n\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/backup/push.go:111\ngithub.com/pingcap/br/pkg/backup.(*Client).BackupRange\n\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/backup/client.go:424\ngithub.com/pingcap/br/pkg/backup.(*Client).BackupRanges.func2\n\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/backup/client.go:337"]
[2020/05/18 15:27:40.476 +08:00] [INFO] [client.go:394] ["backup range finished"] [take=193.818569ms]
[2020/05/18 15:27:40.476 +08:00] [INFO] [client.go:329] ["Backup Ranges"] [take=193.978007ms]
[2020/05/18 15:27:40.478 +08:00] [INFO] [ddl.go:407] ["[ddl] DDL closed"] [ID=f3b4fd37-2134-4760-b186-1adc33e0e973] ["take time"=1.866057ms]
[2020/05/18 15:27:40.478 +08:00] [INFO] [ddl.go:301] ["[ddl] stop DDL"] [ID=f3b4fd37-2134-4760-b186-1adc33e0e973]
[2020/05/18 15:27:40.480 +08:00] [INFO] [domain.go:607] ["domain closed"] ["take time"=4.354146ms]
[2020/05/18 15:27:40.481 +08:00] [INFO] [collector.go:180] ["Full backup Failed summary : total backup ranges: 1, total success: 0, total failed: 1"] ["backup total regions"=2359] [unitName="range start:74800000000000006f5f720000000000000000 end:74800000000000006f5f72ffffffffffffffff00"] [error="msg:\"Io(Os { code: 22, kind: InvalidInput, message: \\\"Invalid argument\\\" })\" "] [errorVerbose="msg:\"Io(Os { code: 22, kind: InvalidInput, message: \\\"Invalid argument\\\" })\" \ngithub.com/pingcap/br/pkg/backup.(*pushDown).pushBackup\n\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/backup/push.go:113\ngithub.com/pingcap/br/pkg/backup.(*Client).BackupRange\n\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/backup/client.go:424\ngithub.com/pingcap/br/pkg/backup.(*Client).BackupRanges.func2\n\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/backup/client.go:337\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"]

您好,请问备份的tikv实例上都有 data_bak的目录吗? 能否反馈下 df -h 的结果,另外 目录权限和属主是什么?

目前有三个TiVK实例

image

image

目前的KV实例是

不过我BR是放到其中一台KV上去的。这个应该没问题吧

你好,

br 备份会在所有 tikv 节点创建备份的目录,请确保执行 user 有相关权限。

尝试执行 ./br backup full --pd "10.204.54.85:2379" --storage "local:///data_bak" --ratelimit 120 是否存在相同报错。

权限都是TiDB。777的目录权限,而且在每个服务器上手动执行 >1 都能正常写入文件

请问 br 的版本信息是多少

4.0.0.RC 和4.0.0.RC.2都试过了呢

导出出错时间段 TiKV 的日志可以提供一下么

TIKV1

[2020/05/19 16:15:17.101 +08:00] [INFO] [gc_manager.rs:456] ["gc_worker: finished auto gc"] [processed_regions=838]
[2020/05/19 16:20:51.743 +08:00] [INFO] [local.rs:42] ["create local storage"] [base=/data_bak]
[2020/05/19 16:20:51.744 +08:00] [INFO] [endpoint.rs:698] ["run backup task"] [task="BackupTask { start_ts: TimeStamp(0), end_ts: TimeStamp(416776572192686113), start_key: \"7480000000000011C45F720000000000000000\", end_key: \"7480000000000011C45F72FFFFFFFFFFFFFFFF00\", is_raw_kv: false, cf: \"default\" }"]
[2020/05/19 16:20:51.745 +08:00] [INFO] [local.rs:42] ["create local storage"] [base=/data_bak]
[2020/05/19 16:20:51.870 +08:00] [ERROR] [endpoint.rs:284] ["backup save file failed"] [error="Io(Os { code: 22, kind: InvalidInput, message: \"Invalid argument\" })"]
[2020/05/19 16:20:51.871 +08:00] [ERROR] [endpoint.rs:669] ["backup region failed"] [error="Io(Os { code: 22, kind: InvalidInput, message: \"Invalid argument\" })"] [end_key=] [start_key=] [region="id: 4473 start_key: 7480000000000011FFC400000000000000F8 end_key: 7480000000000011FFC700000000000000F8 region_epoch { conf_ver: 5 version: 751 } peers { id: 4474 store_id: 1 } peers { id: 4475 store_id: 4 } peers { id: 4476 store_id: 5 }"]
[2020/05/19 16:20:51.871 +08:00] [INFO] [endpoint.rs:686] ["backup finished"] [summary="Statistics { lock: CfStatistics { processed: 0, get: 0, next: 0, prev: 0, seek: 0, seek_for_prev: 0, over_seek_bound: 0, flow_stats: FlowStatistics { read_keys: 0, read_bytes: 0 } }, write: CfStatistics { processed: 0, get: 0, next: 0, prev: 0, seek: 0, seek_for_prev: 0, over_seek_bound: 0, flow_stats: FlowStatistics { read_keys: 0, read_bytes: 0 } }, data: CfStatistics { processed: 0, get: 0, next: 0, prev: 0, seek: 0, seek_for_prev: 0, over_seek_bound: 0, flow_stats: FlowStatistics { read_keys: 0, read_bytes: 0 } } }"] [take=126.837833ms]
[2020/05/19 16:20:51.871 +08:00] [INFO] [service.rs:79] ["backup send half closed"]

TIKV2

[2020/05/19 16:24:14.722 +08:00] [WARN] [endpoint.rs:540] [error-response] [err="Key is locked (will clean up) primary_lock: 74800000000000001D5F69800000000000000103800000000000124B038000000000000001038000000000000003038000000001D4D009 lock_version: 416776539922759681 key: 74800000000000001D5F69800000000000000103800000000000124B038000000000000001038000000000000003038000000001D4D133 lock_ttl: 3020 txn_size: 635 lock_type: Del"]
[2020/05/19 16:26:19.219 +08:00] [INFO] [local.rs:42] ["create local storage"] [base=/data_bak]
[2020/05/19 16:26:19.219 +08:00] [INFO] [endpoint.rs:698] ["run backup task"] [task="BackupTask { start_ts: TimeStamp(0), end_ts: TimeStamp(416776572192686113), start_key: \"7480000000000011C45F720000000000000000\", end_key: \"7480000000000011C45F72FFFFFFFFFFFFFFFF00\", is_raw_kv: false, cf: \"default\" }"]
[2020/05/19 16:26:19.232 +08:00] [INFO] [endpoint.rs:686] ["backup finished"] [summary="Statistics { lock: CfStatistics { processed: 0, get: 0, next: 0, prev: 0, seek: 0, seek_for_prev: 0, over_seek_bound: 0, flow_stats: FlowStatistics { read_keys: 0, read_bytes: 0 } }, write: CfStatistics { processed: 0, get: 0, next: 0, prev: 0, seek: 0, seek_for_prev: 0, over_seek_bound: 0, flow_stats: FlowStatistics { read_keys: 0, read_bytes: 0 } }, data: CfStatistics { processed: 0, get: 0, next: 0, prev: 0, seek: 0, seek_for_prev: 0, over_seek_bound: 0, flow_stats: FlowStatistics { read_keys: 0, read_bytes: 0 } } }"] [take=12.155931ms]
[2020/05/19 16:26:19.232 +08:00] [INFO] [service.rs:79] ["backup send half closed"]

TIKV3

[2020/05/19 16:20:51.329 +08:00] [INFO] [local.rs:42] ["create local storage"] [base=/data_bak]
[2020/05/19 16:20:51.382 +08:00] [INFO] [endpoint.rs:698] ["run backup task"] [task="BackupTask { start_ts: TimeStamp(0), end_ts: TimeStamp(416776572192686113), start_key: \"7480000000000011C45F720000000000000000\", end_key: \"7480000000000011C45F72FFFFFFFFFFFFFFFF00\", is_raw_kv: false, cf: \"default\" }"]
[2020/05/19 16:20:51.384 +08:00] [INFO] [endpoint.rs:686] ["backup finished"] [summary="Statistics { lock: CfStatistics { processed: 0, get: 0, next: 0, prev: 0, seek: 0, seek_for_prev: 0, over_seek_bound: 0, flow_stats: FlowStatistics { read_keys: 0, read_bytes: 0 } }, write: CfStatistics { processed: 0, get: 0, next: 0, prev: 0, seek: 0, seek_for_prev: 0, over_seek_bound: 0, flow_stats: FlowStatistics { read_keys: 0, read_bytes: 0 } }, data: CfStatistics { processed: 0, get: 0, next: 0, prev: 0, seek: 0, seek_for_prev: 0, over_seek_bound: 0, flow_stats: FlowStatistics { read_keys: 0, read_bytes: 0 } } }"] [take=2.553062ms]
[2020/05/19 16:20:51.384 +08:00] [INFO] [service.rs:79] ["backup send half closed"]

感觉是是我哪里配置有问题导致的,比如3.0升级上来的4.0结果导致BR里面直接报错了。目前我在找代码里面哪里有问题,不过第一次看BR代码需要点时间

从日志看是 TiKV 写文件出错了,只有一个 TiKV 报错,另两个是正常的。请问 TiKV1 写的目录是否挂载了 nfs,和 TiKV2/TiKV3 节点有没有不同

可以测试一下更换其他目录,备份一个小表是否正常

备份小表,库都试过了,也是不行。这台KV上部署了BR,跟这个有关系么

三个节点的文件系统和挂载方式都相同么,部署 br 是指在这个节点运行的 br 么?

操作都相同,部署BR是在这个TiKV节点上运行的。

--log-level debug 可以尝试开启 debug 模式,看一下 br 输出的详细日志。

目前提供的信息没办法定位问题,也无法模拟复现。

1赞

这是新集群上的BR导出的完整信息,可以确定我都是同样的操作挂载了网盘。但是依然报错,只是不确定是哪个参数出问题了

[2020/05/20 09:59:24.584 +08:00] [INFO] [version.go:28] ["Welcome to Backup & Restore (BR)"]
[2020/05/20 09:59:24.584 +08:00] [INFO] [version.go:29] [BR] [release-version=v4.0.0-rc]
[2020/05/20 09:59:24.584 +08:00] [INFO] [version.go:30] [BR] [git-hash=194a4ff33ff151bb44795ff1ddff5f54d091b05f]
[2020/05/20 09:59:24.584 +08:00] [INFO] [version.go:31] [BR] [git-branch=heads/refs/tags/v4.0.0-rc]
[2020/05/20 09:59:24.584 +08:00] [INFO] [version.go:32] [BR] [go-version=go1.13]
[2020/05/20 09:59:24.584 +08:00] [INFO] [version.go:33] [BR] [utc-build-time="2020-04-08 07:44:44"]
[2020/05/20 09:59:24.584 +08:00] [INFO] [version.go:34] [BR] [race-enabled=false]
[2020/05/20 09:59:24.584 +08:00] [INFO] [version.go:55] [arguments] [backupts=] [ca=] [cert=] [checksum=true] [concurrency=0] [gcs.credentials-file=] [gcs.endpoint=] [gcs.predefined-acl=] [gcs.storage-class=] [help=false] [key=] [lastbackupts=0] [log-file=back.log] [log-level=debug] [pd="[10.204.9.131:2379]"] [ratelimit=0] [ratelimit-unit=1048576] [s3.acl=] [s3.endpoint=] [s3.provider=] [s3.region=] [s3.sse=] [s3.storage-class=] [send-credentials-to-tikv=true] [slow-log-file=] [status-addr=] [storage=local:///data31bak/back6] [timeago=0s]
[2020/05/20 09:59:24.585 +08:00] [INFO] [client.go:135] ["[pd] create pd client with endpoints"] [pd-address="[10.204.9.131:2379]"]
[2020/05/20 09:59:24.588 +08:00] [INFO] [base_client.go:226] ["[pd] update member urls"] [old-urls="[http://10.204.9.131:2379]"] [new-urls="[http://10.204.9.131:2379,http://10.204.9.132:2379,http://10.204.9.133:2379]"]
[2020/05/20 09:59:24.588 +08:00] [INFO] [base_client.go:242] ["[pd] switch leader"] [new-leader=http://10.204.9.131:2379] [old-leader=]
[2020/05/20 09:59:24.588 +08:00] [INFO] [base_client.go:92] ["[pd] init cluster id"] [cluster-id=6820705896630752929]
[2020/05/20 09:59:24.590 +08:00] [INFO] [client.go:135] ["[pd] create pd client with endpoints"] [pd-address="[10.204.9.131:2379]"]
[2020/05/20 09:59:24.592 +08:00] [INFO] [base_client.go:226] ["[pd] update member urls"] [old-urls="[http://10.204.9.131:2379]"] [new-urls="[http://10.204.9.131:2379,http://10.204.9.132:2379,http://10.204.9.133:2379]"]
[2020/05/20 09:59:24.592 +08:00] [INFO] [base_client.go:242] ["[pd] switch leader"] [new-leader=http://10.204.9.131:2379] [old-leader=]
[2020/05/20 09:59:24.592 +08:00] [INFO] [base_client.go:92] ["[pd] init cluster id"] [cluster-id=6820705896630752929]
[2020/05/20 09:59:24.592 +08:00] [INFO] [conn.go:199] ["new mgr"] [pdAddrs=10.204.9.131:2379]
[2020/05/20 09:59:24.592 +08:00] [INFO] [tidb.go:72] ["new domain"] [store=tikv-6820705896630752929] ["ddl lease"=1s] ["stats lease"=-1ns]
[2020/05/20 09:59:24.599 +08:00] [INFO] [ddl.go:321] ["[ddl] start DDL"] [ID=60ee863d-baa8-4b38-8153-625a1cb3b999] [runWorker=false]
[2020/05/20 09:59:24.612 +08:00] [DEBUG] [scan.go:157] ["txn getData"] [nextStartKey=6d4442730000000000fa0000000000000068] [nextEndKey=6d4442730000000000fa0000000000000069] [reverse=false] [txnStartTS=416793222074597378]
[2020/05/20 09:59:24.613 +08:00] [DEBUG] [scan.go:157] ["txn getData"] [nextStartKey=6d44423a3300000000fb0000000000000068] [nextEndKey=6d44423a3300000000fb0000000000000069] [reverse=false] [txnStartTS=416793222074597378]
[2020/05/20 09:59:24.613 +08:00] [DEBUG] [scan.go:157] ["txn getData"] [nextStartKey=6d44423a3100000000fb0000000000000068] [nextEndKey=6d44423a3100000000fb0000000000000069] [reverse=false] [txnStartTS=416793222074597378]
[2020/05/20 09:59:24.613 +08:00] [DEBUG] [scan.go:157] ["txn getData"] [nextStartKey=6d44423a3435000000fc0000000000000068] [nextEndKey=6d44423a3435000000fc0000000000000069] [reverse=false] [txnStartTS=416793222074597378]
[2020/05/20 09:59:24.613 +08:00] [DEBUG] [scan.go:157] ["txn getData"] [nextStartKey=6d44423a3538380000fd0000000000000068] [nextEndKey=6d44423a3538380000fd0000000000000069] [reverse=false] [txnStartTS=416793222074597378]
[2020/05/20 09:59:24.648 +08:00] [INFO] [domain.go:144] ["full load InfoSchema success"] [usedSchemaVersion=0] [neededSchemaVersion=133] ["start time"=35.693968ms]
[2020/05/20 09:59:24.649 +08:00] [INFO] [domain.go:368] ["full load and reset schema validator"]
[2020/05/20 09:59:24.649 +08:00] [DEBUG] [schema_validator.go:139] ["update schema validator"] [oldVer=0] [currVer=133] [changedTableIDs="[]"]
[2020/05/20 09:59:24.649 +08:00] [INFO] [client.go:68] ["new backup client"]
[2020/05/20 09:59:24.692 +08:00] [INFO] [client.go:112] ["backup encode timestamp"] [BackupTS=416793222100811777]
[2020/05/20 09:59:24.692 +08:00] [DEBUG] [scan.go:157] ["txn getData"] [nextStartKey=6d4442730000000000fa0000000000000068] [nextEndKey=6d4442730000000000fa0000000000000069] [reverse=false] [txnStartTS=416793222100811777]
[2020/05/20 09:59:24.693 +08:00] [DEBUG] [scan.go:157] ["txn getData"] [nextStartKey=6d44423a3435000000fc0000000000000068] [nextEndKey=6d44423a3435000000fc0000000000000069] [reverse=false] [txnStartTS=416793222100811777]
[2020/05/20 09:59:24.693 +08:00] [DEBUG] [scan.go:157] ["txn getData"] [nextStartKey=6d44423a3100000000fb0000000000000068] [nextEndKey=6d44423a3100000000fb0000000000000069] [reverse=false] [txnStartTS=416793222100811777]
[2020/05/20 09:59:24.693 +08:00] [DEBUG] [scan.go:157] ["txn getData"] [nextStartKey=6d44423a3538380000fd0000000000000068] [nextEndKey=6d44423a3538380000fd0000000000000069] [reverse=false] [txnStartTS=416793222100811777]
[2020/05/20 09:59:24.693 +08:00] [DEBUG] [scan.go:157] ["txn getData"] [nextStartKey=6d44423a3300000000fb0000000000000068] [nextEndKey=6d44423a3300000000fb0000000000000069] [reverse=false] [txnStartTS=416793222100811777]
[2020/05/20 09:59:24.722 +08:00] [INFO] [domain.go:144] ["full load InfoSchema success"] [usedSchemaVersion=0] [neededSchemaVersion=133] ["start time"=29.391086ms]
[2020/05/20 09:59:24.724 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=account_mch] [AutoIncID=60003]
[2020/05/20 09:59:24.725 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=cash_personal] [AutoIncID=1]
[2020/05/20 09:59:24.725 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=wallet_other] [AutoIncID=1]
[2020/05/20 09:59:24.726 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=audit_message] [AutoIncID=1]
[2020/05/20 09:59:24.726 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=history_pay_bill_bak20200508] [AutoIncID=1]
[2020/05/20 09:59:24.727 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=account_enterprise_route] [AutoIncID=120003]
[2020/05/20 09:59:24.728 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=statics_bussinetype] [AutoIncID=1]
[2020/05/20 09:59:24.728 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=account_payscore_sign2] [AutoIncID=66718]
[2020/05/20 09:59:24.729 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=statics_daybills] [AutoIncID=1]
[2020/05/20 09:59:24.729 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=pay_password] [AutoIncID=210006]
[2020/05/20 09:59:24.730 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=audit_bill] [AutoIncID=1]
[2020/05/20 09:59:24.730 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=history_pay_withdraw] [AutoIncID=1]
[2020/05/20 09:59:24.731 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=history_pay_bill] [AutoIncID=1]
[2020/05/20 09:59:24.732 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=history_pay_purchase_bak20200508] [AutoIncID=1]
[2020/05/20 09:59:24.732 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=history_pay_sys_bill] [AutoIncID=1]
[2020/05/20 09:59:24.733 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=account_enterprise] [AutoIncID=30005]
[2020/05/20 09:59:24.733 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=transation_log] [AutoIncID=1]
[2020/05/20 09:59:24.734 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=pay_bill] [AutoIncID=1]
[2020/05/20 09:59:24.734 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=cash_zmcert_log] [AutoIncID=1]
[2020/05/20 09:59:24.735 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=history_pay_purchase] [AutoIncID=1]
[2020/05/20 09:59:24.736 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=history_pay_purchase_extend_bak20190312] [AutoIncID=1]
[2020/05/20 09:59:24.736 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=history_pay_transfer] [AutoIncID=1]
[2020/05/20 09:59:24.737 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=account_control_log] [AutoIncID=440952961062130994]
[2020/05/20 09:59:24.738 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=account_netpay_sign] [AutoIncID=1]
[2020/05/20 09:59:24.738 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=account_personal] [AutoIncID=1]
[2020/05/20 09:59:24.739 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=pay_sys_bill] [AutoIncID=344730964]
[2020/05/20 09:59:24.739 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=history_pay_bill_bak20190312] [AutoIncID=1]
[2020/05/20 09:59:24.740 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=transation_log_copy1] [AutoIncID=1]
[2020/05/20 09:59:24.740 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=pay_withdraw] [AutoIncID=1]
[2020/05/20 09:59:24.741 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=wallet_other_refund] [AutoIncID=1]
[2020/05/20 09:59:24.742 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=pay_purchase] [AutoIncID=1]
[2020/05/20 09:59:24.742 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=history_transation_log] [AutoIncID=1]
[2020/05/20 09:59:24.743 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=purchase_statics] [AutoIncID=210007]
[2020/05/20 09:59:24.743 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=pay_purchase_extend] [AutoIncID=1]
[2020/05/20 09:59:24.744 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=pay_msgid_extend] [AutoIncID=1]
[2020/05/20 09:59:24.744 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=history_pay_purchase_bak20190312] [AutoIncID=1]
[2020/05/20 09:59:24.745 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=history_pay_refund] [AutoIncID=1]
[2020/05/20 09:59:24.745 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=pay_sys_bill_20190729] [AutoIncID=1]
[2020/05/20 09:59:24.746 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=pay_sys_bill_tmp] [AutoIncID=1]
[2020/05/20 09:59:24.747 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=account_order_control] [AutoIncID=1]
[2020/05/20 09:59:24.747 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=account_control] [AutoIncID=1]
[2020/05/20 09:59:24.748 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=pay_sys_bill_job] [AutoIncID=1]
[2020/05/20 09:59:24.748 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=history_cash_zmcert_log] [AutoIncID=1]
[2020/05/20 09:59:24.749 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=history_wallet_other] [AutoIncID=1]
[2020/05/20 09:59:24.749 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=pay_transfer] [AutoIncID=1]
[2020/05/20 09:59:24.750 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=rep_deposit_bill] [AutoIncID=412369006]
[2020/05/20 09:59:24.751 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=history_pay_purchase_extend] [AutoIncID=8277]
[2020/05/20 09:59:24.751 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=account_payscore_sign] [AutoIncID=636627]
[2020/05/20 09:59:24.752 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=pay_refund] [AutoIncID=1]
[2020/05/20 09:59:24.752 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=pay_password_log] [AutoIncID=210006]
[2020/05/20 09:59:24.753 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=pay] [table=history_wallet_other_refund] [AutoIncID=1]
[2020/05/20 09:59:24.754 +08:00] [INFO] [client.go:222] ["change table AutoIncID"] [db=tidb_binlog] [table=checkpoint] [AutoIncID=1]
[2020/05/20 09:59:25.119 +08:00] [INFO] [client.go:402] ["backup started"] [StartKey="dIAAAAAAAAB3X3IAAAAAAAAAAA=="] [EndKey="dIAAAAAAAAB3X3L//////////wA="] [RateLimit=0] [Concurrency=4]
[2020/05/20 09:59:25.120 +08:00] [INFO] [push.go:29] ["new backup client"]
[2020/05/20 09:59:25.120 +08:00] [INFO] [client.go:721] ["try backup"] ["backup request"="{\"cluster_id\":6820705896630752929,\"start_key\":\"dIAAAAAAAAB3X3IAAAAAAAAAAA==\",\"end_key\":\"dIAAAAAAAAB3X3L//////////wA=\",\"end_version\":416793222100811777,\"concurrency\":4,\"storage_backend\":{\"Backend\":{\"Local\":{\"path\":\"/data31bak/back6\"}}}}"]
[2020/05/20 09:59:25.121 +08:00] [INFO] [client.go:721] ["try backup"] ["backup request"="{\"cluster_id\":6820705896630752929,\"start_key\":\"dIAAAAAAAAB3X3IAAAAAAAAAAA==\",\"end_key\":\"dIAAAAAAAAB3X3L//////////wA=\",\"end_version\":416793222100811777,\"concurrency\":4,\"storage_backend\":{\"Backend\":{\"Local\":{\"path\":\"/data31bak/back6\"}}}}"]
[2020/05/20 09:59:25.121 +08:00] [INFO] [client.go:721] ["try backup"] ["backup request"="{\"cluster_id\":6820705896630752929,\"start_key\":\"dIAAAAAAAAB3X3IAAAAAAAAAAA==\",\"end_key\":\"dIAAAAAAAAB3X3L//////////wA=\",\"end_version\":416793222100811777,\"concurrency\":4,\"storage_backend\":{\"Backend\":{\"Local\":{\"path\":\"/data31bak/back6\"}}}}"]
[2020/05/20 09:59:25.149 +08:00] [INFO] [client.go:733] ["backup streaming finish"] [StoreID=4]
[2020/05/20 09:59:25.150 +08:00] [INFO] [client.go:733] ["backup streaming finish"] [StoreID=1]
[2020/05/20 09:59:25.178 +08:00] [INFO] [client.go:740] ["range backuped"] [StartKey="dIAAAAAAAAB3X3IAAAAAAAAAAA=="] [EndKey="dIAAAAAAAAB3X3L//////////wA="]
[2020/05/20 09:59:25.178 +08:00] [ERROR] [push.go:111] ["backup occur unknown error"] [error="Io(Os { code: 22, kind: InvalidInput, message: \"Invalid argument\" })"] [stack="github.com/pingcap/log.Error\n\t/go/pkg/mod/github.com/pingcap/log@v0.0.0-20200117041106-d28c14d3b1cd/global.go:42\ngithub.com/pingcap/br/pkg/backup.(*pushDown).pushBackup\n\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/backup/push.go:111\ngithub.com/pingcap/br/pkg/backup.(*Client).BackupRange\n\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/backup/client.go:424\ngithub.com/pingcap/br/pkg/backup.(*Client).BackupRanges.func2\n\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/backup/client.go:337"]
[2020/05/20 09:59:25.179 +08:00] [INFO] [client.go:394] ["backup range finished"] [take=59.349722ms]
[2020/05/20 09:59:25.179 +08:00] [INFO] [client.go:329] ["Backup Ranges"] [take=59.458016ms]
[2020/05/20 09:59:25.180 +08:00] [INFO] [ddl.go:407] ["[ddl] DDL closed"] [ID=60ee863d-baa8-4b38-8153-625a1cb3b999] ["take time"=1.571476ms]
[2020/05/20 09:59:25.180 +08:00] [INFO] [ddl.go:301] ["[ddl] stop DDL"] [ID=60ee863d-baa8-4b38-8153-625a1cb3b999]
[2020/05/20 09:59:25.183 +08:00] [INFO] [domain.go:607] ["domain closed"] ["take time"=4.066417ms]
[2020/05/20 09:59:25.183 +08:00] [INFO] [collector.go:180] ["Full backup Failed summary : total backup ranges: 1, total success: 0, total failed: 1"] ["backup total regions"=43612] [unitName="range start:7480000000000000775f720000000000000000 end:7480000000000000775f72ffffffffffffffff00"] [error="msg:\"Io(Os { code: 22, kind: InvalidInput, message: \\\"Invalid argument\\\" })\" "] [errorVerbose="msg:\"Io(Os { code: 22, kind: InvalidInput, message: \\\"Invalid argument\\\" })\" \ngithub.com/pingcap/br/pkg/backup.(*pushDown).pushBackup\n\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/backup/push.go:113\ngithub.com/pingcap/br/pkg/backup.(*Client).BackupRange\n\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/backup/client.go:424\ngithub.com/pingcap/br/pkg/backup.(*Client).BackupRanges.func2\n\t/home/jenkins/agent/workspace/build_br_multi_branch_v4.0.0-rc/go/src/github.com/pingcap/br/pkg/backup/client.go:337\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"]

你好,

看报错,主要还是写入 NFS 出错,请检查下盘的挂载方式

其次根据文档中的前置条件排查下。

1赞

确实是这个问题,我的网盘使用samba搭的

感谢回复,有问题环境开新帖继续讨论哦~