BR還原From GCS(Error: Cannot read gcs://xxxx/*.sst: authorization failed: HTTP error status: 404 Not Found: [BR:KV:ErrKVDownloadFailed]download sst failed)

【TiDB 版本】: v4.0.10
【BR 版本】: v4.0.10
【问题描述】:

  • GCS備份皆可成功完成

  • GCS還原完整備份時,出現以下錯誤:

Full restore <----------------------------------------------------------------------------------------------------------------------------------------------------------------> 100.00%
Error: Cannot read gcs://dev_tidb_backup_bucketdev-tidb-backup/20210201/full//7_1298_324_40a8022925f58fe0d69ae3487b5253cba26b6a14014a733d0ff80517beaf6770_1612148082299_write.sst into /tidb-data/tikv-20161/import/.temp/97035c3a-3254-40e1-a092-6f368aa02a0b_7857_5_1198_write.sst: authorization failed: HTTP error status: 404 Not Found: [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read gcs://dev_tidb_backup_bucketdev-tidb-backup/20210201/full//7_1298_324_40a8022925f58fe0d69ae3487b5253cba26b6a14014a733d0ff80517beaf6770_1612148082299_write.sst into /tidb-data/tikv-20161/import/.temp/768b6e79-2480-445f-9cf6-b5635098ad95_7857_5_1198_write.sst: authorization failed: HTTP error status: 404 Not Found: [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read gcs://dev_tidb_backup_bucketdev-tidb-backup/20210201/full//7_1298_324_40a8022925f58fe0d69ae3487b5253cba26b6a14014a733d0ff80517beaf6770_1612148082299_write.sst into /tidb-data/tikv-20161/import/.temp/85450c82-0fc5-4d25-8b03-358dc9ebc15c_7857_5_1198_write.sst: authorization failed: HTTP error status: 404 Not Found: [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read gcs://dev_tidb_backup_bucketdev-tidb-backup/20210201/full//7_1298_324_40a8022925f58fe0d69ae3487b5253cba26b6a14014a733d0ff80517beaf6770_1612148082299_write.sst into /tidb-data/tikv-20161/import/.temp/e79b2bd1-94a5-4094-9b19-ff72b00ed37c_7857_5_1198_write.sst: authorization failed: HTTP error status: 404 Not Found: [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read gcs://dev_tidb_backup_bucketdev-tidb-backup/20210201/full//7_1298_324_40a8022925f58fe0d69ae3487b5253cba26b6a14014a733d0ff80517beaf6770_1612148082299_write.sst into /tidb-data/tikv-20161/import/.temp/e0614ff6-fc09-4e62-97b8-8a27b940a8b9_7857_5_1198_write.sst: authorization failed: HTTP error status: 404 Not Found: [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read gcs://dev_tidb_backup_bucketdev-tidb-backup/20210201/full//7_1298_324_40a8022925f58fe0d69ae3487b5253cba26b6a14014a733d0ff80517beaf6770_1612148082299_write.sst into /tidb-data/tikv-20161/import/.temp/d49ab7c4-c3a5-4c82-838d-2cdc3f981fac_7857_5_1198_write.sst: authorization failed: HTTP error status: 404 Not Found: [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read gcs://dev_tidb_backup_bucketdev-tidb-backup/20210201/full//7_1298_324_40a8022925f58fe0d69ae3487b5253cba26b6a14014a733d0ff80517beaf6770_1612148082299_write.sst into /tidb-data/tikv-20161/import/.temp/9f9c9474-55dc-4115-85da-a89bfe7d7f63_7857_5_1198_write.sst: authorization failed: HTTP error status: 404 Not Found: [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read gcs://dev_tidb_backup_bucketdev-tidb-backup/20210201/full//7_1298_324_40a8022925f58fe0d69ae3487b5253cba26b6a14014a733d0ff80517beaf6770_1612148082299_write.sst into /tidb-data/tikv-20161/import/.temp/6bdf88cd-339d-4f51-8ba7-3cc865d410db_7857_5_1198_write.sst: authorization failed: HTTP error status: 404 Not Found: [BR:KV:ErrKVDownloadFailed]download sst failed

1. 已透過以下指令成功備份檔案到GCS上

03:00 - 完整備份

br backup full -u x.x.x.x:2379 --gcs.credentials-file /home/tidb/devops-296503-15712d14f89c.json --send-credentials-to-tikv=true -s gcs://dev_tidb_backup_bucket/dev-tidb-backup/20210201/full

09:00 - 遞增備份1

br backup full
–pd x.x.x.x:2379
–gcs.credentials-file /home/tidb/devops-296503-15712d14f89c.json
-s gcs://dev_tidb_backup_bucket/dev-tidb-backup/20210201/incr/20210201_0900_incr
–lastbackupts 422614943718178818

15:00 - 遞增備份2

br backup full
–pd x.x.x.x:2379
–gcs.credentials-file /home/tidb/devops-296503-15712d14f89c.json
-s gcs://dev_tidb_backup_bucket/dev-tidb-backup/20210201/incr/20210201_1500_incr
–lastbackupts 422614965227094018

21:00 - 遞增備份3

br backup full
–pd x.x.x.x:2379
–gcs.credentials-file /home/tidb/devops-296503-15712d14f89c.json
-s gcs://dev_tidb_backup_bucket/dev-tidb-backup/20210201/incr/20210201_2100_incr
–lastbackupts 422614980392124417

2. 還原完整備份
br restore full
–pd “192.168.33.92:2379”
–gcs.credentials-file /home/tidb/DevOps-263cc3791089.json
–send-credentials-to-tikv=true
–storage “gcs://dev_tidb_backup_bucket/dev-tidb-backup/20210201/full/”
–log-file restorefull.log

完整log檔案:
restorefull.log (2.7 MB)

GCP上帳號權限如下:

想請教顧問,這個錯誤訊息是指GCP我們少了什權限?還是TiDB部份有哪邊我們漏掉地方?

顧問們好,

GCS IAM Roles:

https://cloud.google.com/storage/docs/access-control/iam-roles

我們GCP已配置了storage.admin角色權限,仍會有上述回報的錯誤

再請顧問們不吝指導,感謝!

已配置storage.admin角色權限

還原時log檔案:
restorefull.log (916.8 KB)

1.麻烦检查下,恢复的过程中,是否有读取文件的权限;
2.确认下 GCS 的参数配置
https://docs.pingcap.com/zh/tidb/stable/backup-and-restore-storages#gcs-参数

顧問好,

  1. GCS的storage.admin角色有包含操作所有buckets及其物件的權限(包含read)

  2. GCS參數配置
    已提供完整還原時的指令,能否請顧問提示還需加哪個參數?
    br restore full
    –pd “192.168.33.92:2379”
    –gcs.credentials-file /home/tidb/DevOps-263cc3791089.json
    –send-credentials-to-tikv=true
    –storage “gcs://dev_tidb_backup_bucket/dev-tidb-backup/20210201/full/”
    –log-file restorefull.log

  3. 透過Google官方gutil工具 嘗試拉取GCS 上.sst檔案 是成功的 => 權限應該是沒問題
    gsutil cp gs://dev_tidb_backup_bucket/dev-tidb-backup/20210201/full/1_1010_252_d3b33d5893186673b2b9a1557de630c85fb220e0a47da0a067325af80d1c6809_1612148073689_write.sst .

authorization failed
能否看下在报错的 tikv 节点上是否可以 ping 通 /home/tidb/DevOps-263cc3791089.json 里面的 token_uri ?

顧問好,

token_uri =“https://oauth2.googleapis.com/token

ping是有reply的

  1. 可否打开某一个 tikv 的 debug 日志。
  2. 然后执行恢复
  3. 收集在恢复报错前 tikv 日志上传下,我们分析下

在此之前可否用最新的 br v4.0.10 https://docs.pingcap.com/zh/tidb/stable/download-ecosystem-tools#快速备份和恢复br 试试同样的命令,是否可以?@ jimmyjan0824

注意到 BR 是用的 v4.0.8 恢复的

确认下是否是用的 BR v4.0.10 执行的备份?
如果是这样,那么使用 BR v4.0.10 执行恢复再看看,我们在 v4.0.10 上修复了一个 gcs 兼容性的问题。https://github.com/pingcap/br/pull/677

1赞

顧問好,

將BR工具升級到v4.0.10後,即可正常還原備份檔(full+incr),感謝你的協助!
image

1赞

:+1:

1赞