BR還原From GCS(Error: Cannot read gcs://xxxx/*.sst: authorization failed: HTTP error status: 404 Not Found: [BR:KV:ErrKVDownloadFailed]download sst failed)

jimmyjan0824 · 2021 年2 月 1 日 07:20

【TiDB 版本】: v4.0.10
【BR 版本】: v4.0.10
【问题描述】:

GCS備份皆可成功完成
GCS還原完整備份時，出現以下錯誤:

Full restore <----------------------------------------------------------------------------------------------------------------------------------------------------------------> 100.00%
Error: Cannot read gcs://dev_tidb_backup_bucketdev-tidb-backup/20210201/full//7_1298_324_40a8022925f58fe0d69ae3487b5253cba26b6a14014a733d0ff80517beaf6770_1612148082299_write.sst into /tidb-data/tikv-20161/import/.temp/97035c3a-3254-40e1-a092-6f368aa02a0b_7857_5_1198_write.sst: authorization failed: HTTP error status: 404 Not Found: [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read gcs://dev_tidb_backup_bucketdev-tidb-backup/20210201/full//7_1298_324_40a8022925f58fe0d69ae3487b5253cba26b6a14014a733d0ff80517beaf6770_1612148082299_write.sst into /tidb-data/tikv-20161/import/.temp/768b6e79-2480-445f-9cf6-b5635098ad95_7857_5_1198_write.sst: authorization failed: HTTP error status: 404 Not Found: [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read gcs://dev_tidb_backup_bucketdev-tidb-backup/20210201/full//7_1298_324_40a8022925f58fe0d69ae3487b5253cba26b6a14014a733d0ff80517beaf6770_1612148082299_write.sst into /tidb-data/tikv-20161/import/.temp/85450c82-0fc5-4d25-8b03-358dc9ebc15c_7857_5_1198_write.sst: authorization failed: HTTP error status: 404 Not Found: [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read gcs://dev_tidb_backup_bucketdev-tidb-backup/20210201/full//7_1298_324_40a8022925f58fe0d69ae3487b5253cba26b6a14014a733d0ff80517beaf6770_1612148082299_write.sst into /tidb-data/tikv-20161/import/.temp/e79b2bd1-94a5-4094-9b19-ff72b00ed37c_7857_5_1198_write.sst: authorization failed: HTTP error status: 404 Not Found: [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read gcs://dev_tidb_backup_bucketdev-tidb-backup/20210201/full//7_1298_324_40a8022925f58fe0d69ae3487b5253cba26b6a14014a733d0ff80517beaf6770_1612148082299_write.sst into /tidb-data/tikv-20161/import/.temp/e0614ff6-fc09-4e62-97b8-8a27b940a8b9_7857_5_1198_write.sst: authorization failed: HTTP error status: 404 Not Found: [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read gcs://dev_tidb_backup_bucketdev-tidb-backup/20210201/full//7_1298_324_40a8022925f58fe0d69ae3487b5253cba26b6a14014a733d0ff80517beaf6770_1612148082299_write.sst into /tidb-data/tikv-20161/import/.temp/d49ab7c4-c3a5-4c82-838d-2cdc3f981fac_7857_5_1198_write.sst: authorization failed: HTTP error status: 404 Not Found: [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read gcs://dev_tidb_backup_bucketdev-tidb-backup/20210201/full//7_1298_324_40a8022925f58fe0d69ae3487b5253cba26b6a14014a733d0ff80517beaf6770_1612148082299_write.sst into /tidb-data/tikv-20161/import/.temp/9f9c9474-55dc-4115-85da-a89bfe7d7f63_7857_5_1198_write.sst: authorization failed: HTTP error status: 404 Not Found: [BR:KV:ErrKVDownloadFailed]download sst failed; Cannot read gcs://dev_tidb_backup_bucketdev-tidb-backup/20210201/full//7_1298_324_40a8022925f58fe0d69ae3487b5253cba26b6a14014a733d0ff80517beaf6770_1612148082299_write.sst into /tidb-data/tikv-20161/import/.temp/6bdf88cd-339d-4f51-8ba7-3cc865d410db_7857_5_1198_write.sst: authorization failed: HTTP error status: 404 Not Found: [BR:KV:ErrKVDownloadFailed]download sst failed

1. 已透過以下指令成功備份檔案到GCS上

03:00 - 完整備份

br backup full -u x.x.x.x:2379 --gcs.credentials-file /home/tidb/devops-296503-15712d14f89c.json --send-credentials-to-tikv=true -s gcs://dev_tidb_backup_bucket/dev-tidb-backup/20210201/full

09:00 - 遞增備份1

br backup full
–pd x.x.x.x:2379
–gcs.credentials-file /home/tidb/devops-296503-15712d14f89c.json
-s gcs://dev_tidb_backup_bucket/dev-tidb-backup/20210201/incr/20210201_0900_incr
–lastbackupts 422614943718178818

15:00 - 遞增備份2

br backup full
–pd x.x.x.x:2379
–gcs.credentials-file /home/tidb/devops-296503-15712d14f89c.json
-s gcs://dev_tidb_backup_bucket/dev-tidb-backup/20210201/incr/20210201_1500_incr
–lastbackupts 422614965227094018

21:00 - 遞增備份3

br backup full
–pd x.x.x.x:2379
–gcs.credentials-file /home/tidb/devops-296503-15712d14f89c.json
-s gcs://dev_tidb_backup_bucket/dev-tidb-backup/20210201/incr/20210201_2100_incr
–lastbackupts 422614980392124417

2. 還原完整備份
br restore full
–pd “192.168.33.92:2379”
–gcs.credentials-file /home/tidb/DevOps-263cc3791089.json
–send-credentials-to-tikv=true
–storage “gcs://dev_tidb_backup_bucket/dev-tidb-backup/20210201/full/”
–log-file restorefull.log

完整log檔案:
restorefull.log (2.7 MB)

GCP上帳號權限如下：

想請教顧問，這個錯誤訊息是指GCP我們少了什權限？還是TiDB部份有哪邊我們漏掉地方？

jimmyjan0824 · 2021 年2 月 2 日 01:38

顧問們好,

GCS IAM Roles:

https://cloud.google.com/storage/docs/access-control/iam-roles

我們GCP已配置了storage.admin角色權限，仍會有上述回報的錯誤

再請顧問們不吝指導，感謝！

已配置storage.admin角色權限

還原時log檔案:
restorefull.log (916.8 KB)

小王同学 · 2021 年2 月 2 日 06:32

1.麻烦检查下，恢复的过程中，是否有读取文件的权限；
2.确认下 GCS 的参数配置
https://docs.pingcap.com/zh/tidb/stable/backup-and-restore-storages#gcs-参数

jimmyjan0824 · 2021 年2 月 2 日 07:14

顧問好,

GCS的storage.admin角色有包含操作所有buckets及其物件的權限（包含read)

image716×215 31.4 KB
GCS參數配置
已提供完整還原時的指令，能否請顧問提示還需加哪個參數？
br restore full
–pd “192.168.33.92:2379”
–gcs.credentials-file /home/tidb/DevOps-263cc3791089.json
–send-credentials-to-tikv=true
–storage “gcs://dev_tidb_backup_bucket/dev-tidb-backup/20210201/full/”
–log-file restorefull.log
透過Google官方gutil工具嘗試拉取GCS 上.sst檔案是成功的 => 權限應該是沒問題
gsutil cp gs://dev_tidb_backup_bucket/dev-tidb-backup/20210201/full/1_1010_252_d3b33d5893186673b2b9a1557de630c85fb220e0a47da0a067325af80d1c6809_1612148073689_write.sst .

luancheng_pingcap · 2021 年2 月 2 日 09:32

authorization failed
能否看下在报错的 tikv 节点上是否可以 ping 通 /home/tidb/DevOps-263cc3791089.json 里面的 token_uri ？

jimmyjan0824 · 2021 年2 月 2 日 10:37

顧問好,

token_uri =“https://oauth2.googleapis.com/token”

ping是有reply的

luancheng_pingcap · 2021 年2 月 2 日 11:37

可否打开某一个 tikv 的 debug 日志。
然后执行恢复
收集在恢复报错前 tikv 日志上传下，我们分析下

luancheng_pingcap · 2021 年2 月 2 日 11:56

在此之前可否用最新的 br v4.0.10 https://docs.pingcap.com/zh/tidb/stable/download-ecosystem-tools#快速备份和恢复br 试试同样的命令，是否可以？@ jimmyjan0824

luancheng_pingcap · 2021 年2 月 2 日 12:12

注意到 BR 是用的 v4.0.8 恢复的

确认下是否是用的 BR v4.0.10 执行的备份？
如果是这样，那么使用 BR v4.0.10 执行恢复再看看，我们在 v4.0.10 上修复了一个 gcs 兼容性的问题。https://github.com/pingcap/br/pull/677

jimmyjan0824 · 2021 年2 月 3 日 04:52

顧問好,

將BR工具升級到v4.0.10後，即可正常還原備份檔(full+incr)，感謝你的協助！

来了老弟 · 2021 年2 月 3 日 08:02

system · 2022 年10 月 31 日 19:19

此话题已在最后回复的 1 分钟后被自动关闭。不再允许新回复。