【TiDB 使用环境】
apiVersion: pingcap.com/v1alpha1
kind: TidbCluster
metadata:
name: basic
spec:
version: v8.5.0
timezone: Asia/Shanghai
pvReclaimPolicy: Retain
enableDynamicConfiguration: true
configUpdateStrategy: RollingUpdate
discovery: {}
helper:
image: m.daocloud.io/docker.io/library/alpine:3.16.0
pd:
...
【问题复现路径】做过哪些操作出现的问题
apiVersion: pingcap.com/v1alpha1
kind: BackupSchedule
metadata:
name: basic-backup-schedule-nfs
namespace: tidb-cluster
spec:
maxBackups: 3
pause: false
schedule: "*/30 * * * *"
backupTemplate:
cleanPolicy: Retain
resources:
requests:
cpu: "500m"
memory: "512Mi"
limits:
cpu: "1000m"
memory: "1Gi"
backupMode: snapshot
backupType: full
toolImage: m.daocloud.io/docker.io/pingcap/br:v8.5.0
br:
cluster: basic
clusterNamespace: tidb-cluster
logLevel: info
concurrency: 4
# rateLimit: 0
# checksum: true
local:
prefix: tidb-cluster/basic/full/scheduled
volume:
name: backup
persistentVolumeClaim:
claimName: tidb-cluster-backup-pvc
volumeMount:
name: backup
mountPath: /backup
【遇到的问题：问题现象及影响】
备份失败
【复制黏贴 ERROR 报错的日志】
root@node1:~/ben.wangz/temp# kubectl -n tidb-cluster logs -f backup-basic-backup-schedule-nfs-2025-05-28t08-39-00-jlbt2
Defaulted container "backup" out of: backup, br (init)
Create rclone.conf file.
/tidb-backup-manager backup --namespace=tidb-cluster --backupName=basic-backup-schedule-nfs-2025-05-28t08-39-00 --tikvVersion=v8.5.0 --mode=snapshot
E0528 16:45:15.215547 9 reflector.go:147] k8s.io/client-go@v0.28.14/tools/cache/reflector.go:229: Failed to watch *v1alpha1.Backup: unknown (get backups.pingcap.com)
I0528 16:45:15.266686 9 backup.go:78] start to process backup tidb-cluster/basic-backup-schedule-nfs-2025-05-28t08-39-00
I0528 16:45:15.273192 9 manager.go:109] start to process backup: {"kind":"Backup","apiVersion":"pingcap.com/v1alpha1","metadata":{"name":"basic-backup-schedule-nfs-2025-05-28t08-39-00","namespace":"tidb-cluster","uid":"aad30fd4-6dd3-4125-aff4-070d8361a8f4","resourceVersion":"52543406","generation":7,"creationTimestamp":"2025-05-28T08:39:09Z","labels":{"app.kubernetes.io/instance":"basic-backup-schedule-nfs","app.kubernetes.io/managed-by":"backup-schedule-operator","app.kubernetes.io/name":"backup-schedule","tidb.pingcap.com/backup-schedule":"basic-backup-schedule-nfs"},"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"pingcap.com/v1alpha1\",\"kind\":\"BackupSchedule\",\"metadata\":{\"annotations\":{},\"name\":\"basic-backup-schedule-nfs\",\"namespace\":\"tidb-cluster\"},\"spec\":{\"backupTemplate\":{\"backupMode\":\"snapshot\",\"backupType\":\"full\",\"br\":{\"cluster\":\"basic\",\"clusterNamespace\":\"tidb-cluster\",\"concurrency\":4,\"logLevel\":\"info\"},\"cleanPolicy\":\"Retain\",\"local\":{\"prefix\":\"tidb-cluster/basic/full/scheduled\",\"volume\":{\"name\":\"backup\",\"persistentVolumeClaim\":{\"claimName\":\"tidb-cluster-backup-pvc\"}},\"volumeMount\":{\"mountPath\":\"/backup\",\"name\":\"backup\"}},\"resources\":{\"limits\":{\"cpu\":\"1000m\",\"memory\":\"1Gi\"},\"requests\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"}},\"toolImage\":\"m.daocloud.io/docker.io/pingcap/br:v8.5.0\"},\"maxBackups\":3,\"pause\":false,\"schedule\":\"*/3 * * * *\"}}\n"},"ownerReferences":[{"apiVersion":"pingcap.com/v1alpha1","kind":"BackupSchedule","name":"basic-backup-schedule-nfs","uid":"e6ab694d-7a84-4ec0-8ef9-dbbaf8839c87","controller":true,"blockOwnerDeletion":true}],"managedFields":[{"manager":"tidb-backup-manager","operation":"Update","apiVersion":"pingcap.com/v1alpha1","time":"2025-05-28T08:39:12Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:backupPath":{}}}},{"manager":"tidb-controller-manager","operation":"Update","apiVersion":"pingcap.com/v1alpha1","time":"2025-05-28T08:45:09Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}},"f:labels":{".":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/name":{},"f:tidb.pingcap.com/backup-schedule":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"e6ab694d-7a84-4ec0-8ef9-dbbaf8839c87\"}":{}}},"f:spec":{".":{},"f:backoffRetryPolicy":{".":{},"f:maxRetryTimes":{},"f:minRetryDuration":{},"f:retryTimeout":{}},"f:backupMode":{},"f:backupType":{},"f:br":{".":{},"f:cluster":{},"f:clusterNamespace":{},"f:concurrency":{},"f:logLevel":{}},"f:calcSizeLevel":{},"f:cleanPolicy":{},"f:local":{".":{},"f:prefix":{},"f:volume":{".":{},"f:name":{},"f:persistentVolumeClaim":{".":{},"f:claimName":{}}},"f:volumeMount":{".":{},"f:mountPath":{},"f:name":{}}},"f:resources":{".":{},"f:limits":{".":{},"f:cpu":{},"f:memory":{}},"f:requests":{".":{},"f:cpu":{},"f:memory":{}}},"f:toolImage":{},"f:volumeBackupInitJobMaxActiveSeconds":{}},"f:status":{".":{},"f:backoffRetryStatus":{},"f:conditions":{},"f:phase":{},"f:timeCompleted":{},"f:timeStarted":{}}}}]},"spec":{"resources":{"limits":{"cpu":"1","memory":"1Gi"},"requests":{"cpu":"500m","memory":"512Mi"}},"backupType":"full","backupMode":"snapshot","local":{"volume":{"name":"backup","persistentVolumeClaim":{"claimName":"tidb-cluster-backup-pvc"}},"volumeMount":{"name":"backup","mountPath":"/backup"},"prefix":"tidb-cluster/basic/full/scheduled/basic-pd.tidb-cluster-2379-2025-05-28t08-39-00"},"br":{"cluster":"basic","clusterNamespace":"tidb-cluster","logLevel":"info","concurrency":4},"calcSizeLevel":"all","toolImage":"m.daocloud.io/docker.io/pingcap/br:v8.5.0","cleanPolicy":"Retain","backoffRetryPolicy":{"minRetryDuration":"300s","maxRetryTimes":2,"retryTimeout":"30m"},"volumeBackupInitJobMaxActiveSeconds":600},"status":{"backupPath":"local:///backup/tidb-cluster/basic/full/scheduled/basic-pd.tidb-cluster-2379-2025-05-28t08-39-00","timeStarted":null,"timeCompleted":null,"phase":"Scheduled","conditions":[{"type":"Scheduled","status":"True","lastTransitionTime":"2025-05-28T08:45:09Z"},{"type":"Prepare","status":"True","lastTransitionTime":"2025-05-28T08:39:12Z"},{"type":"RetryFailed","status":"True","lastTransitionTime":"2025-05-28T08:45:09Z","reason":"RetryFailedBackup","message":"reason Job backup-basic-backup-schedule-nfs-2025-05-28t08-39-00 has failed, original reason BackoffLimitExceeded"},{"type":"Restart","status":"True","lastTransitionTime":"2025-05-28T08:45:09Z"}],"backoffRetryStatus":[{"retryNum":1,"detectFailedAt":"2025-05-28T08:39:39Z","expectedRetryAt":"2025-05-28T08:44:39Z","realRetryAt":"2025-05-28T08:45:09Z","retryReason":"Job backup-basic-backup-schedule-nfs-2025-05-28t08-39-00 has failed","originalReason":"BackoffLimitExceeded"}]}}
I0528 16:45:15.273273 9 manager.go:115] snapshot backup tidb-cluster/basic-backup-schedule-nfs-2025-05-28t08-39-00 was restarted, status is Scheduled
I0528 16:45:15.297373 9 backup_status_updater.go:128] Backup: [tidb-cluster/basic-backup-schedule-nfs-2025-05-28t08-39-00] updated successfully
E0528 16:45:15.314313 9 backup_status_updater.go:131] Failed to update backup [tidb-cluster/basic-backup-schedule-nfs-2025-05-28t08-39-00], error: Operation cannot be fulfilled on backups.pingcap.com "basic-backup-schedule-nfs-2025-05-28t08-39-00": the object has been modified; please apply your changes to the latest version and try again
E0528 16:45:15.344480 9 backup_status_updater.go:131] Failed to update backup [tidb-cluster/basic-backup-schedule-nfs-2025-05-28t08-39-00], error: Operation cannot be fulfilled on backups.pingcap.com "basic-backup-schedule-nfs-2025-05-28t08-39-00": the object has been modified; please apply your changes to the latest version and try again
E0528 16:45:15.370330 9 backup_status_updater.go:131] Failed to update backup [tidb-cluster/basic-backup-schedule-nfs-2025-05-28t08-39-00], error: Operation cannot be fulfilled on backups.pingcap.com "basic-backup-schedule-nfs-2025-05-28t08-39-00": the object has been modified; please apply your changes to the latest version and try again
E0528 16:45:15.396101 9 backup_status_updater.go:131] Failed to update backup [tidb-cluster/basic-backup-schedule-nfs-2025-05-28t08-39-00], error: Operation cannot be fulfilled on backups.pingcap.com "basic-backup-schedule-nfs-2025-05-28t08-39-00": the object has been modified; please apply your changes to the latest version and try again
E0528 16:45:15.420317 9 backup_status_updater.go:131] Failed to update backup [tidb-cluster/basic-backup-schedule-nfs-2025-05-28t08-39-00], error: Operation cannot be fulfilled on backups.pingcap.com "basic-backup-schedule-nfs-2025-05-28t08-39-00": the object has been modified; please apply your changes to the latest version and try again
Error from server (Conflict): Operation cannot be fulfilled on backups.pingcap.com "basic-backup-schedule-nfs-2025-05-28t08-39-00": the object has been modified; please apply your changes to the latest version and try again
Sleeping for 10 seconds before exit...
【其他附件：截图/日志/监控】