tidb v4.0.2在k8s v1.18.3上安装,在apply tidb-monitor.yaml时失败

k8s version v1.18.3
pingcap/tidb-operator v1.1.3
pingcap/tidb-monitor-initializer v4.0.2
pingcap/tidb-monitor-reloader v1.0.1
pingcap/tidb-backup-manager v1.1.3
pingcap/ticdc v4.0.2
pingcap/tidb-binlog v4.0.2
pingcap/tikv v4.0.2
pingcap/tidb v4.0.2
pingcap/pd v4.0.2
pingcap/tiflash v4.0.2
pingcap/advanced-statefulset v0.3.3
grafana/grafana 6.0.1
quay.io/external_storage/local-volume-provisioner v2.3.4
busybox 1.26.2
tnir/mysqlclient latest
prom/prometheus v2.18.1
bitnami/kubectl latest
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler v1.18.3
k8s-harbor.neu.tsd.com/library/etcd 3.4.3-0

在应用 kubectl -n tidb-cluster apply -f tidb-monitor.yaml 时
发现 pod tidb-controller-manager-7f56ff5985-bjbjv CrashLoopBackOff 且 tidb-monitor pod未建立
使用 kubectl -n tidb-admin logs -f tidb-controller-manager-7f56ff5985-bjbjv 命令查看,错误信息如下
E0807 01:16:46.119889 1 runtime.go:78] Observed a panic: &errors.errorString{s:“cannot parse ‘’: quantities must match the regular expression ‘^([±]?[0-9.]+)([eEinumkKMGTP][-+]?[0-9])$’”} (cannot parse ‘’: quantities must match the regular expression ‘^([±]?[0-9.]+)([eEinumkKMGTP][-+]?[0-9])$’)
goroutine 822 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x1a32160, 0xc000e17b00)
k8s.io/apimachinery@v0.0.0/pkg/util/runtime/runtime.go:74 +0xa3
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
k8s.io/apimachinery@v0.0.0/pkg/util/runtime/runtime.go:48 +0x82
panic(0x1a32160, 0xc000e17b00)
runtime/panic.go:679 +0x1b2
k8s.io/apimachinery/pkg/api/resource.MustParse(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
k8s.io/apimachinery@v0.0.0/pkg/api/resource/quantity.go:127 +0x1c8
github.com/pingcap/tidb-operator/pkg/monitor/monitor.getMonitorPVC(0xc000bb2380, 0xc000e17ae0)
github.com/pingcap/tidb-operator@/pkg/monitor/monitor/util.go:800 +0x261
github.com/pingcap/tidb-operator/pkg/monitor/monitor.(*MonitorManager).syncTidbMonitorPVC(0xc000d7e000, 0xc000bb2380, 0xc000e17ae0, 0x0, 0x0)
github.com/pingcap/tidb-operator@/pkg/monitor/monitor/monitor_manager.go:191 +0x40
github.com/pingcap/tidb-operator/pkg/monitor/monitor.(*MonitorManager).SyncMonitor(0xc000d7e000, 0xc000bb2380, 0xc0006b08a0, 0xc0002a4e40)
github.com/pingcap/tidb-operator@/pkg/monitor/monitor/monitor_manager.go:136 +0xe7f
github.com/pingcap/tidb-operator/pkg/controller/tidbmonitor.(*defaultTidbMonitorControl).reconcileTidbMonitor(…)
github.com/pingcap/tidb-operator@/pkg/controller/tidbmonitor/tidb_monitor_control.go:51
github.com/pingcap/tidb-operator/pkg/controller/tidbmonitor.(*defaultTidbMonitorControl).ReconcileTidbMonitor(0xc000aff440, 0xc000bb2380, 0x0, 0x0)
github.com/pingcap/tidb-operator@/pkg/controller/tidbmonitor/tidb_monitor_control.go:43 +0x42
github.com/pingcap/tidb-operator/pkg/controller/tidbmonitor.(Controller).sync(0xc000cf4400, 0xc0002a4e40, 0x19, 0x0, 0x0)
github.com/pingcap/tidb-operator@/pkg/controller/tidbmonitor/tidb_monitor_controller.go:148 +0x20a
github.com/pingcap/tidb-operator/pkg/controller/tidbmonitor.(Controller).processNextWorkItem(0xc000cf4400, 0x187ef00)
github.com/pingcap/tidb-operator@/pkg/controller/tidbmonitor/tidb_monitor_controller.go:116 +0x100
github.com/pingcap/tidb-operator/pkg/controller/tidbmonitor.(Controller).worker(0xc000cf4400)
github.com/pingcap/tidb-operator@/pkg/controller/tidbmonitor/tidb_monitor_controller.go:103 +0x2b
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc000e6e0f0)
k8s.io/apimachinery@v0.0.0/pkg/util/wait/wait.go:152 +0x5e
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000e6e0f0, 0x3b9aca00, 0x0, 0x1, 0xc0008225a0)
k8s.io/apimachinery@v0.0.0/pkg/util/wait/wait.go:153 +0xf8
k8s.io/apimachinery/pkg/util/wait.Until(0xc000e6e0f0, 0x3b9aca00, 0xc0008225a0)
k8s.io/apimachinery@v0.0.0/pkg/util/wait/wait.go:88 +0x4d
created by github.com/pingcap/tidb-operator/pkg/controller/tidbmonitor.(Controller).Run
github.com/pingcap/tidb-operator@/pkg/controller/tidbmonitor/tidb_monitor_controller.go:96 +0x1d9
panic: cannot parse ‘’: quantities must match the regular expression '^([±]?[0-9.]+)([eEinumkKMGTP]
[-+]?[0-9]
)$’ [recovered]
panic: cannot parse ‘’: quantities must match the regular expression '^([±]?[0-9.]+)([eEinumkKMGTP]
[-+]?[0-9]
)$’

goroutine 822 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
k8s.io/apimachinery@v0.0.0/pkg/util/runtime/runtime.go:55 +0x105
panic(0x1a32160, 0xc000e17b00)
runtime/panic.go:679 +0x1b2
k8s.io/apimachinery/pkg/api/resource.MustParse(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
k8s.io/apimachinery@v0.0.0/pkg/api/resource/quantity.go:127 +0x1c8
github.com/pingcap/tidb-operator/pkg/monitor/monitor.getMonitorPVC(0xc000bb2380, 0xc000e17ae0)
github.com/pingcap/tidb-operator@/pkg/monitor/monitor/util.go:800 +0x261
github.com/pingcap/tidb-operator/pkg/monitor/monitor.(*MonitorManager).syncTidbMonitorPVC(0xc000d7e000, 0xc000bb2380, 0xc000e17ae0, 0x0, 0x0)
github.com/pingcap/tidb-operator@/pkg/monitor/monitor/monitor_manager.go:191 +0x40
github.com/pingcap/tidb-operator/pkg/monitor/monitor.(*MonitorManager).SyncMonitor(0xc000d7e000, 0xc000bb2380, 0xc0006b08a0, 0xc0002a4e40)
github.com/pingcap/tidb-operator@/pkg/monitor/monitor/monitor_manager.go:136 +0xe7f
github.com/pingcap/tidb-operator/pkg/controller/tidbmonitor.(*defaultTidbMonitorControl).reconcileTidbMonitor(…)
github.com/pingcap/tidb-operator@/pkg/controller/tidbmonitor/tidb_monitor_control.go:51
github.com/pingcap/tidb-operator/pkg/controller/tidbmonitor.(*defaultTidbMonitorControl).ReconcileTidbMonitor(0xc000aff440, 0xc000bb2380, 0x0, 0x0)
github.com/pingcap/tidb-operator@/pkg/controller/tidbmonitor/tidb_monitor_control.go:43 +0x42
github.com/pingcap/tidb-operator/pkg/controller/tidbmonitor.(*Controller).sync(0xc000cf4400, 0xc0002a4e40, 0x19, 0x0, 0x0)
github.com/pingcap/tidb-operator@/pkg/controller/tidbmonitor/tidb_monitor_controller.go:148 +0x20a
github.com/pingcap/tidb-operator/pkg/controller/tidbmonitor.(*Controller).processNextWorkItem(0xc000cf4400, 0x187ef00)
github.com/pingcap/tidb-operator@/pkg/controller/tidbmonitor/tidb_monitor_controller.go:116 +0x100
github.com/pingcap/tidb-operator/pkg/controller/tidbmonitor.(*Controller).worker(0xc000cf4400)
github.com/pingcap/tidb-operator@/pkg/controller/tidbmonitor/tidb_monitor_controller.go:103 +0x2b
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc000e6e0f0)
k8s.io/apimachinery@v0.0.0/pkg/util/wait/wait.go:152 +0x5e
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000e6e0f0, 0x3b9aca00, 0x0, 0x1, 0xc0008225a0)
k8s.io/apimachinery@v0.0.0/pkg/util/wait/wait.go:153 +0xf8
k8s.io/apimachinery/pkg/util/wait.Until(0xc000e6e0f0, 0x3b9aca00, 0xc0008225a0)
k8s.io/apimachinery@v0.0.0/pkg/util/wait/wait.go:88 +0x4d
created by github.com/pingcap/tidb-operator/pkg/controller/tidbmonitor.(*Controller).Run
github.com/pingcap/tidb-operator@/pkg/controller/tidbmonitor/tidb_monitor_controller.go:96 +0x1d9
查看字面意思是为tidb-monitor pod申请 pv资源时出错,tidb-monitor.yaml 内容如下:
apiVersion: pingcap.com/v1alpha1
kind: TidbMonitor
metadata:
name: tidb-cluster
spec:
imagePullPolicy: IfNotPresent
persistent: true
storageClassName: local-storage
clusters:
- name: tidb-cluster
prometheus:
baseImage: prom/prometheus
version: v2.18.1
service:
type: NodePort
portName: http-prometheus
grafana:
baseImage: grafana/grafana
version: 6.0.1
username: admin
password: admin
service:
type: NodePort
portName: http-grafana
initializer:
baseImage: pingcap/tidb-monitor-initializer
version: v4.0.2
resources: {}
reloader:
baseImage: pingcap/tidb-monitor-reloader
version: v1.0.1
service:
type: NodePort
portName: tcp-reloader
resources: {}
使用 kubectl get pv -A |grep local-storage
local-pv-1a21926c 343Gi RWO Delete Available local-storage 16h
local-pv-26886a2e 219Gi RWO Delete Available local-storage 61m
local-pv-27b18074 18Gi RWO Delete Available local-storage 20m
local-pv-56e1e861 17Gi RWO Delete Available local-storage 20m
local-pv-64f8b1d3 343Gi RWO Delete Available local-storage 16h
local-pv-82666f13 18Gi RWO Delete Available local-storage 20m
local-pv-8886866a 219Gi RWO Delete Available local-storage 56m
local-pv-aa2b1ae7 18Gi RWO Delete Available local-storage 20m
local-pv-c5d3f40f 18Gi RWO Delete Available local-storage 20m
local-pv-e6463a67 219Gi RWO Delete Available local-storage 58m
local-pv-e796ead7 18Gi RWO Delete Available local-storage 20m
local-pv-f4682ca3 343Gi RWO Delete Available local-storage 16h

卸载 tidb-monitor.yaml ,删除pod tidb-controller-manager-7f56ff5985-bjbjv ,重load后一切正常,k8s上的tidb可用,只是没有监控

  1. 请问当前的问题是要查看报错 Observed a panic: &errors.errorString{s:“cannot parse ‘’: quantities must match the regular expression ‘^([±]?[0-9.]+)([eEinumkKMGTP] [-+]?[0-9] )$’” 的原因吗?

@truezxd spec.storage 没有配

在tidb-monitor.yaml 文件里有以下参数
storageClassName: local-storage

想请厂家给出如何诊断问题原因的方法,我想按照诊断模式处理,但是没搞明白
https://docs.pingcap.com/zh/tidb-in-kubernetes/stable/tips#诊断模式

在 为待诊断的 Pod 添加 Annotation ,然后 ```
watch kubectl get pod ${pod_name} -n ${namespace}

然后啥信息也没看到

image

老大,你真牛B ,搞定了

:+1:

此话题已在最后回复的 1 分钟后被自动关闭。不再允许新回复。