tidb-controller-manager-CrashLoopBackOff

Hacker_CYAz1WiB · 2021 年11 月 16 日 14:54

为提高效率，请提供以下信息，问题描述清晰能够更快得到解决：
【 TiDB 使用环境】tidb-operatorv1.1.12，k8s1.21+
【概述】tidb-controller-manager突然一直重启，CrashLoopBackOff
【现象】导致没法创建新的tikv集群
【业务影响】
【TiDB 版本】
【附件】
tidb-controller-manager日志如下

引用
I1116 14:45:12.017640 1 version.go:38] Welcome to TiDB Operator.
I1116 14:45:12.017674 1 version.go:39] TiDB Operator Version: version.Info{GitVersion:“v1.1.12”, GitCommit:“267b0d44f8c66f41a3a02685d7100c55d4697715”, GitTreeState:“clean”, BuildDate:“2021-06-24T11:05:14Z”, GoVersion:“go1.13.8”, Compiler:“gc”, Platform:“linux/amd64”}
I1116 14:45:12.017742 1 main.go:72] FLAG: --V=“false”
I1116 14:45:12.017759 1 main.go:72] FLAG: --add_dir_header=“false”
I1116 14:45:12.017767 1 main.go:72] FLAG: --alsologtostderr=“false”
I1116 14:45:12.017773 1 main.go:72] FLAG: --auto-failover=“true”
I1116 14:45:12.017780 1 main.go:72] FLAG: --cluster-scoped=“true”
I1116 14:45:12.017786 1 main.go:72] FLAG: --features=“AdvancedStatefulSet=false,AutoScaling=false,StableScheduling=true”
I1116 14:45:12.017808 1 main.go:72] FLAG: --leader-lease-duration=“15s”
I1116 14:45:12.017817 1 main.go:72] FLAG: --leader-renew-deadline=“10s”
I1116 14:45:12.017824 1 main.go:72] FLAG: --leader-retry-period=“2s”
I1116 14:45:12.017830 1 main.go:72] FLAG: --log_backtrace_at=“:0”
I1116 14:45:12.017840 1 main.go:72] FLAG: --log_dir=“”
I1116 14:45:12.017848 1 main.go:72] FLAG: --log_file=“”
I1116 14:45:12.017854 1 main.go:72] FLAG: --log_file_max_size=“1800”
I1116 14:45:12.017862 1 main.go:72] FLAG: --logtostderr=“true”
I1116 14:45:12.017869 1 main.go:72] FLAG: --pd-failover-period=“5m0s”
I1116 14:45:12.017876 1 main.go:72] FLAG: --pod-webhook-enabled=“false”
I1116 14:45:12.017904 1 main.go:72] FLAG: --resync-duration=“30s”
I1116 14:45:12.017916 1 main.go:72] FLAG: --selector=“”
I1116 14:45:12.017923 1 main.go:72] FLAG: --skip_headers=“false”
I1116 14:45:12.017929 1 main.go:72] FLAG: --skip_log_headers=“false”
I1116 14:45:12.017936 1 main.go:72] FLAG: --stderrthreshold=“2”
I1116 14:45:12.017945 1 main.go:72] FLAG: --test-mode=“false”
I1116 14:45:12.017951 1 main.go:72] FLAG: --tidb-backup-manager-image=“pingcap/tidb-backup-manager:v1.1.12”
I1116 14:45:12.017958 1 main.go:72] FLAG: --tidb-discovery-image=“pingcap/tidb-operator:v1.1.12”
I1116 14:45:12.017965 1 main.go:72] FLAG: --tidb-failover-period=“5m0s”
I1116 14:45:12.017972 1 main.go:72] FLAG: --tiflash-failover-period=“5m0s”
I1116 14:45:12.017978 1 main.go:72] FLAG: --tikv-failover-period=“5m0s”
I1116 14:45:12.017985 1 main.go:72] FLAG: --v=“2”
I1116 14:45:12.017992 1 main.go:72] FLAG: --version=“false”
I1116 14:45:12.017999 1 main.go:72] FLAG: --vmodule=“”
I1116 14:45:12.018007 1 main.go:72] FLAG: --workers=“5”
I1116 14:45:12.018049 1 main.go:89] HELM_RELEASE environment variable not set
I1116 14:45:12.051292 1 leaderelection.go:241] attempting to acquire leader lease tidb-admin/tidb-controller-manager…
I1116 14:45:12.059153 1 leaderelection.go:251] successfully acquired lease tidb-admin/tidb-controller-manager
I1116 14:45:12.062166 1 upgrader.go:106] Upgrader: APIGroup apps.pingcap.com is not registered, skip checking Advanced Statfulset
I1116 14:45:14.067007 1 main.go:180] cache of informer factories sync successfully
I1116 14:45:14.067082 1 tidb_monitor_controller.go:65] Starting tidbmonitor controller
I1116 14:45:14.067089 1 backup_controller.go:72] Starting backup controller
I1116 14:45:14.067124 1 restore_controller.go:71] Starting restore controller
I1116 14:45:14.067140 1 backup_schedule_controller.go:70] Starting backup schedule controller
I1116 14:45:14.067134 1 tidb_cluster_controller.go:98] Starting tidbcluster controller
I1116 14:45:14.067352 1 tidb_initializer_controller.go:69] Starting tidbinitializer controller
I1116 14:45:14.330599 1 tidbcluster_control.go:66] TidbCluster: [tidb-cluster-2/test-2] updated successfully
I1116 14:45:14.379955 1 tidbcluster_control.go:66] TidbCluster: [tidb-cluster/basicc] updated successfully
I1116 14:45:15.852628 1 tidbcluster_control.go:66] TidbCluster: [tidb-cluster-2/test-2] updated successfully
E1116 14:45:16.829171 1 runtime.go:78] Observed a panic: “invalid memory address or nil pointer dereference” (runtime error: invalid memory address or nil pointer dereference)
goroutine 734 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x1b139a0, 0x30a1510)
k8s.io/apimachinery@v0.0.0/pkg/util/runtime/runtime.go:74 +0xa3
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
k8s.io/apimachinery@v0.0.0/pkg/util/runtime/runtime.go:48 +0x82
panic(0x1b139a0, 0x30a1510)
runtime/panic.go:679 +0x1b2
github.com/pingcap/tidb-operator/pkg/apis/pingcap/v1alpha1.(*TidbCluster).IsTiDBBinlogEnabled(...)
github.com/pingcap/tidb-operator@/pkg/apis/pingcap/v1alpha1/tidbcluster.go:580
github.com/pingcap/tidb-operator/pkg/monitor/monitor.getMonitorInitContainer(0xc000172e00, 0xc002a20000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, …)
github.com/pingcap/tidb-operator@/pkg/monitor/monitor/util.go:300 +0x1de
github.com/pingcap/tidb-operator/pkg/monitor/monitor.getMonitorDeployment(0xc0028a8c60, 0xc00295f680, 0xc00295fcc0, 0xc000172e00, 0xc002a20000, 0x0, 0x17fde55, 0xc000b4a540)
github.com/pingcap/tidb-operator@/pkg/monitor/monitor/util.go:207 +0x7b
github.com/pingcap/tidb-operator/pkg/monitor/monitor.(*MonitorManager).syncTidbMonitorDeployment(0xc000570ff0, 0xc002a20000, 0xc000172e00, 0x0, 0x0)
github.com/pingcap/tidb-operator@/pkg/monitor/monitor/monitor_manager.go:197 +0x4fd
github.com/pingcap/tidb-operator/pkg/monitor/monitor.(*MonitorManager).SyncMonitor(0xc000570ff0, 0xc000172e00, 0xc000b4a500, 0xc000e24cc0)
github.com/pingcap/tidb-operator@/pkg/monitor/monitor/monitor_manager.go:117 +0x825
github.com/pingcap/tidb-operator/pkg/controller/tidbmonitor.(*defaultTidbMonitorControl).reconcileTidbMonitor(...)
github.com/pingcap/tidb-operator@/pkg/controller/tidbmonitor/tidb_monitor_control.go:46
github.com/pingcap/tidb-operator/pkg/controller/tidbmonitor.(*defaultTidbMonitorControl).ReconcileTidbMonitor(0xc00013a630, 0xc000172e00, 0x0, 0x0)
github.com/pingcap/tidb-operator@/pkg/controller/tidbmonitor/tidb_monitor_control.go:39 +0x41
github.com/pingcap/tidb-operator/pkg/controller/tidbmonitor.(*Controller).sync(0xc0005710e0, 0xc000e24cc0, 0x15, 0x0, 0x0)
github.com/pingcap/tidb-operator@/pkg/controller/tidbmonitor/tidb_monitor_controller.go:120 +0x213
github.com/pingcap/tidb-operator/pkg/controller/tidbmonitor.(*Controller).processNextWorkItem(0xc0005710e0, 0x0)
github.com/pingcap/tidb-operator@/pkg/controller/tidbmonitor/tidb_monitor_controller.go:88 +0x100
github.com/pingcap/tidb-operator/pkg/controller/tidbmonitor.(*Controller).worker(0xc0005710e0)
github.com/pingcap/tidb-operator@/pkg/controller/tidbmonitor/tidb_monitor_controller.go:76 +0x2b
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc00295a4d0)
k8s.io/apimachinery@v0.0.0/pkg/util/wait/wait.go:152 +0x5e
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00295a4d0, 0x3b9aca00, 0x0, 0x45ef01, 0xc000b741e0)
k8s.io/apimachinery@v0.0.0/pkg/util/wait/wait.go:153 +0xf8
k8s.io/apimachinery/pkg/util/wait.Until(0xc00295a4d0, 0x3b9aca00, 0xc000b741e0)
k8s.io/apimachinery@v0.0.0/pkg/util/wait/wait.go:88 +0x4d
created by github.com/pingcap/tidb-operator/pkg/controller/tidbmonitor.(*Controller).Run
github.com/pingcap/tidb-operator@/pkg/controller/tidbmonitor/tidb_monitor_controller.go:69 +0x1d9
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x160 pc=0x1904a3e]

相关日志和监控

TiUP Cluster Display 信息
TiUP Cluster Edit Config 信息
TiDB- Overview 监控

对应模块日志（包含问题前后1小时日志）

qizheng · 2021 年11 月 17 日 02:29

可以升级到 v1.1.13 或 v1.1.14 版本
https://github.com/pingcap/tidb-operator/issues/4051

system · 2022 年10 月 31 日 19:16

此话题已在最后回复的 1 分钟后被自动关闭。不再允许新回复。