TiDB Server 几分钟就重启一次

【 TiDB 使用环境】生产环境 /测试/ Poc
【 TiDB 版本】v7.1.1
【复现路径】正常运行的环境,突然出现所有 tidb server oom重启,最后重新搭建了一套集群将业务迁移至新集群之后,现在来排查问题,表现出的问题是tidb server10分钟准时重启一次。
【遇到的问题:问题现象及影响】
【资源配置】Dashboard显示异常

【附件:截图/日志/监控】

[2023/08/21 16:01:59.977 +08:00] [INFO] [domain.go:2668] ["refreshServerIDTTL succeed"] [serverID=142875] ["lease id"=1c688a165d8e1dc4]
[2023/08/21 16:02:20.623 +08:00] [INFO] [cpuprofile.go:113] ["parallel cpu profiler started"]
[2023/08/21 16:02:20.625 +08:00] [INFO] [printer.go:48] ["Welcome to TiDB."] ["Release Version"=v7.1.1] [Edition=Community] ["Git Commit Hash"=cf441574864be63938524e7dfcf7cc659edc3dd8] ["Git Branch"=heads/refs/tags/v7.1.1] ["UTC Build Time"="2023-07-19 10:16:40"] [GoVersion=go1.20.6] ["Race Enabled"=false] ["Check Table Before Drop"=false] ["TiKV Min Version"=6.2.0-alpha]
[2023/08/21 16:02:20.626 +08:00] [INFO] [printer.go:53] ["loaded config"] [config="{\"host\":\"0.0.0.0\",\"advertise-address\":\"10.223.225.84\",\"port\":4000,\"cors\":\"\",\"store\":\"tikv\",\"path\":\"10.223.225.165:2379,10.223.225.166:2379,10.223.225.167:2379\",\"socket\":\"/tmp/tidb-4000.sock\",\"lease\":\"45s\",\"split-table\":true,\"token-limit\":1000,\"temp-dir\":\"/tmp/tidb\",\"tmp-storage-path\":\"/tmp/1001_tidb/MC4wLjAuMDo0MDAwLzAuMC4wLjA6MTAwODA=/tmp-storage\",\"tmp-storage-quota\":-1,\"server-version\":\"\",\"version-comment\":\"\",\"tidb-edition\":\"\",\"tidb-release-version\":\"\",\"keyspace-name\":\"\",\"log\":{\"level\":\"info\",\"format\":\"text\",\"disable-timestamp\":null,\"enable-timestamp\":null,\"disable-error-stack\":null,\"enable-error-stack\":null,\"file\":{\"filename\":\"/data/tidb/tidb-deploy/tidb-4000/log/tidb.log\",\"max-size\":300,\"max-days\":0,\"max-backups\":0},\"slow-query-file\":\"/data/tidb/tidb-deploy/tidb-4000/log/tidb_slow_query.log\",\"expensive-threshold\":10000,\"query-log-max-len\":4096,\"enable-slow-log\":true,\"slow-threshold\":300,\"record-plan-in-slow-log\":1,\"timeout\":0},\"instance\":{\"tidb_general_log\":false,\"tidb_pprof_sql_cpu\":false,\"ddl_slow_threshold\":300,\"tidb_expensive_query_time_threshold\":60,\"tidb_stmt_summary_enable_persistent\":false,\"tidb_stmt_summary_filename\":\"tidb-statements.log\",\"tidb_stmt_summary_file_max_days\":3,\"tidb_stmt_summary_file_max_size\":64,\"tidb_stmt_summary_file_max_backups\":0,\"tidb_enable_slow_log\":true,\"tidb_slow_log_threshold\":300,\"tidb_record_plan_in_slow_log\":1,\"tidb_check_mb4_value_in_utf8\":true,\"tidb_force_priority\":\"NO_PRIORITY\",\"tidb_memory_usage_alarm_ratio\":0.8,\"tidb_enable_collect_execution_info\":true,\"plugin_dir\":\"/data/deploy/plugin\",\"plugin_load\":\"\",\"max_connections\":0,\"tidb_enable_ddl\":true,\"tidb_rc_read_check_ts\":false},\"security\":{\"skip-grant-table\":false,\"ssl-ca\":\"\",\"ssl-cert\":\"\",\"ssl-key\":\"\",\"cluster-ssl-ca\":\"\",\"cluster-ssl-cert\":\"\",\"cluster-ssl-key\":\"\",\"cluster-verify-cn\":null,\"session-token-signing-cert\":\"\",\"session-token-signing-key\":\"\",\"spilled-file-encryption-method\":\"plaintext\",\"enable-sem\":false,\"auto-tls\":false,\"tls-version\":\"\",\"rsa-key-size\":4096,\"secure-bootstrap\":false,\"auth-token-jwks\":\"\",\"auth-token-refresh-interval\":\"1h0m0s\",\"disconnect-on-expired-password\":true},\"status\":{\"status-host\":\"0.0.0.0\",\"metrics-addr\":\"\",\"status-port\":10080,\"metrics-interval\":15,\"report-status\":true,\"record-db-qps\":false,\"record-db-label\":false,\"grpc-keepalive-time\":10,\"grpc-keepalive-timeout\":3,\"grpc-concurrent-streams\":1024,\"grpc-initial-window-size\":2097152,\"grpc-max-send-msg-size\":2147483647},\"performance\":{\"max-procs\":0,\"max-memory\":0,\"server-memory-quota\":0,\"stats-lease\":\"3s\",\"stmt-count-limit\":5000,\"pseudo-estimate-ratio\":0.8,\"bind-info-lease\":\"3s\",\"txn-entry-size-limit\":6291456,\"txn-total-size-limit\":104857600,\"tcp-keep-alive\":true,\"tcp-no-delay\":true,\"cross-join\":true,\"distinct-agg-push-down\":false,\"projection-push-down\":false,\"max-txn-ttl\":3600000,\"index-usage-sync-lease\":\"0s\",\"plan-replayer-gc-lease\":\"10m\",\"gogc\":100,\"enforce-mpp\":false,\"stats-load-concurrency\":5,\"stats-load-queue-size\":1000,\"analyze-partition-concurrency-quota\":16,\"plan-replayer-dump-worker-concurrency\":1,\"enable-stats-cache-mem-quota\":false,\"committer-concurrency\":128,\"run-auto-analyze\":true,\"force-priority\":\"NO_PRIORITY\",\"memory-usage-alarm-ratio\":0.8,\"enable-load-fmsketch\":false,\"lite-init-stats\":false,\"force-init-stats\":false},\"prepared-plan-cache\":{\"enabled\":true,\"capacity\":100,\"memory-guard-ratio\":0.1},\"opentracing\":{\"enable\":false,\"rpc-metrics\":false,\"sampler\":{\"type\":\"const\",\"param\":1,\"sampling-server-url\":\"\",\"max-operations\":0,\"sampling-refresh-interval\":0},\"reporter\":{\"queue-size\":0,\"buffer-flush-interval\":0,\"log-spans\":false,\"local-agent-host-port\":\"\"}},\"proxy-protocol\":{\"networks\":\"\",\"header-timeout\":5,\"fallbackable\":false},\"pd-client\":{\"pd-server-timeout\":3},\"tikv-client\":{\"grpc-connection-count\":4,\"grpc-keepalive-time\":10,\"grpc-keepalive-timeout\":3,\"grpc-compression-type\":\"none\",\"commit-timeout\":\"41s\",\"async-commit\":{\"keys-limit\":256,\"total-key-size-limit\":4096,\"safe-window\":2000000000,\"allowed-clock-drift\":500000000},\"max-batch-size\":128,\"overload-threshold\":200,\"max-batch-wait-time\":0,\"batch-wait-size\":8,\"enable-chunk-rpc\":true,\"region-cache-ttl\":600,\"store-limit\":0,\"store-liveness-timeout\":\"1s\",\"copr-cache\":{\"capacity-mb\":1000},\"ttl-refreshed-txn-size\":33554432,\"resolve-lock-lite-threshold\":16},\"binlog\":{\"enable\":false,\"ignore-error\":false,\"write-timeout\":\"15s\",\"binlog-socket\":\"\",\"strategy\":\"range\"},\"compatible-kill-query\":false,\"pessimistic-txn\":{\"max-retry-count\":256,\"deadlock-history-capacity\":10,\"deadlock-history-collect-retryable\":false,\"pessimistic-auto-commit\":false,\"constraint-check-in-place-pessimistic\":true},\"max-index-length\":3072,\"index-limit\":64,\"table-column-count-limit\":1017,\"graceful-wait-before-shutdown\":0,\"alter-primary-key\":false,\"treat-old-version-utf8-as-utf8mb4\":true,\"enable-table-lock\":false,\"delay-clean-table-lock\":0,\"split-region-max-num\":1000,\"top-sql\":{\"receiver-address\":\"\"},\"repair-mode\":false,\"repair-table-list\":[],\"isolation-read\":{\"engines\":[\"tikv\",\"tiflash\",\"tidb\"]},\"new_collations_enabled_on_first_bootstrap\":true,\"experimental\":{\"allow-expression-index\":false},\"skip-register-to-dashboard\":false,\"enable-telemetry\":false,\"labels\":{},\"enable-global-index\":false,\"deprecate-integer-display-length\":false,\"enable-enum-length-limit\":true,\"stores-refresh-interval\":60,\"enable-tcp4-only\":false,\"enable-forwarding\":false,\"max-ballast-object-size\":0,\"ballast-object-size\":0,\"transaction-summary\":{\"transaction-summary-capacity\":500,\"transaction-id-digest-min-duration\":2147483647},\"enable-global-kill\":true,\"initialize-sql-file\":\"\",\"enable-batch-dml\":false,\"mem-quota-query\":1073741824,\"oom-action\":\"cancel\",\"oom-use-tmp-storage\":true,\"check-mb4-value-in-utf8\":true,\"enable-collect-execution-info\":true,\"plugin\":{\"dir\":\"/data/deploy/plugin\",\"load\":\"\"},\"max-server-connections\":0,\"run-ddl\":true,\"disaggregated-tiflash\":false,\"autoscaler-type\":\"aws\",\"autoscaler-addr\":\"tiflash-autoscale-lb.tiflash-autoscale.svc.cluster.local:8081\",\"is-tiflashcompute-fixed-pool\":false,\"autoscaler-cluster-id\":\"\",\"use-autoscaler\":false,\"tidb-max-reuse-chunk\":64,\"tidb-max-reuse-column\":256,\"tidb-enable-exit-check\":false}"]
[2023/08/21 16:02:20.626 +08:00] [INFO] [main.go:394] ["disable Prometheus push client"]
[2023/08/21 16:02:20.626 +08:00] [INFO] [store.go:76] ["new store"] [path=tikv://10.223.225.165:2379,10.223.225.166:2379,10.223.225.167:2379]
[2023/08/21 16:02:20.626 +08:00] [INFO] [client.go:311] ["[pd] create pd client with endpoints and keyspace"] [pd-address="[10.223.225.165:2379,10.223.225.166:2379,10.223.225.167:2379]"] [keyspace-id=0]
[2023/08/21 16:02:20.626 +08:00] [INFO] [systime_mon.go:26] ["start system time monitor"]
[2023/08/21 16:02:20.635 +08:00] [INFO] [pd_service_discovery.go:543] ["[pd] switch leader"] [new-leader=http://10.223.225.166:2379] [old-leader=]
[2023/08/21 16:02:20.635 +08:00] [INFO] [pd_service_discovery.go:175] ["[pd] init cluster id"] [cluster-id=7173638068559301829]
[2023/08/21 16:02:20.636 +08:00] [INFO] [client.go:386] ["[pd] changing service mode"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE]
[2023/08/21 16:02:20.636 +08:00] [INFO] [tso_client.go:230] ["[tso] switch dc tso allocator serving address"] [dc-location=global] [new-address=http://10.223.225.166:2379]
[2023/08/21 16:02:20.637 +08:00] [INFO] [tso_dispatcher.go:290] ["[tso] tso dispatcher created"] [dc-location=global]
[2023/08/21 16:02:20.637 +08:00] [INFO] [client.go:428] ["[pd] service mode changed"] [old-mode=PD_SVC_MODE] [new-mode=PD_SVC_MODE]
[2023/08/21 16:02:20.637 +08:00] [INFO] [tikv_driver.go:221] ["using API V1."]
[2023/08/21 16:02:20.638 +08:00] [INFO] [store.go:82] ["new store with retry success"]
[2023/08/21 16:02:20.653 +08:00] [INFO] [tidb.go:80] ["new domain"] [store=tikv-7173638068559301829] ["ddl lease"=45s] ["stats lease"=3s] ["index usage sync lease"=0s]
[2023/08/21 16:02:20.669 +08:00] [INFO] [domain.go:2650] [acquireServerID] [serverID=4112750] ["lease id"=24ef8a165d8d3262]
[2023/08/21 16:02:20.686 +08:00] [WARN] [info.go:304] ["init TiFlashReplicaManager"] ["pd addrs"="[10.223.225.165:2379,10.223.225.167:2379,10.223.225.166:2379]"]
[2023/08/21 16:02:20.882 +08:00] [INFO] [domain.go:272] ["full load InfoSchema success"] [currentSchemaVersion=0] [neededSchemaVersion=3140] ["start time"=181.931223ms]
[2023/08/21 16:02:20.885 +08:00] [INFO] [domain.go:587] ["full load and reset schema validator"]
[2023/08/21 16:02:20.886 +08:00] [INFO] [ddl.go:752] ["[ddl] start DDL"] [ID=b38db565-4618-4f1a-b37d-f5a96ed63997] [runWorker=true]
[2023/08/21 16:02:20.886 +08:00] [INFO] [ddl.go:715] ["[ddl] start delRangeManager OK"] ["is a emulator"=false]
[2023/08/21 16:02:20.889 +08:00] [INFO] [manager.go:178] ["start campaign owner"] [ownerInfo="[ddl] /tidb/ddl/fg/owner"]
[2023/08/21 16:02:20.892 +08:00] [INFO] [env.go:90] ["[ddl-ingest] the ingest sorted directory"] ["data path:"=/tmp/tidb/tmp_ddl-4000]
[2023/08/21 16:02:20.892 +08:00] [WARN] [backend_mgr.go:59] ["[ddl-ingest] ingest backfill may not be available"] [error="the available disk space(33598537728) in /tmp/tidb/tmp_ddl-4000 should be greater than @@tidb_ddl_disk_quota(107374182400)"]
[2023/08/21 16:02:20.892 +08:00] [INFO] [env.go:68] ["[ddl-ingest] init global ingest backend environment finished"] ["memory limitation"=2147483648] ["disk usage info"="disk usage: 1114112/33599651840, backend usage: 0"] ["max open file number"=1000000] ["lightning is initialized"=true]
[2023/08/21 16:02:20.892 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=loadSchemaInLoop]
[2023/08/21 16:02:20.892 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=mdlCheckLoop]
[2023/08/21 16:02:20.892 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=topNSlowQueryLoop]
[2023/08/21 16:02:20.892 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=infoSyncerKeeper]
[2023/08/21 16:02:20.892 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=globalConfigSyncerKeeper]
[2023/08/21 16:02:20.892 +08:00] [INFO] [job_table.go:328] ["[ddl] get global state and global state change"] [oldState=false] [currState=false]
[2023/08/21 16:02:20.892 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=topologySyncerKeeper]
[2023/08/21 16:02:20.892 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=closestReplicaReadCheckLoop]
[2023/08/21 16:02:20.892 +08:00] [INFO] [owner_daemon.go:70] ["begin advancer daemon"] [daemon-id=LogBackup::Advancer]
[2023/08/21 16:02:20.892 +08:00] [INFO] [manager.go:178] ["start campaign owner"] [ownerInfo="[log-backup] /tidb/br-stream/owner"]
[2023/08/21 16:02:20.898 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=logBackupAdvancer]
[2023/08/21 16:02:20.899 +08:00] [INFO] [owner_daemon.go:81] ["begin running daemon"] [id=bee7ead2-8f49-4af9-80bf-383cabffcbfe] [daemon-id=LogBackup::Advancer]
[2023/08/21 16:02:20.908 +08:00] [INFO] [manager.go:178] ["start campaign owner"] [ownerInfo="[bindinfo] /tidb/bindinfo/owner"]
[2023/08/21 16:02:20.910 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=globalBindHandleWorkerLoop]
[2023/08/21 16:02:20.911 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=handleEvolvePlanTasksLoop]
[2023/08/21 16:02:20.911 +08:00] [WARN] [sysvar_cache.go:50] ["sysvar cache is empty, triggering rebuild"]
[2023/08/21 16:02:20.918 +08:00] [INFO] [sysvar.go:2432] ["set resource control"] [enable=true]
[2023/08/21 16:02:21.070 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=loadPrivilegeInLoop]
[2023/08/21 16:02:21.074 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=LoadSysVarCacheLoop]
[2023/08/21 16:02:21.077 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=PlanReplayerTaskCollectHandle]
[2023/08/21 16:02:21.077 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=PlanReplayerTaskDumpHandle]
[2023/08/21 16:02:21.077 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=dumpFileGcChecker]
[2023/08/21 16:02:21.077 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=HistoricalStatsWorker]
[2023/08/21 16:02:21.077 +08:00] [INFO] [domain.go:1968] ["PlanReplayerTaskDumpHandle started"]
[2023/08/21 16:02:21.077 +08:00] [INFO] [plan_replayer.go:381] ["planReplayerTaskDumpWorker started."]
[2023/08/21 16:02:21.077 +08:00] [INFO] [domain.go:1946] ["PlanReplayerTaskCollectHandle started"]
[2023/08/21 16:02:21.077 +08:00] [INFO] [domain.go:2000] ["dumpFileGcChecker started"]
[2023/08/21 16:02:21.077 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=loadStatsWorker]
[2023/08/21 16:02:21.077 +08:00] [INFO] [manager.go:178] ["start campaign owner"] [ownerInfo="[stats] /tidb/stats/owner"]
[2023/08/21 16:02:21.077 +08:00] [INFO] [domain.go:2033] ["HistoricalStatsWorker started"]
[2023/08/21 16:02:21.080 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=updateStatsWorker]
[2023/08/21 16:02:21.080 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=autoAnalyzeWorker]
[2023/08/21 16:02:21.080 +08:00] [INFO] [domain.go:2303] ["updateStatsWorker started."]
[2023/08/21 16:02:21.080 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=gcAnalyzeHistory]
[2023/08/21 16:02:21.080 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=ttlJobManager]
[2023/08/21 16:02:21.080 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=loadSigningCertLoop]
[2023/08/21 16:02:21.080 +08:00] [INFO] [gc_worker.go:212] ["[gc worker] start"] [uuid=6285c770bcc0009]
[2023/08/21 16:02:21.081 +08:00] [INFO] [task_manager.go:215] ["scale ttl worker"] [ttl-worker=job-manager] [ttl-worker=task-manager] [originalCount=0] [newCount=4]
[2023/08/21 16:02:21.081 +08:00] [INFO] [task_manager.go:215] ["scale ttl worker"] [ttl-worker=job-manager] [ttl-worker=task-manager] [originalCount=0] [newCount=4]
[2023/08/21 16:02:21.085 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=distTaskFrameworkLoop]
[2023/08/21 16:02:21.086 +08:00] [INFO] [domain.go:1405] ["dist task scheduler started"]
[2023/08/21 16:02:21.086 +08:00] [WARN] [misc.go:458] ["Automatic TLS Certificate creation is disabled"] []
[2023/08/21 16:02:21.086 +08:00] [INFO] [server.go:271] ["server is running MySQL protocol"] [addr=0.0.0.0:4000]
[2023/08/21 16:02:21.086 +08:00] [INFO] [server.go:285] ["server is running MySQL protocol"] [socket=/tmp/tidb-4000.sock]
[2023/08/21 16:02:21.086 +08:00] [INFO] [http_status.go:90] ["for status and metrics report"] ["listening on addr"=0.0.0.0:10080]
[2023/08/21 16:02:21.087 +08:00] [INFO] [cpu.go:83] ["sql cpu collector started"]
[2023/08/21 16:02:21.089 +08:00] [INFO] [store.go:76] ["new store"] [path=tikv://10.223.225.165:2379,10.223.225.166:2379,10.223.225.167:2379]
[2023/08/21 16:02:21.089 +08:00] [INFO] [client.go:311] ["[pd] create pd client with endpoints and keyspace"] [pd-address="[10.223.225.165:2379,10.223.225.166:2379,10.223.225.167:2379]"] [keyspace-id=0]
[2023/08/21 16:02:21.095 +08:00] [INFO] [pd_service_discovery.go:543] ["[pd] switch leader"] [new-leader=http://10.223.225.166:2379] [old-leader=]
[2023/08/21 16:02:21.095 +08:00] [INFO] [pd_service_discovery.go:175] ["[pd] init cluster id"] [cluster-id=7173638068559301829]
[2023/08/21 16:02:21.095 +08:00] [INFO] [client.go:386] ["[pd] changing service mode"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE]
[2023/08/21 16:02:21.096 +08:00] [INFO] [tso_client.go:230] ["[tso] switch dc tso allocator serving address"] [dc-location=global] [new-address=http://10.223.225.166:2379]
[2023/08/21 16:02:21.096 +08:00] [INFO] [tso_dispatcher.go:290] ["[tso] tso dispatcher created"] [dc-location=global]
[2023/08/21 16:02:21.096 +08:00] [INFO] [client.go:428] ["[pd] service mode changed"] [old-mode=PD_SVC_MODE] [new-mode=PD_SVC_MODE]
[2023/08/21 16:02:21.096 +08:00] [INFO] [tso_client.go:134] ["closing tso client"]
[2023/08/21 16:02:21.096 +08:00] [INFO] [tso_dispatcher.go:162] ["exit tso requests cancel loop"]
[2023/08/21 16:02:21.096 +08:00] [WARN] [resource_manager_client.go:295] ["[resource_manager] get token stream error"] [error="rpc error: code = Canceled desc = context canceled"]
[2023/08/21 16:02:21.096 +08:00] [ERROR] [tso_dispatcher.go:514] ["[tso] update connection contexts failed"] [dc=global] [error="rpc error: code = Canceled desc = context canceled"]
[2023/08/21 16:02:21.096 +08:00] [INFO] [tso_dispatcher.go:311] ["[tso] exit tso dispatcher"] [dc-location=global]
[2023/08/21 16:02:21.096 +08:00] [INFO] [tso_dispatcher.go:214] ["exit tso dispatcher loop"]
[2023/08/21 16:02:21.096 +08:00] [INFO] [tso_client.go:139] ["close tso client"]
[2023/08/21 16:02:21.097 +08:00] [INFO] [tso_client.go:150] ["tso client is closed"]
[2023/08/21 16:02:21.097 +08:00] [INFO] [pd_service_discovery.go:250] ["[pd] close pd service discovery client"]
[2023/08/21 16:02:21.097 +08:00] [INFO] [store.go:82] ["new store with retry success"]
[2023/08/21 16:02:21.098 +08:00] [INFO] [manager.go:178] ["start campaign owner"] [ownerInfo="[autoid] tidb/autoid/leader"]
[2023/08/21 16:02:21.101 +08:00] [INFO] [http_status.go:492] ["register auto service at"] [addr=10.223.225.84:10080]
[2023/08/21 16:02:21.989 +08:00] [INFO] [domain.go:2208] ["init stats info time"] [lite=false] ["take time"=911.362361ms]
[2023/08/21 16:02:22.892 +08:00] [INFO] [info.go:1189] [SetTiFlashGroupConfig]
[2023/08/21 16:03:01.559 +08:00] [INFO] [manager.go:344] ["get owner"] ["owner info"="[ddl] /tidb/ddl/fg/owner ownerManager b38db565-4618-4f1a-b37d-f5a96ed63997"] ["owner key"=/tidb/ddl/fg/owner/24ef8a165d8d326b] [ownerID=b38db565-4618-4f1a-b37d-f5a96ed63997] [op=none]
[2023/08/21 16:03:01.559 +08:00] [INFO] [manager.go:344] ["get owner"] ["owner info"="[bindinfo] /tidb/bindinfo/owner ownerManager b38db565-4618-4f1a-b37d-f5a96ed63997"] ["owner key"=/tidb/bindinfo/owner/77568a165d6afaec] [ownerID=b38db565-4618-4f1a-b37d-f5a96ed63997] [op=none]
[2023/08/21 16:03:01.563 +08:00] [INFO] [manager.go:344] ["get owner"] ["owner info"="[autoid] tidb/autoid/leader ownerManager 10.223.225.84:10080"] ["owner key"=tidb/autoid/leader/1c688a165d8e20d5] [ownerID=10.223.225.84:10080] [op=none]
[2023/08/21 16:03:01.563 +08:00] [INFO] [manager.go:344] ["get owner"] ["owner info"="[stats] /tidb/stats/owner ownerManager b38db565-4618-4f1a-b37d-f5a96ed63997"] ["owner key"=/tidb/stats/owner/77568a165d6afaee] [ownerID=b38db565-4618-4f1a-b37d-f5a96ed63997] [op=none]
[2023/08/21 16:03:01.563 +08:00] [INFO] [manager.go:344] ["get owner"] ["owner info"="[log-backup] /tidb/br-stream/owner ownerManager bee7ead2-8f49-4af9-80bf-383cabffcbfe"] ["owner key"=/tidb/br-stream/owner/24ef8a165d8d326d] [ownerID=bee7ead2-8f49-4af9-80bf-383cabffcbfe] [op=none]
[2023/08/21 16:03:01.916 +08:00] [INFO] [job_table.go:328] ["[ddl] get global state and global state change"] [oldState=false] [currState=false]
[2023/08/21 16:03:01.917 +08:00] [INFO] [manager.go:344] ["get owner"] ["owner info"="[ddl] /tidb/ddl/fg/owner ownerManager b38db565-4618-4f1a-b37d-f5a96ed63997"] ["owner key"=/tidb/ddl/fg/owner/24ef8a165d8d326b] [ownerID=b38db565-4618-4f1a-b37d-f5a96ed63997] [op=none]
[2023/08/21 16:03:01.918 +08:00] [INFO] [manager.go:384] ["set owner op is the same as the original, so do nothing."] ["owner info"="[ddl] /tidb/ddl/fg/owner ownerManager b38db565-4618-4f1a-b37d-f5a96ed63997"] [op=none]
[2023/08/21 16:03:01.918 +08:00] [INFO] [job_table.go:343] ["[ddl] the owner sets owner operator value"] [ownerOp=none]
[2023/08/21 16:03:02.086 +08:00] [INFO] [domain.go:1425] ["a new dist task dispatcher started for current node becomes the DDL owner"]
[2023/08/21 16:03:02.086 +08:00] [INFO] [dispatcher.go:141] ["dispatch task loop start"]
[2023/08/21 16:03:02.086 +08:00] [INFO] [dispatcher.go:250] ["detect task loop start"]
[2023/08/21 16:03:08.899 +08:00] [INFO] [owner_daemon.go:56] ["daemon became owner"] [id=bee7ead2-8f49-4af9-80bf-383cabffcbfe] [daemon-id=LogBackup::Advancer]
[2023/08/21 16:04:21.148 +08:00] [INFO] [gc_worker.go:442] ["[gc worker] starts the whole job"] [uuid=6285c770bcc0009] [safePoint=443706103848763392] [concurrency=3]
[2023/08/21 16:04:21.150 +08:00] [INFO] [gc_worker.go:1250] ["[gc worker] start resolve locks"] [uuid=6285c770bcc0009] [safePoint=443706103848763392] [try-resolve-locks-ts=443706103848763392] [concurrency=3]
[2023/08/21 16:04:21.150 +08:00] [INFO] [range_task.go:137] ["range task started"] [name=resolve-locks-runner] [startKey=] [endKey=] [concurrency=3]
[2023/08/21 16:04:21.442 +08:00] [INFO] [range_task.go:246] ["range task finished"] [name=resolve-locks-runner] [startKey=] [endKey=] ["cost time"=291.030565ms] ["completed regions"=1401]
[2023/08/21 16:04:21.442 +08:00] [INFO] [gc_worker.go:1272] ["[gc worker] finish resolve locks"] [uuid=6285c770bcc0009] [safePoint=443706103848763392] [try-resolve-locks-ts=443706103848763392] [regions=1401]
[2023/08/21 16:05:21.096 +08:00] [INFO] [gc_worker.go:389] ["[gc worker] there's already a gc job running, skipped"] ["leaderTick on"=6285c770bcc0009]
[2023/08/21 16:06:01.462 +08:00] [INFO] [gc_worker.go:888] ["[gc worker] start delete ranges"] [uuid=6285c770bcc0009] [ranges=0]
[2023/08/21 16:06:01.462 +08:00] [INFO] [gc_worker.go:941] ["[gc worker] finish delete ranges"] [uuid=6285c770bcc0009] ["num of ranges"=0] ["cost time"=940ns]
[2023/08/21 16:06:01.464 +08:00] [INFO] [gc_worker.go:964] ["[gc worker] start redo-delete ranges"] [uuid=6285c770bcc0009] ["num of ranges"=0]
[2023/08/21 16:06:01.464 +08:00] [INFO] [gc_worker.go:993] ["[gc worker] finish redo-delete ranges"] [uuid=6285c770bcc0009] ["num of ranges"=0] ["cost time"=550ns]
[2023/08/21 16:06:01.472 +08:00] [INFO] [gc_worker.go:1845] ["[gc worker] sent safe point to PD"] [uuid=6285c770bcc0009] ["safe point"=443706103848763392]
[2023/08/21 16:07:20.673 +08:00] [INFO] [domain.go:2668] ["refreshServerIDTTL succeed"] [serverID=4112750] ["lease id"=24ef8a165d8d3262]
[2023/08/21 16:08:32.289 +08:00] [INFO] [cpuprofile.go:113] ["parallel cpu profiler started"]
[2023/08/21 16:08:32.291 +08:00] [INFO] [printer.go:48] ["Welcome to TiDB."] ["Release Version"=v7.1.1] [Edition=Community] ["Git Commit Hash"=cf441574864be63938524e7dfcf7cc659edc3dd8] ["Git Branch"=heads/refs/tags/v7.1.1] ["UTC Build Time"="2023-07-19 10:16:40"] [GoVersion=go1.20.6] ["Race Enabled"=false] ["Check Table Before Drop"=false] ["TiKV Min Version"=6.2.0-alpha]
[2023/08/21 16:08:32.292 +08:00] [INFO] [printer.go:53] ["loaded config"] [config="{\"host\":\"0.0.0.0\",\"advertise-address\":\"10.223.225.84\",\"port\":4000,\"cors\":\"\",\"store\":\"tikv\",\"path\":\"10.223.225.165:2379,10.223.225.166:2379,10.223.225.167:2379\",\"socket\":\"/tmp/tidb-4000.sock\",\"lease\":\"45s\",\"split-table\":true,\"token-limit\":1000,\"temp-dir\":\"/tmp/tidb\",\"tmp-storage-path\":\"/tmp/1001_tidb/MC4wLjAuMDo0MDAwLzAuMC4wLjA6MTAwODA=/tmp-storage\",\"tmp-storage-quota\":-1,\"server-version\":\"\",\"version-comment\":\"\",\"tidb-edition\":\"\",\"tidb-release-version\":\"\",\"keyspace-name\":\"\",\"log\":{\"level\":\"info\",\"format\":\"text\",\"disable-timestamp\":null,\"enable-timestamp\":null,\"disable-error-stack\":null,\"enable-error-stack\":null,\"file\":{\"filename\":\"/data/tidb/tidb-deploy/tidb-4000/log/tidb.log\",\"max-size\":300,\"max-days\":0,\"max-backups\":0},\"slow-query-file\":\"/data/tidb/tidb-deploy/tidb-4000/log/tidb_slow_query.log\",\"expensive-threshold\":10000,\"query-log-max-len\":4096,\"enable-slow-log\":true,\"slow-threshold\":300,\"record-plan-in-slow-log\":1,\"timeout\":0},\"instance\":{\"tidb_general_log\":false,\"tidb_pprof_sql_cpu\":false,\"ddl_slow_threshold\":300,\"tidb_expensive_query_time_threshold\":60,\"tidb_stmt_summary_enable_persistent\":false,\"tidb_stmt_summary_filename\":\"tidb-statements.log\",\"tidb_stmt_summary_file_max_days\":3,\"tidb_stmt_summary_file_max_size\":64,\"tidb_stmt_summary_file_max_backups\":0,\"tidb_enable_slow_log\":true,\"tidb_slow_log_threshold\":300,\"tidb_record_plan_in_slow_log\":1,\"tidb_check_mb4_value_in_utf8\":true,\"tidb_force_priority\":\"NO_PRIORITY\",\"tidb_memory_usage_alarm_ratio\":0.8,\"tidb_enable_collect_execution_info\":true,\"plugin_dir\":\"/data/deploy/plugin\",\"plugin_load\":\"\",\"max_connections\":0,\"tidb_enable_ddl\":true,\"tidb_rc_read_check_ts\":false},\"security\":{\"skip-grant-table\":false,\"ssl-ca\":\"\",\"ssl-cert\":\"\",\"ssl-key\":\"\",\"cluster-ssl-ca\":\"\",\"cluster-ssl-cert\":\"\",\"cluster-ssl-key\":\"\",\"cluster-verify-cn\":null,\"session-token-signing-cert\":\"\",\"session-token-signing-key\":\"\",\"spilled-file-encryption-method\":\"plaintext\",\"enable-sem\":false,\"auto-tls\":false,\"tls-version\":\"\",\"rsa-key-size\":4096,\"secure-bootstrap\":false,\"auth-token-jwks\":\"\",\"auth-token-refresh-interval\":\"1h0m0s\",\"disconnect-on-expired-password\":true},\"status\":{\"status-host\":\"0.0.0.0\",\"metrics-addr\":\"\",\"status-port\":10080,\"metrics-interval\":15,\"report-status\":true,\"record-db-qps\":false,\"record-db-label\":false,\"grpc-keepalive-time\":10,\"grpc-keepalive-timeout\":3,\"grpc-concurrent-streams\":1024,\"grpc-initial-window-size\":2097152,\"grpc-max-send-msg-size\":2147483647},\"performance\":{\"max-procs\":0,\"max-memory\":0,\"server-memory-quota\":0,\"stats-lease\":\"3s\",\"stmt-count-limit\":5000,\"pseudo-estimate-ratio\":0.8,\"bind-info-lease\":\"3s\",\"txn-entry-size-limit\":6291456,\"txn-total-size-limit\":104857600,\"tcp-keep-alive\":true,\"tcp-no-delay\":true,\"cross-join\":true,\"distinct-agg-push-down\":false,\"projection-push-down\":false,\"max-txn-ttl\":3600000,\"index-usage-sync-lease\":\"0s\",\"plan-replayer-gc-lease\":\"10m\",\"gogc\":100,\"enforce-mpp\":false,\"stats-load-concurrency\":5,\"stats-load-queue-size\":1000,\"analyze-partition-concurrency-quota\":16,\"plan-replayer-dump-worker-concurrency\":1,\"enable-stats-cache-mem-quota\":false,\"committer-concurrency\":128,\"run-auto-analyze\":true,\"force-priority\":\"NO_PRIORITY\",\"memory-usage-alarm-ratio\":0.8,\"enable-load-fmsketch\":false,\"lite-init-stats\":false,\"force-init-stats\":false},\"prepared-plan-cache\":{\"enabled\":true,\"capacity\":100,\"memory-guard-ratio\":0.1},\"opentracing\":{\"enable\":false,\"rpc-metrics\":false,\"sampler\":{\"type\":\"const\",\"param\":1,\"sampling-server-url\":\"\",\"max-operations\":0,\"sampling-refresh-interval\":0},\"reporter\":{\"queue-size\":0,\"buffer-flush-interval\":0,\"log-spans\":false,\"local-agent-host-port\":\"\"}},\"proxy-protocol\":{\"networks\":\"\",\"header-timeout\":5,\"fallbackable\":false},\"pd-client\":{\"pd-server-timeout\":3},\"tikv-client\":{\"grpc-connection-count\":4,\"grpc-keepalive-time\":10,\"grpc-keepalive-timeout\":3,\"grpc-compression-type\":\"none\",\"commit-timeout\":\"41s\",\"async-commit\":{\"keys-limit\":256,\"total-key-size-limit\":4096,\"safe-window\":2000000000,\"allowed-clock-drift\":500000000},\"max-batch-size\":128,\"overload-threshold\":200,\"max-batch-wait-time\":0,\"batch-wait-size\":8,\"enable-chunk-rpc\":true,\"region-cache-ttl\":600,\"store-limit\":0,\"store-liveness-timeout\":\"1s\",\"copr-cache\":{\"capacity-mb\":1000},\"ttl-refreshed-txn-size\":33554432,\"resolve-lock-lite-threshold\":16},\"binlog\":{\"enable\":false,\"ignore-error\":false,\"write-timeout\":\"15s\",\"binlog-socket\":\"\",\"strategy\":\"range\"},\"compatible-kill-query\":false,\"pessimistic-txn\":{\"max-retry-count\":256,\"deadlock-history-capacity\":10,\"deadlock-history-collect-retryable\":false,\"pessimistic-auto-commit\":false,\"constraint-check-in-place-pessimistic\":true},\"max-index-length\":3072,\"index-limit\":64,\"table-column-count-limit\":1017,\"graceful-wait-before-shutdown\":0,\"alter-primary-key\":false,\"treat-old-version-utf8-as-utf8mb4\":true,\"enable-table-lock\":false,\"delay-clean-table-lock\":0,\"split-region-max-num\":1000,\"top-sql\":{\"receiver-address\":\"\"},\"repair-mode\":false,\"repair-table-list\":[],\"isolation-read\":{\"engines\":[\"tikv\",\"tiflash\",\"tidb\"]},\"new_collations_enabled_on_first_bootstrap\":true,\"experimental\":{\"allow-expression-index\":false},\"skip-register-to-dashboard\":false,\"enable-telemetry\":false,\"labels\":{},\"enable-global-index\":false,\"deprecate-integer-display-length\":false,\"enable-enum-length-limit\":true,\"stores-refresh-interval\":60,\"enable-tcp4-only\":false,\"enable-forwarding\":false,\"max-ballast-object-size\":0,\"ballast-object-size\":0,\"transaction-summary\":{\"transaction-summary-capacity\":500,\"transaction-id-digest-min-duration\":2147483647},\"enable-global-kill\":true,\"initialize-sql-file\":\"\",\"enable-batch-dml\":false,\"mem-quota-query\":1073741824,\"oom-action\":\"cancel\",\"oom-use-tmp-storage\":true,\"check-mb4-value-in-utf8\":true,\"enable-collect-execution-info\":true,\"plugin\":{\"dir\":\"/data/deploy/plugin\",\"load\":\"\"},\"max-server-connections\":0,\"run-ddl\":true,\"disaggregated-tiflash\":false,\"autoscaler-type\":\"aws\",\"autoscaler-addr\":\"tiflash-autoscale-lb.tiflash-autoscale.svc.cluster.local:8081\",\"is-tiflashcompute-fixed-pool\":false,\"autoscaler-cluster-id\":\"\",\"use-autoscaler\":false,\"tidb-max-reuse-chunk\":64,\"tidb-max-reuse-column\":256,\"tidb-enable-exit-check\":false}"]
[2023/08/21 16:08:32.292 +08:00] [INFO] [main.go:394] ["disable Prometheus push client"]
[2023/08/21 16:08:32.292 +08:00] [INFO] [store.go:76] ["new store"] [path=tikv://10.223.225.165:2379,10.223.225.166:2379,10.223.225.167:2379]
[2023/08/21 16:08:32.292 +08:00] [INFO] [systime_mon.go:26] ["start system time monitor"]
[2023/08/21 16:08:32.292 +08:00] [INFO] [client.go:311] ["[pd] create pd client with endpoints and keyspace"] [pd-address="[10.223.225.165:2379,10.223.225.166:2379,10.223.225.167:2379]"] [keyspace-id=0]
[2023/08/21 16:08:32.299 +08:00] [INFO] [pd_service_discovery.go:543] ["[pd] switch leader"] [new-leader=http://10.223.225.166:2379] [old-leader=]
[2023/08/21 16:08:32.299 +08:00] [INFO] [pd_service_discovery.go:175] ["[pd] init cluster id"] [cluster-id=7173638068559301829]
[2023/08/21 16:08:32.300 +08:00] [INFO] [client.go:386] ["[pd] changing service mode"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE]
[2023/08/21 16:08:32.300 +08:00] [INFO] [tso_client.go:230] ["[tso] switch dc tso allocator serving address"] [dc-location=global] [new-address=http://10.223.225.166:2379]
[2023/08/21 16:08:32.300 +08:00] [INFO] [tso_dispatcher.go:290] ["[tso] tso dispatcher created"] [dc-location=global]
[2023/08/21 16:08:32.301 +08:00] [INFO] [client.go:428] ["[pd] service mode changed"] [old-mode=PD_SVC_MODE] [new-mode=PD_SVC_MODE]
[2023/08/21 16:08:32.301 +08:00] [INFO] [tikv_driver.go:221] ["using API V1."]
[2023/08/21 16:08:32.302 +08:00] [INFO] [store.go:82] ["new store with retry success"]
[2023/08/21 16:08:32.312 +08:00] [INFO] [tidb.go:80] ["new domain"] [store=tikv-7173638068559301829] ["ddl lease"=45s] ["stats lease"=3s] ["index usage sync lease"=0s]
[2023/08/21 16:08:32.324 +08:00] [INFO] [domain.go:2650] [acquireServerID] [serverID=3833586] ["lease id"=24ef8a165d8d3439]
[2023/08/21 16:08:32.339 +08:00] [WARN] [info.go:304] ["init TiFlashReplicaManager"] ["pd addrs"="[10.223.225.165:2379,10.223.225.167:2379,10.223.225.166:2379]"]
[2023/08/21 16:08:32.534 +08:00] [INFO] [domain.go:272] ["full load InfoSchema success"] [currentSchemaVersion=0] [neededSchemaVersion=3140] ["start time"=181.469975ms]
[2023/08/21 16:08:32.537 +08:00] [INFO] [domain.go:587] ["full load and reset schema validator"]
[2023/08/21 16:08:32.537 +08:00] [INFO] [ddl.go:752] ["[ddl] start DDL"] [ID=d27c8315-cd4a-491e-a799-b347cf7d4542] [runWorker=true]
[2023/08/21 16:08:32.537 +08:00] [INFO] [ddl.go:715] ["[ddl] start delRangeManager OK"] ["is a emulator"=false]
[2023/08/21 16:08:32.544 +08:00] [INFO] [manager.go:178] ["start campaign owner"] [ownerInfo="[ddl] /tidb/ddl/fg/owner"]
[2023/08/21 16:08:32.546 +08:00] [INFO] [env.go:90] ["[ddl-ingest] the ingest sorted directory"] ["data path:"=/tmp/tidb/tmp_ddl-4000]
[2023/08/21 16:08:32.546 +08:00] [WARN] [backend_mgr.go:59] ["[ddl-ingest] ingest backfill may not be available"] [error="the available disk space(33598537728) in /tmp/tidb/tmp_ddl-4000 should be greater than @@tidb_ddl_disk_quota(107374182400)"]
[2023/08/21 16:08:32.546 +08:00] [INFO] [env.go:68] ["[ddl-ingest] init global ingest backend environment finished"] ["memory limitation"=2147483648] ["disk usage info"="disk usage: 1114112/33599651840, backend usage: 0"] ["max open file number"=1000000] ["lightning is initialized"=true]
[2023/08/21 16:08:32.546 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=loadSchemaInLoop]
[2023/08/21 16:08:32.546 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=mdlCheckLoop]
[2023/08/21 16:08:32.547 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=topNSlowQueryLoop]
[2023/08/21 16:08:32.547 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=infoSyncerKeeper]
[2023/08/21 16:08:32.547 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=globalConfigSyncerKeeper]
[2023/08/21 16:08:32.547 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=topologySyncerKeeper]
[2023/08/21 16:08:32.547 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=closestReplicaReadCheckLoop]
[2023/08/21 16:08:32.547 +08:00] [INFO] [job_table.go:328] ["[ddl] get global state and global state change"] [oldState=false] [currState=false]
[2023/08/21 16:08:32.547 +08:00] [INFO] [owner_daemon.go:70] ["begin advancer daemon"] [daemon-id=LogBackup::Advancer]
[2023/08/21 16:08:32.547 +08:00] [INFO] [manager.go:178] ["start campaign owner"] [ownerInfo="[log-backup] /tidb/br-stream/owner"]
[2023/08/21 16:08:32.552 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=logBackupAdvancer]
[2023/08/21 16:08:32.552 +08:00] [INFO] [owner_daemon.go:81] ["begin running daemon"] [id=baf832af-fe57-4544-a5d0-6c36d9935dcc] [daemon-id=LogBackup::Advancer]
[2023/08/21 16:08:32.570 +08:00] [INFO] [manager.go:178] ["start campaign owner"] [ownerInfo="[bindinfo] /tidb/bindinfo/owner"]
[2023/08/21 16:08:32.576 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=globalBindHandleWorkerLoop]
[2023/08/21 16:08:32.576 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=handleEvolvePlanTasksLoop]
[2023/08/21 16:08:32.576 +08:00] [WARN] [sysvar_cache.go:50] ["sysvar cache is empty, triggering rebuild"]
[2023/08/21 16:08:32.581 +08:00] [INFO] [sysvar.go:2432] ["set resource control"] [enable=true]
[2023/08/21 16:08:32.717 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=loadPrivilegeInLoop]
[2023/08/21 16:08:32.721 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=LoadSysVarCacheLoop]
[2023/08/21 16:08:32.723 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=PlanReplayerTaskCollectHandle]
[2023/08/21 16:08:32.723 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=PlanReplayerTaskDumpHandle]
[2023/08/21 16:08:32.723 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=dumpFileGcChecker]
[2023/08/21 16:08:32.723 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=HistoricalStatsWorker]
[2023/08/21 16:08:32.723 +08:00] [INFO] [domain.go:1968] ["PlanReplayerTaskDumpHandle started"]
[2023/08/21 16:08:32.723 +08:00] [INFO] [plan_replayer.go:381] ["planReplayerTaskDumpWorker started."]
[2023/08/21 16:08:32.723 +08:00] [INFO] [domain.go:1946] ["PlanReplayerTaskCollectHandle started"]
[2023/08/21 16:08:32.723 +08:00] [INFO] [domain.go:2000] ["dumpFileGcChecker started"]
[2023/08/21 16:08:32.724 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=loadStatsWorker]
[2023/08/21 16:08:32.724 +08:00] [INFO] [domain.go:2033] ["HistoricalStatsWorker started"]
[2023/08/21 16:08:32.724 +08:00] [INFO] [manager.go:178] ["start campaign owner"] [ownerInfo="[stats] /tidb/stats/owner"]
[2023/08/21 16:08:32.726 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=updateStatsWorker]
[2023/08/21 16:08:32.726 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=autoAnalyzeWorker]
[2023/08/21 16:08:32.726 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=gcAnalyzeHistory]
[2023/08/21 16:08:32.726 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=ttlJobManager]
[2023/08/21 16:08:32.726 +08:00] [INFO] [domain.go:2303] ["updateStatsWorker started."]
[2023/08/21 16:08:32.726 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=loadSigningCertLoop]
[2023/08/21 16:08:32.727 +08:00] [INFO] [gc_worker.go:212] ["[gc worker] start"] [uuid=6285c8dbad40009]
[2023/08/21 16:08:32.728 +08:00] [INFO] [task_manager.go:215] ["scale ttl worker"] [ttl-worker=job-manager] [ttl-worker=task-manager] [originalCount=0] [newCount=4]
[2023/08/21 16:08:32.728 +08:00] [INFO] [task_manager.go:215] ["scale ttl worker"] [ttl-worker=job-manager] [ttl-worker=task-manager] [originalCount=0] [newCount=4]
[2023/08/21 16:08:32.733 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=distTaskFrameworkLoop]
[2023/08/21 16:08:32.733 +08:00] [WARN] [misc.go:458] ["Automatic TLS Certificate creation is disabled"] []
[2023/08/21 16:08:32.733 +08:00] [INFO] [domain.go:1405] ["dist task scheduler started"]
[2023/08/21 16:08:32.733 +08:00] [INFO] [server.go:271] ["server is running MySQL protocol"] [addr=0.0.0.0:4000]
[2023/08/21 16:08:32.733 +08:00] [INFO] [server.go:285] ["server is running MySQL protocol"] [socket=/tmp/tidb-4000.sock]
[2023/08/21 16:08:32.733 +08:00] [INFO] [http_status.go:90] ["for status and metrics report"] ["listening on addr"=0.0.0.0:10080]
[2023/08/21 16:08:32.735 +08:00] [INFO] [cpu.go:83] ["sql cpu collector started"]
[2023/08/21 16:08:32.737 +08:00] [INFO] [store.go:76] ["new store"] [path=tikv://10.223.225.165:2379,10.223.225.166:2379,10.223.225.167:2379]
[2023/08/21 16:08:32.737 +08:00] [INFO] [client.go:311] ["[pd] create pd client with endpoints and keyspace"] [pd-address="[10.223.225.165:2379,10.223.225.166:2379,10.223.225.167:2379]"] [keyspace-id=0]
[2023/08/21 16:08:32.742 +08:00] [INFO] [pd_service_discovery.go:543] ["[pd] switch leader"] [new-leader=http://10.223.225.166:2379] [old-leader=]
[2023/08/21 16:08:32.742 +08:00] [INFO] [pd_service_discovery.go:175] ["[pd] init cluster id"] [cluster-id=7173638068559301829]
[2023/08/21 16:08:32.743 +08:00] [INFO] [client.go:386] ["[pd] changing service mode"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE]
[2023/08/21 16:08:32.743 +08:00] [INFO] [tso_client.go:230] ["[tso] switch dc tso allocator serving address"] [dc-location=global] [new-address=http://10.223.225.166:2379]
[2023/08/21 16:08:32.744 +08:00] [INFO] [tso_dispatcher.go:290] ["[tso] tso dispatcher created"] [dc-location=global]
[2023/08/21 16:08:32.744 +08:00] [INFO] [client.go:428] ["[pd] service mode changed"] [old-mode=PD_SVC_MODE] [new-mode=PD_SVC_MODE]
[2023/08/21 16:08:32.744 +08:00] [INFO] [tso_client.go:134] ["closing tso client"]
[2023/08/21 16:08:32.744 +08:00] [INFO] [tso_dispatcher.go:214] ["exit tso dispatcher loop"]
[2023/08/21 16:08:32.744 +08:00] [INFO] [tso_dispatcher.go:162] ["exit tso requests cancel loop"]
[2023/08/21 16:08:32.744 +08:00] [WARN] [resource_manager_client.go:295] ["[resource_manager] get token stream error"] [error="rpc error: code = Canceled desc = context canceled"]
[2023/08/21 16:08:32.744 +08:00] [ERROR] [tso_dispatcher.go:514] ["[tso] update connection contexts failed"] [dc=global] [error="rpc error: code = Canceled desc = context canceled"]
[2023/08/21 16:08:32.744 +08:00] [INFO] [tso_dispatcher.go:311] ["[tso] exit tso dispatcher"] [dc-location=global]
[2023/08/21 16:08:32.744 +08:00] [INFO] [tso_client.go:139] ["close tso client"]
[2023/08/21 16:08:32.744 +08:00] [INFO] [tso_client.go:150] ["tso client is closed"]
[2023/08/21 16:08:32.744 +08:00] [INFO] [pd_service_discovery.go:250] ["[pd] close pd service discovery client"]
[2023/08/21 16:08:32.744 +08:00] [INFO] [store.go:82] ["new store with retry success"]
[2023/08/21 16:08:32.745 +08:00] [INFO] [manager.go:178] ["start campaign owner"] [ownerInfo="[autoid] tidb/autoid/leader"]
[2023/08/21 16:08:32.789 +08:00] [INFO] [http_status.go:492] ["register auto service at"] [addr=10.223.225.84:10080]
[2023/08/21 16:08:33.689 +08:00] [INFO] [domain.go:2208] ["init stats info time"] [lite=false] ["take time"=965.536212ms]
[2023/08/21 16:08:34.547 +08:00] [INFO] [info.go:1189] [SetTiFlashGroupConfig]
[2023/08/21 16:09:01.940 +08:00] [INFO] [manager.go:344] ["get owner"] ["owner info"="[bindinfo] /tidb/bindinfo/owner ownerManager d27c8315-cd4a-491e-a799-b347cf7d4542"] ["owner key"=/tidb/bindinfo/owner/24ef8a165d8d3442] [ownerID=d27c8315-cd4a-491e-a799-b347cf7d4542] [op=none]
[2023/08/21 16:09:01.941 +08:00] [INFO] [manager.go:344] ["get owner"] ["owner info"="[autoid] tidb/autoid/leader ownerManager 10.223.225.84:10080"] ["owner key"=tidb/autoid/leader/1c688a165d8e22ac] [ownerID=10.223.225.84:10080] [op=none]
[2023/08/21 16:09:01.941 +08:00] [INFO] [manager.go:344] ["get owner"] ["owner info"="[stats] /tidb/stats/owner ownerManager d27c8315-cd4a-491e-a799-b347cf7d4542"] ["owner key"=/tidb/stats/owner/1c688a165d8e22a9] [ownerID=d27c8315-cd4a-491e-a799-b347cf7d4542] [op=none]
[2023/08/21 16:09:01.941 +08:00] [INFO] [manager.go:344] ["get owner"] ["owner info"="[log-backup] /tidb/br-stream/owner ownerManager baf832af-fe57-4544-a5d0-6c36d9935dcc"] ["owner key"=/tidb/br-stream/owner/77568a165d6afce8] [ownerID=baf832af-fe57-4544-a5d0-6c36d9935dcc] [op=none]
[2023/08/21 16:09:01.942 +08:00] [INFO] [manager.go:344] ["get owner"] ["owner info"="[ddl] /tidb/ddl/fg/owner ownerManager d27c8315-cd4a-491e-a799-b347cf7d4542"] ["owner key"=/tidb/ddl/fg/owner/77568a165d6afce6] [ownerID=d27c8315-cd4a-491e-a799-b347cf7d4542] [op=none]
[2023/08/21 16:09:02.565 +08:00] [INFO] [job_table.go:328] ["[ddl] get global state and global state change"] [oldState=false] [currState=false]
[2023/08/21 16:09:02.566 +08:00] [INFO] [manager.go:344] ["get owner"] ["owner info"="[ddl] /tidb/ddl/fg/owner ownerManager d27c8315-cd4a-491e-a799-b347cf7d4542"] ["owner key"=/tidb/ddl/fg/owner/77568a165d6afce6] [ownerID=d27c8315-cd4a-491e-a799-b347cf7d4542] [op=none]
[2023/08/21 16:09:02.566 +08:00] [INFO] [manager.go:384] ["set owner op is the same as the original, so do nothing."] ["owner info"="[ddl] /tidb/ddl/fg/owner ownerManager d27c8315-cd4a-491e-a799-b347cf7d4542"] [op=none]
[2023/08/21 16:09:02.566 +08:00] [INFO] [job_table.go:343] ["[ddl] the owner sets owner operator value"] [ownerOp=none]
[2023/08/21 16:09:02.734 +08:00] [INFO] [domain.go:1425] ["a new dist task dispatcher started for current node becomes the DDL owner"]
[2023/08/21 16:09:02.734 +08:00] [INFO] [dispatcher.go:141] ["dispatch task loop start"]
[2023/08/21 16:09:02.734 +08:00] [INFO] [dispatcher.go:250] ["detect task loop start"]
[2023/08/21 16:09:08.553 +08:00] [INFO] [owner_daemon.go:56] ["daemon became owner"] [id=baf832af-fe57-4544-a5d0-6c36d9935dcc] [daemon-id=LogBackup::Advancer]
[2023/08/21 16:13:32.328 +08:00] [INFO] [domain.go:2668] ["refreshServerIDTTL succeed"] [serverID=3833586] ["lease id"=24ef8a165d8d3439]
[2023/08/21 16:14:47.400 +08:00] [INFO] [cpuprofile.go:113] ["parallel cpu profiler started"]
[2023/08/21 16:14:47.402 +08:00] [INFO] [printer.go:48] ["Welcome to TiDB."] ["Release Version"=v7.1.1] [Edition=Community] ["Git Commit Hash"=cf441574864be63938524e7dfcf7cc659edc3dd8] ["Git Branch"=heads/refs/tags/v7.1.1] ["UTC Build Time"="2023-07-19 10:16:40"] [GoVersion=go1.20.6] ["Race Enabled"=false] ["Check Table Before Drop"=false] ["TiKV Min Version"=6.2.0-alpha]
[2023/08/21 16:14:47.403 +08:00] [INFO] [printer.go:53] ["loaded config"] [config="{\"host\":\"0.0.0.0\",\"advertise-address\":\"10.223.225.84\",\"port\":4000,\"cors\":\"\",\"store\":\"tikv\",\"path\":\"10.223.225.165:2379,10.223.225.166:2379,10.223.225.167:2379\",\"socket\":\"/tmp/tidb-4000.sock\",\"lease\":\"45s\",\"split-table\":true,\"token-limit\":1000,\"temp-dir\":\"/tmp/tidb\",\"tmp-storage-path\":\"/tmp/1001_tidb/MC4wLjAuMDo0MDAwLzAuMC4wLjA6MTAwODA=/tmp-storage\",\"tmp-storage-quota\":-1,\"server-version\":\"\",\"version-comment\":\"\",\"tidb-edition\":\"\",\"tidb-release-version\":\"\",\"keyspace-name\":\"\",\"log\":{\"level\":\"info\",\"format\":\"text\",\"disable-timestamp\":null,\"enable-timestamp\":null,\"disable-error-stack\":null,\"enable-error-stack\":null,\"file\":{\"filename\":\"/data/tidb/tidb-deploy/tidb-4000/log/tidb.log\",\"max-size\":300,\"max-days\":0,\"max-backups\":0},\"slow-query-file\":\"/data/tidb/tidb-deploy/tidb-4000/log/tidb_slow_query.log\",\"expensive-threshold\":10000,\"query-log-max-len\":4096,\"enable-slow-log\":true,\"slow-threshold\":300,\"record-plan-in-slow-log\":1,\"timeout\":0},\"instance\":{\"tidb_general_log\":false,\"tidb_pprof_sql_cpu\":false,\"ddl_slow_threshold\":300,\"tidb_expensive_query_time_threshold\":60,\"tidb_stmt_summary_enable_persistent\":false,\"tidb_stmt_summary_filename\":\"tidb-statements.log\",\"tidb_stmt_summary_file_max_days\":3,\"tidb_stmt_summary_file_max_size\":64,\"tidb_stmt_summary_file_max_backups\":0,\"tidb_enable_slow_log\":true,\"tidb_slow_log_threshold\":300,\"tidb_record_plan_in_slow_log\":1,\"tidb_check_mb4_value_in_utf8\":true,\"tidb_force_priority\":\"NO_PRIORITY\",\"tidb_memory_usage_alarm_ratio\":0.8,\"tidb_enable_collect_execution_info\":true,\"plugin_dir\":\"/data/deploy/plugin\",\"plugin_load\":\"\",\"max_connections\":0,\"tidb_enable_ddl\":true,\"tidb_rc_read_check_ts\":false},\"security\":{\"skip-grant-table\":false,\"ssl-ca\":\"\",\"ssl-cert\":\"\",\"ssl-key\":\"\",\"cluster-ssl-ca\":\"\",\"cluster-ssl-cert\":\"\",\"cluster-ssl-key\":\"\",\"cluster-verify-cn\":null,\"session-token-signing-cert\":\"\",\"session-token-signing-key\":\"\",\"spilled-file-encryption-method\":\"plaintext\",\"enable-sem\":false,\"auto-tls\":false,\"tls-version\":\"\",\"rsa-key-size\":4096,\"secure-bootstrap\":false,\"auth-token-jwks\":\"\",\"auth-token-refresh-interval\":\"1h0m0s\",\"disconnect-on-expired-password\":true},\"status\":{\"status-host\":\"0.0.0.0\",\"metrics-addr\":\"\",\"status-port\":10080,\"metrics-interval\":15,\"report-status\":true,\"record-db-qps\":false,\"record-db-label\":false,\"grpc-keepalive-time\":10,\"grpc-keepalive-timeout\":3,\"grpc-concurrent-streams\":1024,\"grpc-initial-window-size\":2097152,\"grpc-max-send-msg-size\":2147483647},\"performance\":{\"max-procs\":0,\"max-memory\":0,\"server-memory-quota\":0,\"stats-lease\":\"3s\",\"stmt-count-limit\":5000,\"pseudo-estimate-ratio\":0.8,\"bind-info-lease\":\"3s\",\"txn-entry-size-limit\":6291456,\"txn-total-size-limit\":104857600,\"tcp-keep-alive\":true,\"tcp-no-delay\":true,\"cross-join\":true,\"distinct-agg-push-down\":false,\"projection-push-down\":false,\"max-txn-ttl\":3600000,\"index-usage-sync-lease\":\"0s\",\"plan-replayer-gc-lease\":\"10m\",\"gogc\":100,\"enforce-mpp\":false,\"stats-load-concurrency\":5,\"stats-load-queue-size\":1000,\"analyze-partition-concurrency-quota\":16,\"plan-replayer-dump-worker-concurrency\":1,\"enable-stats-cache-mem-quota\":false,\"committer-concurrency\":128,\"run-auto-analyze\":true,\"force-priority\":\"NO_PRIORITY\",\"memory-usage-alarm-ratio\":0.8,\"enable-load-fmsketch\":false,\"lite-init-stats\":false,\"force-init-stats\":false},\"prepared-plan-cache\":{\"enabled\":true,\"capacity\":100,\"memory-guard-ratio\":0.1},\"opentracing\":{\"enable\":false,\"rpc-metrics\":false,\"sampler\":{\"type\":\"const\",\"param\":1,\"sampling-server-url\":\"\",\"max-operations\":0,\"sampling-refresh-interval\":0},\"reporter\":{\"queue-size\":0,\"buffer-flush-interval\":0,\"log-spans\":false,\"local-agent-host-port\":\"\"}},\"proxy-protocol\":{\"networks\":\"\",\"header-timeout\":5,\"fallbackable\":false},\"pd-client\":{\"pd-server-timeout\":3},\"tikv-client\":{\"grpc-connection-count\":4,\"grpc-keepalive-time\":10,\"grpc-keepalive-timeout\":3,\"grpc-compression-type\":\"none\",\"commit-timeout\":\"41s\",\"async-commit\":{\"keys-limit\":256,\"total-key-size-limit\":4096,\"safe-window\":2000000000,\"allowed-clock-drift\":500000000},\"max-batch-size\":128,\"overload-threshold\":200,\"max-batch-wait-time\":0,\"batch-wait-size\":8,\"enable-chunk-rpc\":true,\"region-cache-ttl\":600,\"store-limit\":0,\"store-liveness-timeout\":\"1s\",\"copr-cache\":{\"capacity-mb\":1000},\"ttl-refreshed-txn-size\":33554432,\"resolve-lock-lite-threshold\":16},\"binlog\":{\"enable\":false,\"ignore-error\":false,\"write-timeout\":\"15s\",\"binlog-socket\":\"\",\"strategy\":\"range\"},\"compatible-kill-query\":false,\"pessimistic-txn\":{\"max-retry-count\":256,\"deadlock-history-capacity\":10,\"deadlock-history-collect-retryable\":false,\"pessimistic-auto-commit\":false,\"constraint-check-in-place-pessimistic\":true},\"max-index-length\":3072,\"index-limit\":64,\"table-column-count-limit\":1017,\"graceful-wait-before-shutdown\":0,\"alter-primary-key\":false,\"treat-old-version-utf8-as-utf8mb4\":true,\"enable-table-lock\":false,\"delay-clean-table-lock\":0,\"split-region-max-num\":1000,\"top-sql\":{\"receiver-address\":\"\"},\"repair-mode\":false,\"repair-table-list\":[],\"isolation-read\":{\"engines\":[\"tikv\",\"tiflash\",\"tidb\"]},\"new_collations_enabled_on_first_bootstrap\":true,\"experimental\":{\"allow-expression-index\":false},\"skip-register-to-dashboard\":false,\"enable-telemetry\":false,\"labels\":{},\"enable-global-index\":false,\"deprecate-integer-display-length\":false,\"enable-enum-length-limit\":true,\"stores-refresh-interval\":60,\"enable-tcp4-only\":false,\"enable-forwarding\":false,\"max-ballast-object-size\":0,\"ballast-object-size\":0,\"transaction-summary\":{\"transaction-summary-capacity\":500,\"transaction-id-digest-min-duration\":2147483647},\"enable-global-kill\":true,\"initialize-sql-file\":\"\",\"enable-batch-dml\":false,\"mem-quota-query\":1073741824,\"oom-action\":\"cancel\",\"oom-use-tmp-storage\":true,\"check-mb4-value-in-utf8\":true,\"enable-collect-execution-info\":true,\"plugin\":{\"dir\":\"/data/deploy/plugin\",\"load\":\"\"},\"max-server-connections\":0,\"run-ddl\":true,\"disaggregated-tiflash\":false,\"autoscaler-type\":\"aws\",\"autoscaler-addr\":\"tiflash-autoscale-lb.tiflash-autoscale.svc.cluster.local:8081\",\"is-tiflashcompute-fixed-pool\":false,\"autoscaler-cluster-id\":\"\",\"use-autoscaler\":false,\"tidb-max-reuse-chunk\":64,\"tidb-max-reuse-column\":256,\"tidb-enable-exit-check\":false}"]
[2023/08/21 16:14:47.403 +08:00] [INFO] [main.go:394] ["disable Prometheus push client"]
[2023/08/21 16:14:47.403 +08:00] [INFO] [store.go:76] ["new store"] [path=tikv://10.223.225.165:2379,10.223.225.166:2379,10.223.225.167:2379]
[2023/08/21 16:14:47.403 +08:00] [INFO] [client.go:311] ["[pd] create pd client with endpoints and keyspace"] [pd-address="[10.223.225.165:2379,10.223.225.166:2379,10.223.225.167:2379]"] [keyspace-id=0]
[2023/08/21 16:14:47.403 +08:00] [INFO] [systime_mon.go:26] ["start system time monitor"]
[2023/08/21 16:14:47.410 +08:00] [INFO] [pd_service_discovery.go:543] ["[pd] switch leader"] [new-leader=http://10.223.225.166:2379] [old-leader=]
[2023/08/21 16:14:47.410 +08:00] [INFO] [pd_service_discovery.go:175] ["[pd] init cluster id"] [cluster-id=7173638068559301829]
[2023/08/21 16:14:47.410 +08:00] [INFO] [client.go:386] ["[pd] changing service mode"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE]
[2023/08/21 16:14:47.410 +08:00] [INFO] [tso_client.go:230] ["[tso] switch dc tso allocator serving address"] [dc-location=global] [new-address=http://10.223.225.166:2379]
[2023/08/21 16:14:47.411 +08:00] [INFO] [tso_dispatcher.go:290] ["[tso] tso dispatcher created"] [dc-location=global]
[2023/08/21 16:14:47.411 +08:00] [INFO] [client.go:428] ["[pd] service mode changed"] [old-mode=PD_SVC_MODE] [new-mode=PD_SVC_MODE]
[2023/08/21 16:14:47.412 +08:00] [INFO] [tikv_driver.go:221] ["using API V1."]
[2023/08/21 16:14:47.412 +08:00] [INFO] [store.go:82] ["new store with retry success"]
[2023/08/21 16:14:47.422 +08:00] [INFO] [tidb.go:80] ["new domain"] [store=tikv-7173638068559301829] ["ddl lease"=45s] ["stats lease"=3s] ["index usage sync lease"=0s]
[2023/08/21 16:14:47.436 +08:00] [INFO] [domain.go:2650] [acquireServerID] [serverID=2725826] ["lease id"=1c688a165d8e2478]
[2023/08/21 16:14:47.457 +08:00] [WARN] [info.go:304] ["init TiFlashReplicaManager"] ["pd addrs"="[10.223.225.165:2379,10.223.225.167:2379,10.223.225.166:2379]"]
[2023/08/21 16:14:47.644 +08:00] [INFO] [domain.go:272] ["full load InfoSchema success"] [currentSchemaVersion=0] [neededSchemaVersion=3140] ["start time"=174.169259ms]
[2023/08/21 16:14:47.647 +08:00] [INFO] [domain.go:587] ["full load and reset schema validator"]
[2023/08/21 16:14:47.647 +08:00] [INFO] [ddl.go:752] ["[ddl] start DDL"] [ID=c443219b-e234-44b8-b14e-096fb82e6977] [runWorker=true]
[2023/08/21 16:14:47.648 +08:00] [INFO] [ddl.go:715] ["[ddl] start delRangeManager OK"] ["is a emulator"=false]
[2023/08/21 16:14:47.651 +08:00] [INFO] [manager.go:178] ["start campaign owner"] [ownerInfo="[ddl] /tidb/ddl/fg/owner"]
[2023/08/21 16:14:47.654 +08:00] [INFO] [env.go:90] ["[ddl-ingest] the ingest sorted directory"] ["data path:"=/tmp/tidb/tmp_ddl-4000]
[2023/08/21 16:14:47.654 +08:00] [WARN] [backend_mgr.go:59] ["[ddl-ingest] ingest backfill may not be available"] [error="the available disk space(33598537728) in /tmp/tidb/tmp_ddl-4000 should be greater than @@tidb_ddl_disk_quota(107374182400)"]
[2023/08/21 16:14:47.654 +08:00] [INFO] [env.go:68] ["[ddl-ingest] init global ingest backend environment finished"] ["memory limitation"=2147483648] ["disk usage info"="disk usage: 1114112/33599651840, backend usage: 0"] ["max open file number"=1000000] ["lightning is initialized"=true]
[2023/08/21 16:14:47.654 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=loadSchemaInLoop]
[2023/08/21 16:14:47.654 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=mdlCheckLoop]
[2023/08/21 16:14:47.654 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=topNSlowQueryLoop]
[2023/08/21 16:14:47.654 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=infoSyncerKeeper]
[2023/08/21 16:14:47.654 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=globalConfigSyncerKeeper]
[2023/08/21 16:14:47.654 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=topologySyncerKeeper]
[2023/08/21 16:14:47.654 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=closestReplicaReadCheckLoop]
[2023/08/21 16:14:47.654 +08:00] [INFO] [job_table.go:328] ["[ddl] get global state and global state change"] [oldState=false] [currState=false]
[2023/08/21 16:14:47.654 +08:00] [INFO] [owner_daemon.go:70] ["begin advancer daemon"] [daemon-id=LogBackup::Advancer]
[2023/08/21 16:14:47.654 +08:00] [INFO] [manager.go:178] ["start campaign owner"] [ownerInfo="[log-backup] /tidb/br-stream/owner"]
[2023/08/21 16:14:47.659 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=logBackupAdvancer]
[2023/08/21 16:14:47.659 +08:00] [INFO] [owner_daemon.go:81] ["begin running daemon"] [id=48415bc5-6998-4e91-a930-330c4be312da] [daemon-id=LogBackup::Advancer]
[2023/08/21 16:14:47.669 +08:00] [INFO] [manager.go:178] ["start campaign owner"] [ownerInfo="[bindinfo] /tidb/bindinfo/owner"]
[2023/08/21 16:14:47.671 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=globalBindHandleWorkerLoop]
[2023/08/21 16:14:47.671 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=handleEvolvePlanTasksLoop]
[2023/08/21 16:14:47.671 +08:00] [WARN] [sysvar_cache.go:50] ["sysvar cache is empty, triggering rebuild"]
[2023/08/21 16:14:47.675 +08:00] [INFO] [sysvar.go:2432] ["set resource control"] [enable=true]
[2023/08/21 16:14:47.808 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=loadPrivilegeInLoop]
[2023/08/21 16:14:47.812 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=LoadSysVarCacheLoop]
[2023/08/21 16:14:47.814 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=PlanReplayerTaskCollectHandle]
[2023/08/21 16:14:47.814 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=PlanReplayerTaskDumpHandle]
[2023/08/21 16:14:47.814 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=dumpFileGcChecker]
[2023/08/21 16:14:47.814 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=HistoricalStatsWorker]
[2023/08/21 16:14:47.814 +08:00] [INFO] [domain.go:1968] ["PlanReplayerTaskDumpHandle started"]
[2023/08/21 16:14:47.814 +08:00] [INFO] [domain.go:2000] ["dumpFileGcChecker started"]
[2023/08/21 16:14:47.814 +08:00] [INFO] [plan_replayer.go:381] ["planReplayerTaskDumpWorker started."]
[2023/08/21 16:14:47.814 +08:00] [INFO] [domain.go:1946] ["PlanReplayerTaskCollectHandle started"]
[2023/08/21 16:14:47.815 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=loadStatsWorker]
[2023/08/21 16:14:47.815 +08:00] [INFO] [domain.go:2033] ["HistoricalStatsWorker started"]
[2023/08/21 16:14:47.815 +08:00] [INFO] [manager.go:178] ["start campaign owner"] [ownerInfo="[stats] /tidb/stats/owner"]
[2023/08/21 16:14:47.817 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=updateStatsWorker]
[2023/08/21 16:14:47.817 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=autoAnalyzeWorker]
[2023/08/21 16:14:47.817 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=gcAnalyzeHistory]
[2023/08/21 16:14:47.818 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=ttlJobManager]
[2023/08/21 16:14:47.818 +08:00] [INFO] [domain.go:2303] ["updateStatsWorker started."]
[2023/08/21 16:14:47.818 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=loadSigningCertLoop]
[2023/08/21 16:14:47.818 +08:00] [INFO] [gc_worker.go:212] ["[gc worker] start"] [uuid=6285ca49fc40009]
[2023/08/21 16:14:47.819 +08:00] [INFO] [task_manager.go:215] ["scale ttl worker"] [ttl-worker=job-manager] [ttl-worker=task-manager] [originalCount=0] [newCount=4]
[2023/08/21 16:14:47.819 +08:00] [INFO] [task_manager.go:215] ["scale ttl worker"] [ttl-worker=job-manager] [ttl-worker=task-manager] [originalCount=0] [newCount=4]
[2023/08/21 16:14:47.823 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=distTaskFrameworkLoop]
[2023/08/21 16:14:47.824 +08:00] [WARN] [misc.go:458] ["Automatic TLS Certificate creation is disabled"] []
[2023/08/21 16:14:47.824 +08:00] [INFO] [domain.go:1405] ["dist task scheduler started"]
[2023/08/21 16:14:47.824 +08:00] [INFO] [server.go:271] ["server is running MySQL protocol"] [addr=0.0.0.0:4000]
[2023/08/21 16:14:47.824 +08:00] [INFO] [server.go:285] ["server is running MySQL protocol"] [socket=/tmp/tidb-4000.sock]
[2023/08/21 16:14:47.824 +08:00] [INFO] [http_status.go:90] ["for status and metrics report"] ["listening on addr"=0.0.0.0:10080]
[2023/08/21 16:14:47.825 +08:00] [INFO] [cpu.go:83] ["sql cpu collector started"]
[2023/08/21 16:14:47.827 +08:00] [INFO] [store.go:76] ["new store"] [path=tikv://10.223.225.165:2379,10.223.225.166:2379,10.223.225.167:2379]
[2023/08/21 16:14:47.827 +08:00] [INFO] [client.go:311] ["[pd] create pd client with endpoints and keyspace"] [pd-address="[10.223.225.165:2379,10.223.225.166:2379,10.223.225.167:2379]"] [keyspace-id=0]
[2023/08/21 16:14:47.832 +08:00] [INFO] [pd_service_discovery.go:543] ["[pd] switch leader"] [new-leader=http://10.223.225.166:2379] [old-leader=]
[2023/08/21 16:14:47.832 +08:00] [INFO] [pd_service_discovery.go:175] ["[pd] init cluster id"] [cluster-id=7173638068559301829]
[2023/08/21 16:14:47.833 +08:00] [INFO] [client.go:386] ["[pd] changing service mode"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE]
[2023/08/21 16:14:47.833 +08:00] [INFO] [tso_client.go:230] ["[tso] switch dc tso allocator serving address"] [dc-location=global] [new-address=http://10.223.225.166:2379]
[2023/08/21 16:14:47.833 +08:00] [INFO] [tso_dispatcher.go:290] ["[tso] tso dispatcher created"] [dc-location=global]
[2023/08/21 16:14:47.833 +08:00] [INFO] [client.go:428] ["[pd] service mode changed"] [old-mode=PD_SVC_MODE] [new-mode=PD_SVC_MODE]
[2023/08/21 16:14:47.833 +08:00] [INFO] [tso_dispatcher.go:375] ["[tso] stop fetching the pending tso requests due to context canceled"] [dc-location=global]
[2023/08/21 16:14:47.834 +08:00] [INFO] [tso_dispatcher.go:311] ["[tso] exit tso dispatcher"] [dc-location=global]
[2023/08/21 16:14:47.834 +08:00] [INFO] [tso_dispatcher.go:214] ["exit tso dispatcher loop"]
[2023/08/21 16:14:47.834 +08:00] [INFO] [tso_dispatcher.go:162] ["exit tso requests cancel loop"]
[2023/08/21 16:14:47.834 +08:00] [WARN] [resource_manager_client.go:295] ["[resource_manager] get token stream error"] [error="rpc error: code = Canceled desc = context canceled"]
[2023/08/21 16:14:47.834 +08:00] [INFO] [pd_service_discovery.go:435] ["[pd] cannot update member from this address"] [address=http://10.223.225.165:2379] [error="[PD:client:ErrClientGetMember]error:rpc error: code = Canceled desc = context canceled target:10.223.225.165:2379 status:READY: error:rpc error: code = Canceled desc = context canceled target:10.223.225.165:2379 status:READY"]
[2023/08/21 16:14:47.834 +08:00] [ERROR] [pd_service_discovery.go:221] ["[pd] failed to update member"] [urls="[http://10.223.225.165:2379,http://10.223.225.166:2379,http://10.223.225.167:2379]"] [error="[PD:client:ErrClientGetMember]error:rpc error: code = Canceled desc = context canceled target:10.223.225.165:2379 status:READY: error:rpc error: code = Canceled desc = context canceled target:10.223.225.165:2379 status:READY"]
[2023/08/21 16:14:47.834 +08:00] [INFO] [tso_client.go:134] ["closing tso client"]
[2023/08/21 16:14:47.834 +08:00] [INFO] [tso_client.go:139] ["close tso client"]
[2023/08/21 16:14:47.834 +08:00] [INFO] [tso_client.go:150] ["tso client is closed"]
[2023/08/21 16:14:47.834 +08:00] [INFO] [pd_service_discovery.go:250] ["[pd] close pd service discovery client"]
[2023/08/21 16:14:47.834 +08:00] [INFO] [store.go:82] ["new store with retry success"]
[2023/08/21 16:14:47.836 +08:00] [INFO] [manager.go:178] ["start campaign owner"] [ownerInfo="[autoid] tidb/autoid/leader"]
[2023/08/21 16:14:47.841 +08:00] [INFO] [http_status.go:492] ["register auto service at"] [addr=10.223.225.84:10080]
[2023/08/21 16:14:49.654 +08:00] [INFO] [info.go:1189] [SetTiFlashGroupConfig]
[2023/08/21 16:14:49.790 +08:00] [INFO] [domain.go:2208] ["init stats info time"] [lite=false] ["take time"=1.975129926s]
[2023/08/21 16:15:15.205 +08:00] [INFO] [cpuprofile.go:113] ["parallel cpu profiler started"]
[2023/08/21 16:15:15.206 +08:00] [INFO] [printer.go:48] ["Welcome to TiDB."] ["Release Version"=v7.1.1] [Edition=Community] ["Git Commit Hash"=cf441574864be63938524e7dfcf7cc659edc3dd8] ["Git Branch"=heads/refs/tags/v7.1.1] ["UTC Build Time"="2023-07-19 10:16:40"] [GoVersion=go1.20.6] ["Race Enabled"=false] ["Check Table Before Drop"=false] ["TiKV Min Version"=6.2.0-alpha]
[2023/08/21 16:15:15.207 +08:00] [INFO] [printer.go:53] ["loaded config"] [config="{\"host\":\"0.0.0.0\",\"advertise-address\":\"10.223.225.84\",\"port\":4000,\"cors\":\"\",\"store\":\"tikv\",\"path\":\"10.223.225.165:2379,10.223.225.166:2379,10.223.225.167:2379\",\"socket\":\"/tmp/tidb-4000.sock\",\"lease\":\"45s\",\"split-table\":true,\"token-limit\":1000,\"temp-dir\":\"/tmp/tidb\",\"tmp-storage-path\":\"/tmp/1001_tidb/MC4wLjAuMDo0MDAwLzAuMC4wLjA6MTAwODA=/tmp-storage\",\"tmp-storage-quota\":-1,\"server-version\":\"\",\"version-comment\":\"\",\"tidb-edition\":\"\",\"tidb-release-version\":\"\",\"keyspace-name\":\"\",\"log\":{\"level\":\"info\",\"format\":\"text\",\"disable-timestamp\":null,\"enable-timestamp\":null,\"disable-error-stack\":null,\"enable-error-stack\":null,\"file\":{\"filename\":\"/data/tidb/tidb-deploy/tidb-4000/log/tidb.log\",\"max-size\":300,\"max-days\":0,\"max-backups\":0},\"slow-query-file\":\"/data/tidb/tidb-deploy/tidb-4000/log/tidb_slow_query.log\",\"expensive-threshold\":10000,\"query-log-max-len\":4096,\"enable-slow-log\":true,\"slow-threshold\":300,\"record-plan-in-slow-log\":1,\"timeout\":0},\"instance\":{\"tidb_general_log\":false,\"tidb_pprof_sql_cpu\":false,\"ddl_slow_threshold\":300,\"tidb_expensive_query_time_threshold\":60,\"tidb_stmt_summary_enable_persistent\":false,\"tidb_stmt_summary_filename\":\"tidb-statements.log\",\"tidb_stmt_summary_file_max_days\":3,\"tidb_stmt_summary_file_max_size\":64,\"tidb_stmt_summary_file_max_backups\":0,\"tidb_enable_slow_log\":true,\"tidb_slow_log_threshold\":300,\"tidb_record_plan_in_slow_log\":1,\"tidb_check_mb4_value_in_utf8\":true,\"tidb_force_priority\":\"NO_PRIORITY\",\"tidb_memory_usage_alarm_ratio\":0.8,\"tidb_enable_collect_execution_info\":true,\"plugin_dir\":\"/data/deploy/plugin\",\"plugin_load\":\"\",\"max_connections\":0,\"tidb_enable_ddl\":true,\"tidb_rc_read_check_ts\":false},\"security\":{\"skip-grant-table\":false,\"ssl-ca\":\"\",\"ssl-cert\":\"\",\"ssl-key\":\"\",\"cluster-ssl-ca\":\"\",\"cluster-ssl-cert\":\"\",\"cluster-ssl-key\":\"\",\"cluster-verify-cn\":null,\"session-token-signing-cert\":\"\",\"session-token-signing-key\":\"\",\"spilled-file-encryption-method\":\"plaintext\",\"enable-sem\":false,\"auto-tls\":false,\"tls-version\":\"\",\"rsa-key-size\":4096,\"secure-bootstrap\":false,\"auth-token-jwks\":\"\",\"auth-token-refresh-interval\":\"1h0m0s\",\"disconnect-on-expired-password\":true},\"status\":{\"status-host\":\"0.0.0.0\",\"metrics-addr\":\"\",\"status-port\":10080,\"metrics-interval\":15,\"report-status\":true,\"record-db-qps\":false,\"record-db-label\":false,\"grpc-keepalive-time\":10,\"grpc-keepalive-timeout\":3,\"grpc-concurrent-streams\":1024,\"grpc-initial-window-size\":2097152,\"grpc-max-send-msg-size\":2147483647},\"performance\":{\"max-procs\":0,\"max-memory\":0,\"server-memory-quota\":0,\"stats-lease\":\"3s\",\"stmt-count-limit\":5000,\"pseudo-estimate-ratio\":0.8,\"bind-info-lease\":\"3s\",\"txn-entry-size-limit\":6291456,\"txn-total-size-limit\":104857600,\"tcp-keep-alive\":true,\"tcp-no-delay\":true,\"cross-join\":true,\"distinct-agg-push-down\":false,\"projection-push-down\":false,\"max-txn-ttl\":3600000,\"index-usage-sync-lease\":\"0s\",\"plan-replayer-gc-lease\":\"10m\",\"gogc\":100,\"enforce-mpp\":false,\"stats-load-concurrency\":5,\"stats-load-queue-size\":1000,\"analyze-partition-concurrency-quota\":16,\"plan-replayer-dump-worker-concurrency\":1,\"enable-stats-cache-mem-quota\":false,\"committer-concurrency\":128,\"run-auto-analyze\":true,\"force-priority\":\"NO_PRIORITY\",\"memory-usage-alarm-ratio\":0.8,\"enable-load-fmsketch\":false,\"lite-init-stats\":false,\"force-init-stats\":false},\"prepared-plan-cache\":{\"enabled\":true,\"capacity\":100,\"memory-guard-ratio\":0.1},\"opentracing\":{\"enable\":false,\"rpc-metrics\":false,\"sampler\":{\"type\":\"const\",\"param\":1,\"sampling-server-url\":\"\",\"max-operations\":0,\"sampling-refresh-interval\":0},\"reporter\":{\"queue-size\":0,\"buffer-flush-interval\":0,\"log-spans\":false,\"local-agent-host-port\":\"\"}},\"proxy-protocol\":{\"networks\":\"\",\"header-timeout\":5,\"fallbackable\":false},\"pd-client\":{\"pd-server-timeout\":3},\"tikv-client\":{\"grpc-connection-count\":4,\"grpc-keepalive-time\":10,\"grpc-keepalive-timeout\":3,\"grpc-compression-type\":\"none\",\"commit-timeout\":\"41s\",\"async-commit\":{\"keys-limit\":256,\"total-key-size-limit\":4096,\"safe-window\":2000000000,\"allowed-clock-drift\":500000000},\"max-batch-size\":128,\"overload-threshold\":200,\"max-batch-wait-time\":0,\"batch-wait-size\":8,\"enable-chunk-rpc\":true,\"region-cache-ttl\":600,\"store-limit\":0,\"store-liveness-timeout\":\"1s\",\"copr-cache\":{\"capacity-mb\":1000},\"ttl-refreshed-txn-size\":33554432,\"resolve-lock-lite-threshold\":16},\"binlog\":{\"enable\":false,\"ignore-error\":false,\"write-timeout\":\"15s\",\"binlog-socket\":\"\",\"strategy\":\"range\"},\"compatible-kill-query\":false,\"pessimistic-txn\":{\"max-retry-count\":256,\"deadlock-history-capacity\":10,\"deadlock-history-collect-retryable\":false,\"pessimistic-auto-commit\":false,\"constraint-check-in-place-pessimistic\":true},\"max-index-length\":3072,\"index-limit\":64,\"table-column-count-limit\":1017,\"graceful-wait-before-shutdown\":0,\"alter-primary-key\":false,\"treat-old-version-utf8-as-utf8mb4\":true,\"enable-table-lock\":false,\"delay-clean-table-lock\":0,\"split-region-max-num\":1000,\"top-sql\":{\"receiver-address\":\"\"},\"repair-mode\":false,\"repair-table-list\":[],\"isolation-read\":{\"engines\":[\"tikv\",\"tiflash\",\"tidb\"]},\"new_collations_enabled_on_first_bootstrap\":true,\"experimental\":{\"allow-expression-index\":false},\"skip-register-to-dashboard\":false,\"enable-telemetry\":false,\"labels\":{},\"enable-global-index\":false,\"deprecate-integer-display-length\":false,\"enable-enum-length-limit\":true,\"stores-refresh-interval\":60,\"enable-tcp4-only\":false,\"enable-forwarding\":false,\"max-ballast-object-size\":0,\"ballast-object-size\":0,\"transaction-summary\":{\"transaction-summary-capacity\":500,\"transaction-id-digest-min-duration\":2147483647},\"enable-global-kill\":true,\"initialize-sql-file\":\"\",\"enable-batch-dml\":false,\"mem-quota-query\":1073741824,\"oom-action\":\"cancel\",\"oom-use-tmp-storage\":true,\"check-mb4-value-in-utf8\":true,\"enable-collect-execution-info\":true,\"plugin\":{\"dir\":\"/data/deploy/plugin\",\"load\":\"\"},\"max-server-connections\":0,\"run-ddl\":true,\"disaggregated-tiflash\":false,\"autoscaler-type\":\"aws\",\"autoscaler-addr\":\"tiflash-autoscale-lb.tiflash-autoscale.svc.cluster.local:8081\",\"is-tiflashcompute-fixed-pool\":false,\"autoscaler-cluster-id\":\"\",\"use-autoscaler\":false,\"tidb-max-reuse-chunk\":64,\"tidb-max-reuse-column\":256,\"tidb-enable-exit-check\":false}"]
[2023/08/21 16:15:15.207 +08:00] [INFO] [main.go:394] ["disable Prometheus push client"]
[2023/08/21 16:15:15.207 +08:00] [INFO] [store.go:76] ["new store"] [path=tikv://10.223.225.165:2379,10.223.225.166:2379,10.223.225.167:2379]
[2023/08/21 16:15:15.207 +08:00] [INFO] [client.go:311] ["[pd] create pd client with endpoints and keyspace"] [pd-address="[10.223.225.165:2379,10.223.225.166:2379,10.223.225.167:2379]"] [keyspace-id=0]
[2023/08/21 16:15:15.208 +08:00] [INFO] [systime_mon.go:26] ["start system time monitor"]
[2023/08/21 16:15:15.214 +08:00] [INFO] [pd_service_discovery.go:543] ["[pd] switch leader"] [new-leader=http://10.223.225.166:2379] [old-leader=]
[2023/08/21 16:15:15.214 +08:00] [INFO] [pd_service_discovery.go:175] ["[pd] init cluster id"] [cluster-id=7173638068559301829]
[2023/08/21 16:15:15.215 +08:00] [INFO] [client.go:386] ["[pd] changing service mode"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE]
[2023/08/21 16:15:15.215 +08:00] [INFO] [tso_client.go:230] ["[tso] switch dc tso allocator serving address"] [dc-location=global] [new-address=http://10.223.225.166:2379]
[2023/08/21 16:15:15.216 +08:00] [INFO] [tso_dispatcher.go:290] ["[tso] tso dispatcher created"] [dc-location=global]
[2023/08/21 16:15:15.216 +08:00] [INFO] [client.go:428] ["[pd] service mode changed"] [old-mode=PD_SVC_MODE] [new-mode=PD_SVC_MODE]
[2023/08/21 16:15:15.216 +08:00] [INFO] [tikv_driver.go:221] ["using API V1."]
[2023/08/21 16:15:15.217 +08:00] [INFO] [store.go:82] ["new store with retry success"]
[2023/08/21 16:15:15.228 +08:00] [INFO] [tidb.go:80] ["new domain"] [store=tikv-7173638068559301829] ["ddl lease"=45s] ["stats lease"=3s] ["index usage sync lease"=0s]
[2023/08/21 16:15:15.240 +08:00] [INFO] [domain.go:2650] [acquireServerID] [serverID=3117095] ["lease id"=77568a165d6aff79]
[2023/08/21 16:15:15.257 +08:00] [WARN] [info.go:304] ["init TiFlashReplicaManager"] ["pd addrs"="[10.223.225.165:2379,10.223.225.167:2379,10.223.225.166:2379]"]
[2023/08/21 16:15:15.442 +08:00] [INFO] [domain.go:272] ["full load InfoSchema success"] [currentSchemaVersion=0] [neededSchemaVersion=3140] ["start time"=170.140027ms]
[2023/08/21 16:15:15.445 +08:00] [INFO] [domain.go:587] ["full load and reset schema validator"]
[2023/08/21 16:15:15.445 +08:00] [INFO] [ddl.go:752] ["[ddl] start DDL"] [ID=45e84fb0-d912-4865-b29e-5dc3a04b4a7b] [runWorker=true]
[2023/08/21 16:15:15.445 +08:00] [INFO] [ddl.go:715] ["[ddl] start delRangeManager OK"] ["is a emulator"=false]
[2023/08/21 16:15:15.450 +08:00] [INFO] [manager.go:178] ["start campaign owner"] [ownerInfo="[ddl] /tidb/ddl/fg/owner"]
[2023/08/21 16:15:15.452 +08:00] [INFO] [env.go:90] ["[ddl-ingest] the ingest sorted directory"] ["data path:"=/tmp/tidb/tmp_ddl-4000]
[2023/08/21 16:15:15.452 +08:00] [WARN] [backend_mgr.go:59] ["[ddl-ingest] ingest backfill may not be available"] [error="the available disk space(33598537728) in /tmp/tidb/tmp_ddl-4000 should be greater than @@tidb_ddl_disk_quota(107374182400)"]
[2023/08/21 16:15:15.452 +08:00] [INFO] [env.go:68] ["[ddl-ingest] init global ingest backend environment finished"] ["memory limitation"=2147483648] ["disk usage info"="disk usage: 1114112/33599651840, backend usage: 0"] ["max open file number"=1000000] ["lightning is initialized"=true]
[2023/08/21 16:15:15.452 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=loadSchemaInLoop]
[2023/08/21 16:15:15.452 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=mdlCheckLoop]
[2023/08/21 16:15:15.452 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=topNSlowQueryLoop]
[2023/08/21 16:15:15.452 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=infoSyncerKeeper]
[2023/08/21 16:15:15.452 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=globalConfigSyncerKeeper]
[2023/08/21 16:15:15.452 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=topologySyncerKeeper]
[2023/08/21 16:15:15.452 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=closestReplicaReadCheckLoop]
[2023/08/21 16:15:15.452 +08:00] [INFO] [job_table.go:328] ["[ddl] get global state and global state change"] [oldState=false] [currState=false]
[2023/08/21 16:15:15.452 +08:00] [INFO] [owner_daemon.go:70] ["begin advancer daemon"] [daemon-id=LogBackup::Advancer]
[2023/08/21 16:15:15.452 +08:00] [INFO] [manager.go:178] ["start campaign owner"] [ownerInfo="[log-backup] /tidb/br-stream/owner"]
[2023/08/21 16:15:15.462 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=logBackupAdvancer]
[2023/08/21 16:15:15.462 +08:00] [INFO] [owner_daemon.go:81] ["begin running daemon"] [id=7c9c8812-b598-4b5e-8925-f4b66456b23f] [daemon-id=LogBackup::Advancer]
[2023/08/21 16:15:15.473 +08:00] [INFO] [manager.go:178] ["start campaign owner"] [ownerInfo="[bindinfo] /tidb/bindinfo/owner"]
[2023/08/21 16:15:15.476 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=globalBindHandleWorkerLoop]
[2023/08/21 16:15:15.476 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=handleEvolvePlanTasksLoop]
[2023/08/21 16:15:15.476 +08:00] [WARN] [sysvar_cache.go:50] ["sysvar cache is empty, triggering rebuild"]
[2023/08/21 16:15:15.479 +08:00] [INFO] [sysvar.go:2432] ["set resource control"] [enable=true]
[2023/08/21 16:15:15.611 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=loadPrivilegeInLoop]
[2023/08/21 16:15:15.615 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=LoadSysVarCacheLoop]
[2023/08/21 16:15:15.618 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=PlanReplayerTaskCollectHandle]
[2023/08/21 16:15:15.618 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=PlanReplayerTaskDumpHandle]
[2023/08/21 16:15:15.618 +08:00] [INFO] [domain.go:1946] ["PlanReplayerTaskCollectHandle started"]
[2023/08/21 16:15:15.618 +08:00] [INFO] [domain.go:1968] ["PlanReplayerTaskDumpHandle started"]
[2023/08/21 16:15:15.618 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=dumpFileGcChecker]
[2023/08/21 16:15:15.618 +08:00] [INFO] [plan_replayer.go:381] ["planReplayerTaskDumpWorker started."]
[2023/08/21 16:15:15.618 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=HistoricalStatsWorker]
[2023/08/21 16:15:15.618 +08:00] [INFO] [domain.go:2000] ["dumpFileGcChecker started"]
[2023/08/21 16:15:15.618 +08:00] [INFO] [domain.go:2033] ["HistoricalStatsWorker started"]
[2023/08/21 16:15:15.618 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=loadStatsWorker]
[2023/08/21 16:15:15.618 +08:00] [INFO] [manager.go:178] ["start campaign owner"] [ownerInfo="[stats] /tidb/stats/owner"]
[2023/08/21 16:15:15.622 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=updateStatsWorker]
[2023/08/21 16:15:15.622 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=autoAnalyzeWorker]
[2023/08/21 16:15:15.622 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=gcAnalyzeHistory]
[2023/08/21 16:15:15.622 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=ttlJobManager]
[2023/08/21 16:15:15.622 +08:00] [INFO] [domain.go:2303] ["updateStatsWorker started."]
[2023/08/21 16:15:15.623 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=loadSigningCertLoop]
[2023/08/21 16:15:15.623 +08:00] [INFO] [gc_worker.go:212] ["[gc worker] start"] [uuid=6285ca652280009]
[2023/08/21 16:15:15.623 +08:00] [INFO] [task_manager.go:215] ["scale ttl worker"] [ttl-worker=job-manager] [ttl-worker=task-manager] [originalCount=0] [newCount=4]
[2023/08/21 16:15:15.623 +08:00] [INFO] [task_manager.go:215] ["scale ttl worker"] [ttl-worker=job-manager] [ttl-worker=task-manager] [originalCount=0] [newCount=4]
[2023/08/21 16:15:15.628 +08:00] [INFO] [wait_group_wrapper.go:130] ["background process started"] [source=domain] [process=distTaskFrameworkLoop]
[2023/08/21 16:15:15.628 +08:00] [WARN] [misc.go:458] ["Automatic TLS Certificate creation is disabled"] []
[2023/08/21 16:15:15.629 +08:00] [INFO] [domain.go:1405] ["dist task scheduler started"]
[2023/08/21 16:15:15.629 +08:00] [INFO] [server.go:271] ["server is running MySQL protocol"] [addr=0.0.0.0:4000]
[2023/08/21 16:15:15.629 +08:00] [INFO] [server.go:285] ["server is running MySQL protocol"] [socket=/tmp/tidb-4000.sock]
[2023/08/21 16:15:15.629 +08:00] [INFO] [http_status.go:90] ["for status and metrics report"] ["listening on addr"=0.0.0.0:10080]
[2023/08/21 16:15:15.630 +08:00] [INFO] [cpu.go:83] ["sql cpu collector started"]
[2023/08/21 16:15:15.632 +08:00] [INFO] [store.go:76] ["new store"] [path=tikv://10.223.225.165:2379,10.223.225.166:2379,10.223.225.167:2379]
[2023/08/21 16:15:15.632 +08:00] [INFO] [client.go:311] ["[pd] create pd client with endpoints and keyspace"] [pd-address="[10.223.225.165:2379,10.223.225.166:2379,10.223.225.167:2379]"] [keyspace-id=0]
[2023/08/21 16:15:15.637 +08:00] [INFO] [pd_service_discovery.go:543] ["[pd] switch leader"] [new-leader=http://10.223.225.166:2379] [old-leader=]
[2023/08/21 16:15:15.637 +08:00] [INFO] [pd_service_discovery.go:175] ["[pd] init cluster id"] [cluster-id=7173638068559301829]
[2023/08/21 16:15:15.637 +08:00] [INFO] [client.go:386] ["[pd] changing service mode"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE]
[2023/08/21 16:15:15.637 +08:00] [INFO] [tso_client.go:230] ["[tso] switch dc tso allocator serving address"] [dc-location=global] [new-address=http://10.223.225.166:2379]
[2023/08/21 16:15:15.638 +08:00] [INFO] [tso_dispatcher.go:290] ["[tso] tso dispatcher created"] [dc-location=global]
[2023/08/21 16:15:15.638 +08:00] [INFO] [client.go:428] ["[pd] service mode changed"] [old-mode=PD_SVC_MODE] [new-mode=PD_SVC_MODE]
[2023/08/21 16:15:15.638 +08:00] [ERROR] [tso_dispatcher.go:514] ["[tso] update connection contexts failed"] [dc=global] [error="rpc error: code = Canceled desc = context canceled"]
[2023/08/21 16:15:15.638 +08:00] [INFO] [tso_dispatcher.go:214] ["exit tso dispatcher loop"]
[2023/08/21 16:15:15.638 +08:00] [INFO] [tso_dispatcher.go:311] ["[tso] exit tso dispatcher"] [dc-location=global]
[2023/08/21 16:15:15.638 +08:00] [WARN] [resource_manager_client.go:295] ["[resource_manager] get token stream error"] [error="rpc error: code = Canceled desc = context canceled"]
[2023/08/21 16:15:15.638 +08:00] [INFO] [tso_client.go:134] ["closing tso client"]
[2023/08/21 16:15:15.638 +08:00] [INFO] [tso_dispatcher.go:162] ["exit tso requests cancel loop"]
[2023/08/21 16:15:15.638 +08:00] [INFO] [tso_client.go:139] ["close tso client"]
[2023/08/21 16:15:15.638 +08:00] [INFO] [tso_client.go:150] ["tso client is closed"]
[2023/08/21 16:15:15.638 +08:00] [INFO] [pd_service_discovery.go:250] ["[pd] close pd service discovery client"]
[2023/08/21 16:15:15.639 +08:00] [INFO] [store.go:82] ["new store with retry success"]
[2023/08/21 16:15:15.639 +08:00] [INFO] [manager.go:178] ["start campaign owner"] [ownerInfo="[autoid] tidb/autoid/leader"]
[2023/08/21 16:15:15.643 +08:00] [INFO] [http_status.go:492] ["register auto service at"] [addr=10.223.225.84:10080]
[2023/08/21 16:15:16.510 +08:00] [INFO] [domain.go:2208] ["init stats info time"] [lite=false] ["take time"=891.5183ms]
[2023/08/21 16:15:17.452 +08:00] [INFO] [info.go:1189] [SetTiFlashGroupConfig]
[2023/08/21 16:15:46.184 +08:00] [INFO] [cpuprofile.go:113] ["parallel cpu profiler started"]
[2023/08/21 16:15:46.185 +08:00] [INFO] [printer.go:48] ["Welcome to TiDB."] ["Release Version"=v7.1.1] [Edition=Community] ["Git Commit Hash"=cf441574864be63938524e7dfcf7cc659edc3dd8] ["Git Branch"=heads/refs/tags/v7.1.1] ["UTC Build Time"="2023-07-19 10:16:40"] [GoVersion=go1.20.6] ["Race Enabled"=false] ["Check Table Before Drop"=false] ["TiKV Min Version"=6.2.0-alpha]

1 个赞

请描述清楚集群的硬件配置信息,最好提供相关的tiup 集群信息来帮助判断


[2023/08/21 16:08:32.744 +08:00] [WARN] [resource_manager_client.go:295] [“[resource_manager] get token stream error”] [error=“rpc error: code = Canceled desc = context canceled”]
[2023/08/21 16:08:32.744 +08:00] [ERROR] [tso_dispatcher.go:514] [“[tso] update connection contexts failed”] [dc=global] [error=“rpc error: code = Canceled desc = context canceled”]

从这两条日志基本上可以判断集群的 PD 都没正常工作…

1 个赞

pd 的 日志看看,还有磁盘分配也不合适

[2023/08/21 16:15:15.452 +08:00] [WARN] [backend_mgr.go:59] [“[ddl-ingest] ingest backfill may not be available”] [error=“the available disk space(33598537728) in /tmp/tidb/tmp_ddl-4000 should be greater than @@tidb_ddl_disk_quota(107374182400)”]

1 个赞

tiup cluster display tidb-xxxx看一下集群状态。。。

集群的配置,组件分布情况说说

我们这边6.5.2版本也出现过这种问题,月初大量复杂报表并行查询,TiDB Server节点OOM了。现在正在从业务层做限流,将可以异步的导出任务做排队处理。

1 个赞

TIDB Server节点存储空间不足

1、你们的TiDB Server配置是什么样的?迁移后的TiDB版本一致吗?OOM相关参数是如何设置有差别吗,特别是这些参数:
tidb_server_memory_limit 、tidb_mem_oom_action、tidb_mem_quota_query
2、10分钟重启1次,业务层有没有定时任务?
3、 DashBoard上的慢查询中”最大内存“使用的SQL及其执行频率如何?