为提高效率,提问时请提供以下信息,问题描述清晰可优先响应。
-
【TiDB 版本】:v3.0.12
-
【问题描述】: 各位大神好,情况如下所示 1,一开始使用的是v2.1.12,后来发现tidb-server日志刷以下内容
[2020/03/19 14:39:42.695 +08:00] [INFO] [printer.go:41] [“Welcome to TiDB.”] [“Release Version”=v3.0.12] [“Git Commit Hash”=8c4696b3f3408c61dd7454204ddd67c93501609a] [“Git Branch”=heads/refs/tags/v3.0.12] [“UTC Build Time”=“2020-03-16 09:56:22”] [GoVersion=“go version go1.13 linux/amd64”] [“Race Enabled”=false] [“Check Table Before Drop”=false] [“TiKV Min Version”=v3.0.0-60965b006877ca7234adaced7890d7b029ed1306] [2020/03/19 14:39:42.696 +08:00] [INFO] [printer.go:54] [“loaded config”] [config="{“host”:“0.0.0.0”,“advertise-address”:“10.203.20.78”,“port”:4000,“cors”:"",“store”:“tikv”,“path”:“10.203.20.78:2379,10.203.20.79:2379,10.203.20.80:2379”,“socket”:"",“lease”:“45s”,“run-ddl”:true,“split-table”:true,“token-limit”:1000,“oom-action”:“log”,“mem-quota-query”:34359738368,“enable-streaming”:false,“enable-batch-dml”:false,“txn-local-latches”:{“enabled”:false,“capacity”:2048000},“lower-case-table-names”:2,“server-version”:"",“log”:{“level”:“info”,“format”:“text”,“disable-timestamp”:false,“file”:{“filename”:"",“log-rotate”:true,“max-size”:300,“max-days”:0,“max-backups”:0},“slow-query-file”:“tidb-slow.log”,“slow-threshold”:300,“expensive-threshold”:10000,“query-log-max-len”:2048,“record-plan-in-slow-log”:1},“security”:{“skip-grant-table”:false,“ssl-ca”:"",“ssl-cert”:"",“ssl-key”:"",“cluster-ssl-ca”:"",“cluster-ssl-cert”:"",“cluster-ssl-key”:"",“cluster-verify-cn”:null},“status”:{“report-status”:true,“status-host”:“0.0.0.0”,“status-port”:10080,“metrics-addr”:"",“metrics-interval”:15,“record-db-qps”:false},“performance”:{“max-procs”:0,“max-memory”:0,“tcp-keep-alive”:true,“cross-join”:true,“stats-lease”:“3s”,“run-auto-analyze”:true,“stmt-count-limit”:5000,“feedback-probability”:0.05,“query-feedback-limit”:1024,“pseudo-estimate-ratio”:0.8,“force-priority”:“NO_PRIORITY”,“bind-info-lease”:“3s”,“txn-entry-count-limit”:300000,“txn-total-size-limit”:104857600},“prepared-plan-cache”:{“enabled”:false,“capacity”:100,“memory-guard-ratio”:0.1},“opentracing”:{“enable”:false,“sampler”:{“type”:“const”,“param”:1,“sampling-server-url”:"",“max-operations”:0,“sampling-refresh-interval”:0},“reporter”:{“queue-size”:0,“buffer-flush-interval”:0,“log-spans”:false,“local-agent-host-port”:""},“rpc-metrics”:false},“proxy-protocol”:{“networks”:"",“header-timeout”:5},“tikv-client”:{“grpc-connection-count”:16,“grpc-keepalive-time”:10,“grpc-keepalive-timeout”:3,“commit-timeout”:“41s”,“max-txn-time-use”:590,“max-batch-size”:128,“overload-threshold”:200,“max-batch-wait-time”:0,“batch-wait-size”:8,“region-cache-ttl”:600,“store-limit”:0},“binlog”:{“enable”:false,“write-timeout”:“15s”,“ignore-error”:false,“binlog-socket”:"",“strategy”:“range”},“compatible-kill-query”:false,“plugin”:{“dir”:"",“load”:""},“pessimistic-txn”:{“enable”:true,“max-retry-count”:256},“check-mb4-value-in-utf8”:true,“max-index-length”:3072,“alter-primary-key”:false,“treat-old-version-utf8-as-utf8mb4”:true,“split-region-max-num”:1000,“stmt-summary”:{“enable”:false,“max-stmt-count”:200,“max-sql-length”:4096,“refresh-interval”:1800,“history-size”:24}}"] [2020/03/19 14:39:42.696 +08:00] [INFO] [main.go:255] [“disable Prometheus push client”] [2020/03/19 14:39:42.696 +08:00] [INFO] [store.go:69] [“new store”] [path=tikv://10.203.20.78:2379,10.203.20.79:2379,10.203.20.80:2379] [2020/03/19 14:39:42.696 +08:00] [INFO] [client.go:162] ["[pd] create pd client with endpoints"] [pd-address="[10.203.20.78:2379,10.203.20.79:2379,10.203.20.80:2379]"] [2020/03/19 14:39:42.696 +08:00] [INFO] [systime_mon.go:26] [“start system time monitor”] [2020/03/19 14:39:42.699 +08:00] [INFO] [client.go:291] ["[pd] switch leader"] [new-leader=http://10.203.20.78:2379] [old-leader=] [2020/03/19 14:39:42.699 +08:00] [INFO] [client.go:186] ["[pd] init cluster id"] [cluster-id=6780342731695471745] [2020/03/19 14:39:42.700 +08:00] [INFO] [store.go:75] [“new store with retry success”] [2020/03/19 14:40:03.370 +08:00] [WARN] [backoff.go:305] [“pdRPC backoffer.maxSleep 20000ms is exceeded, errors:\ loadRegion from PD failed, key: “mBootstra\xffpKey\x00\x00\x00\x00\xfb\x00\x00\x00\x00\x00\x00\x00s”, err: invalid marker byte, group bytes “P_1832258” at 2020-03-19T14:39:58.785522324+08:00\ loadRegion from PD failed, key: “mBootstra\xffpKey\x00\x00\x00\x00\xfb\x00\x00\x00\x00\x00\x00\x00s”, err: invalid marker byte, group bytes “P_1832258” at 2020-03-19T14:40:00.486149711+08:00\ loadRegion from PD failed, key: “mBootstra\xffpKey\x00\x00\x00\x00\xfb\x00\x00\x00\x00\x00\x00\x00s”, err: invalid marker byte, group bytes “P_1832258” at 2020-03-19T14:40:03.370740153+08:00”] [2020/03/19 14:40:03.370 +08:00] [FATAL] [session.go:1663] [“check bootstrapped failed”] [error="[tikv:9001]PD server timeout"] [errorVerbose="[tikv:9001]PD server timeout\ngithub.com/pingcap/errors.AddStack\ \t/home/jenkins/agent/workspace/tidb_v3.0.12/go/pkg/mod/github.com/pingcap/errors@v0.11.4/errors.go:174\ github.com/pingcap/errors.Trace\ \t/home/jenkins/agent/workspace/tidb_v3.0.12/go/pkg/mod/github.com/pingcap/errors@v0.11.4/juju_adaptor.go:15\ github.com/pingcap/tidb/store/tikv.(*RegionCache).loadRegion\ \t/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/store/tikv/region_cache.go:612\ngithub.com/pingcap/tidb/store/tikv.(*RegionCache).findRegionByKey\ \t/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/store/tikv/region_cache.go:366\ngithub.com/pingcap/tidb/store/tikv.(*RegionCache).LocateKey\ \t/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/store/tikv/region_cache.go:326\ngithub.com/pingcap/tidb/store/tikv.(*tikvSnapshot).get\ \t/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/store/tikv/snapshot.go:306\ngithub.com/pingcap/tidb/store/tikv.(*tikvSnapshot).Get\ \t/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/store/tikv/snapshot.go:265\ngithub.com/pingcap/tidb/kv.(*unionStore).Get\ \t/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/kv/union_store.go:194\ngithub.com/pingcap/tidb/store/tikv.(*tikvTxn).Get\ \t/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/store/tikv/txn.go:135\ngithub.com/pingcap/tidb/structure.(*TxStructure).Get\ \t/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/structure/string.go:35\ngithub.com/pingcap/tidb/structure.(*TxStructure).GetInt64\ \t/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/structure/string.go:44\ngithub.com/pingcap/tidb/meta.(*Meta).GetBootstrapVersion\ \t/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/meta/meta.go:714\ngithub.com/pingcap/tidb/session.getStoreBootstrapVersion.func1\ \t/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/session/session.go:1658\ github.com/pingcap/tidb/kv.RunInNewTxn\ \t/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/kv/txn.go:50\ github.com/pingcap/tidb/session.getStoreBootstrapVersion\ \t/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/session/session.go:1655\ github.com/pingcap/tidb/session.BootstrapSession\ \t/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/session/session.go:1496\ main.createStoreAndDomain\ \t/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/tidb-server/main.go:210\ main.main\ \t/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/tidb-server/main.go:172\ runtime.main\ \t/usr/local/go/src/runtime/proc.go:203\ runtime.goexit\ \t/usr/local/go/src/runtime/asm_amd64.s:1357"] [stack=“github.com/pingcap/tidb/session.getStoreBootstrapVersion\ \t/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/session/session.go:1663\ github.com/pingcap/tidb/session.BootstrapSession\ \t/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/session/session.go:1496\ main.createStoreAndDomain\ \t/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/tidb-server/main.go:210\ main.main\ \t/home/jenkins/agent/workspace/tidb_v3.0.12/go/src/github.com/pingcap/tidb/tidb-server/main.go:172\ runtime.main\ \t/usr/local/go/src/runtime/proc.go:203”]
2,当时重启了tidb-server就再也起不来了 ,后来把pd和tikv数据清理了,又可以启动了,隔了不久,系统还是刷以上日志,以为是版本问题,所以把版本升级到了3.0.12,pd和tikv启动正常,通过命令查看pd状态正常
3,询问开发存放数据方式,开发反馈是通过tikvrpc(开发存放的数据库表无法通过4000端口查看)
若提问为性能优化、故障排查类问题,请下载脚本运行。终端输出的打印结果,请务必全选并复制粘贴上传。