升级v2.1.9至v3.0.5出错,日志报 [fatal] [schema:1146]Table 'mysql.tidb' doesn't exist

升级v2.1.9至v3.0.5出错,日志报 [fatal] [schema:1146]Table ‘mysql.tidb’ doesn’t exist

TiKV,PD节点已经升级成功,但是在启动TiDB节点的时候报错误,是测试环境,只有一个TiDB节点,有没有办法挽救,测试数据还有用的。

thub.com/pingcap/tidb@/session/bootstrap.go:409 github.com/pingcap/tidb/session.runInBootstrapSession github.com/pingcap/tidb@/session/session.go:1557 github.com/pingcap/tidb/session.BootstrapSession github.com/pingcap/tidb@/session/session.go:1473 main.createStoreAndDomain github.com/pingcap/tidb@/tidb-server/main.go:205 main.main github.com/pingcap/tidb@/tidb-server/main.go:171 runtime.main runtime/proc.go:203 runtime.goexit runtime/asm_amd64.s:1357"] [sql=“SELECT HIGH_PRIORITY VARIABLE_VALUE FROM mysql.tidb WHERE VARIABLE_NAME=“tidb_server_version””] 2019/11/01 16:33:08.860 terror.go:342: [fatal] [schema:1146]Table ‘mysql.tidb’ doesn’t exist github.com/pingcap/errors.AddStack gtp2github.com/pingcap/errors@v0.11.4/errors.go:174 github.com/pingcap/parser/terror.(*Error).GenWithStackByArgs gtp2github.com/pingcap/parser@v0.0.0-20191018040038-555b97093a2a/terror/terror.go:238 github.com/pingcap/tidb/infoschema.(*infoSchema).TableByName gtp2github.com/pingcap/tidb@/infoschema/infoschema.go:181 github.com/pingcap/tidb/planner/core.(*preprocessor).handleTableName gtp2github.com/pingcap/tidb@/planner/core/preprocess.go:710 github.com/pingcap/tidb/planner/core.(*preprocessor).Leave gtp2github.com/pingcap/tidb@/planner/core/preprocess.go:170 github.com/pingcap/parser/ast.(*TableName).Accept gtp2github.com/pingcap/parser@v0.0.0-20191018040038-555b97093a2a/ast/dml.go:295 github.com/pingcap/parser/ast.(*TableSource).Accept gtp2github.com/pingcap/parser@v0.0.0-20191018040038-555b97093a2a/ast/dml.go:410 github.com/pingcap/parser/ast.(*Join).Accept gtp2github.com/pingcap/parser@v0.0.0-20191018040038-555b97093a2a/ast/dml.go:154 github.com/pingcap/parser/ast.(*TableRefsClause).Accept gtp2github.com/pingcap/parser@v0.0.0-20191018040038-555b97093a2a/ast/dml.go:588 github.com/pingcap/parser/ast.(*SelectStmt).Accept gtp2github.com/pingcap/parser@v0.0.0-20191018040038-555b97093a2a/ast/dml.go:912 github.com/pingcap/tidb/planner/core.Preprocess gtp2github.com/pingcap/tidb@/planner/core/preprocess.go:54 github.

可不可以重建TiDB节点?

能详细描述一下整个升级过程的步骤吗?

完全按下面的文档操作的

麻烦确认下当前的 PD 以及 TiKV 的版本。

怎么看?

确认下每个节点上面 {deploy_path}/bin/tikv-server -V , {deploy_path}/bin/pd-server -V , {ansible_path}/resource/bin/pd-ctl config show cluster-version -u http://{pd_ip}:{pd_port}

[tidb@gtp1 bin]$ ./tikv-server -V TiKV Release Version: 3.0.5 Git Commit Hash: 01c872bf105dc68dda346ceda087e994c56e2702 Git Commit Branch: HEAD UTC Build Time: 2019-10-25 01:03:08 Rust Version: rustc 1.37.0-nightly (0e4a56b4b 2019-06-13)

[tidb@gtp2 bin]$ pd-server -V bash: pd-server: command not found [tidb@gtp2 bin]$ ./pd-server -V Release Version: v3.0.5 Git Commit Hash: 4934a651d2d428411a9610a5b6b9d7156a439355 Git Branch: HEAD UTC Build Time: 2019-10-25 03:24:57

[tidb@gtp2 bin]$ ./tidb-server -V Release Version: v3.0.5 Git Commit Hash: c9000abdc216b6a02efbcc578af8be1f98ba280d Git Branch: HEAD UTC Build Time: 2019-10-25 03:25:32 GoVersion: go version go1.13 linux/amd64 Race Enabled: false TiKV Min Version: v3.0.0-60965b006877ca7234adaced7890d7b029ed1306 Check Table Before Drop: false

都是3.0.5,是不是PD节点数据损坏了?

PD的数据好像还在

  • total 9968
  • -rw-r–r-- 1 tidb tidb 11075 Oct 17 18:15 000000000000000d-0000000000200b35.snap
  • -rw-r–r-- 1 tidb tidb 11075 Oct 21 05:05 000000000000000d-00000000002191d6.snap
  • -rw-r–r-- 1 tidb tidb 11075 Oct 24 14:19 000000000000000d-0000000000231877.snap
  • -rw-r–r-- 1 tidb tidb 11075 Oct 27 22:36 000000000000000d-0000000000249f18.snap
  • -rw-r–r-- 1 tidb tidb 11075 Oct 31 05:09 000000000000000d-00000000002625b9.snap
  • -rwxrwxrwx 1 tidb tidb 10141696 Nov 1 17:40 db
  • [tidb@gtp2 snap]$

{ansible_path}/resource/bin/pd-ctl config show cluster-version -u http://{pd_ip}:{pd_port} 这个执行一下。

确认下 TiKV TiDB 相关的 deploy_path 的目录对应的权限。另外麻烦给下 TiDB 的 log

权限都查过没有问题,看了一下升级之前的Log,貌似已经发生错误了,全是

2019/11/01 13:43:03.864 terror.go:344: [error] lookup 47.100.23.254: Name or service not known
github.com/pingcap/errors.AddStack
        /home/jenkins/workspace/release_tidb_2.1-ga/go/pkg/mod/github.com/pingcap/errors@v0.11.1/errors.go:174
github.com/pingcap/errors.Trace
        /home/jenkins/workspace/release_tidb_2.1-ga/go/pkg/mod/github.com/pingcap/errors@v0.11.1/juju_adaptor.go:15
github.com/pingcap/tidb/session.getHostByIP
        /home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/session/session.go:1134
github.com/pingcap/tidb/session.(*session).Auth
        /home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/session/session.go:1114
github.com/pingcap/tidb/server.(*TiDBContext).Auth
        /home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/server/driver_tidb.go:274
github.com/pingcap/tidb/server.(*clientConn).openSessionAndDoAuth
        /home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/server/conn.go:414
github.com/pingcap/tidb/server.(*clientConn).readOptionalSSLRequestAndHandshakeResponse
        /home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/server/conn.go:395
github.com/pingcap/tidb/server.(*clientConn).handshake
        /home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/server/conn.go:133
github.com/pingcap/tidb/server.(*Server).onConn
        /home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/server/server.go:353
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1337

然后再15:49,w我开始升级的,就持续报 Table ‘mysql.tidb’ doesn’t exist

[2019/11/01 15:49:11.169 +08:00] [INFO] [main.go:240] ["disable Prometheus push client"]
[2019/11/01 15:49:11.169 +08:00] [INFO] [tidb.go:311] ["new store"]
[2019/11/01 15:49:11.169 +08:00] [INFO] [systime_mon.go:26] ["start system time monitor"]
2019/11/01 15:49:11.169 client.go:124: [info] [pd] create pd client with endpoints [192.168.3.62:2379]
2019/11/01 15:49:11.171 client.go:226: [info] [pd] leader switches to: http://192.168.3.62:2379, previous:
2019/11/01 15:49:11.171 client.go:143: [info] [pd] init cluster id 6601270415411399747
[2019/11/01 15:49:11.178 +08:00] [INFO] [tidb.go:64] ["new domain"] [store=tikv-6601270415411399747] ["ddl lease"=45s] ["stats lease"=3s]
[2019/11/01 15:49:11.178 +08:00] [INFO] [ddl.go:369] ["[ddl] start DDL"] [ID=0f32ebce-63b0-4a5e-aca8-c8754e25b62a] [runWorker=true]
[2019/11/01 15:49:11.179 +08:00] [INFO] [ddl_worker.go:84] ["[ddl] start delRangeManager OK"] ["is a emulator"=false]
[2019/11/01 15:49:11.179 +08:00] [INFO] [ddl_worker.go:84] ["[ddl] start delRangeManager OK"] ["is a emulator"=false]
[2019/11/01 15:49:11.179 +08:00] [INFO] [ddl_worker.go:120] ["[ddl] start DDL worker"] [worker="worker 1, tp general"]
[2019/11/01 15:49:11.179 +08:00] [INFO] [ddl_worker.go:120] ["[ddl] start DDL worker"] [worker="worker 2, tp add index"]
[2019/11/01 15:49:12.929 +08:00] [INFO] [domain.go:131] ["full load InfoSchema success"] [usedSchemaVersion=0] [latestSchemaVersion=34655] ["start time"=1.739547838s]
[2019/11/01 15:49:12.930 +08:00] [INFO] [domain.go:332] ["full load and reset schema validator"]
[2019/11/01 15:49:12.934 +08:00] [WARN] [session.go:875] ["compile sql error"] [error="[schema:1146]Table 'mysql.tidb' doesn't exist"] [errorVerbose="[schema:1146]Table 'mysql.tidb' doesn't exist
github.com/pingcap/errors.AddStack
	/home/jenkins/workspace/release_tidb_2.1-ga/go/pkg/mod/github.com/pingcap/errors@v0.11.1/errors.go:174
github.com/pingcap/parser/terror.(*Error).GenWithStackByArgs
	/home/jenkins/workspace/release_tidb_2.1-ga/go/pkg/mod/github.com/pingcap/parser@v0.0.0-20190505094039-595d728571a7/terror/terror.go:233
github.com/pingcap/tidb/infoschema.(*infoSchema).TableByName
	/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/infoschema/infoschema.go:177
github.com/pingcap/tidb/planner/core.(*preprocessor).handleTableName
	/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/planner/core/preprocess.go:584
github.com/pingcap/tidb/planner/core.(*preprocessor).Leave
	/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/planner/core/preprocess.go:118
github.com/pingcap/parser/ast.(*TableName).Accept
	/home/jenkins/workspace/release_tidb_2.1-ga/go/pkg/mod/github.com/pingcap/parser@v0.0.0-20190505094039-595d728571a7/ast/dml.go:163
github.com/pingcap/parser/ast.(*TableSource).Accept
	/home/jenkins/workspace/release_tidb_2.1-ga/go/pkg/mod/github.com/pingcap/parser@v0.0.0-20190505094039-595d728571a7/ast/dml.go:232
github.com/pingcap/parser/ast.(*Join).Accept
	/home/jenkins/workspace/release_tidb_2.1-ga/go/pkg/mod/github.com/pingcap/parser@v0.0.0-20190505094039-595d728571a7/ast/dml.go:92
github.com/pingcap/parser/ast.(*TableRefsClause).Accept
	/home/jenkins/workspace/release_tidb_2.1-ga/go/pkg/mod/github.com/pingcap/parser@v0.0.0-20190505094039-595d728571a7/ast/dml.go:356
github.com/pingcap/parser/ast.(*SelectStmt).Accept
	/home/jenkins/workspace/release_tidb_2.1-ga/go/pkg/mod/github.com/pingcap/parser@v0.0.0-20190505094039-595d728571a7/ast/dml.go:512
github.com/pingcap/tidb/planner/core.Preprocess
	/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/planner/core/preprocess.go:35
github.com/pingcap/tidb/executor.(*Compiler).Compile
	/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/executor/compiler.go:45
github.com/pingcap/tidb/session.(*session).execute
	/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/session/session.go:872
github.com/pingcap/tidb/session.(*session).Execute
	/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/session/session.go:832
github.com/pingcap/tidb/session.loadSystemTZ
	/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/session/session.go:1184
github.com/pingcap/tidb/session.BootstrapSession
	/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/session/session.go:1230
main.createStoreAndDomain
	/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/tidb-server/main.go:195
main.main
	/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/tidb-server/main.go:161
runtime.main
	/usr/local/go/src/runtime/proc.go:200
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1337"] [sql="select variable_value from mysql.tidb where variable_name = 'system_tz'"]
2019/11/01 15:49:12.935 terror.go:329: [fatal] [schema:1146]Table 'mysql.tidb' doesn't exist

还有没有救了?

麻烦直接发下升级前后的 TiDB 日志文件看下。