升级v2.1.9至v3.0.5出错，日志报 [fatal] [schema:1146]Table 'mysql.tidb' doesn't exist

jim.ni · 2019 年11 月 1 日 08:46

升级v2.1.9至v3.0.5出错，日志报 [fatal] [schema:1146]Table ‘mysql.tidb’ doesn’t exist

TiKV,PD节点已经升级成功，但是在启动TiDB节点的时候报错误，是测试环境，只有一个TiDB节点，有没有办法挽救，测试数据还有用的。

jim.ni · 2019 年11 月 1 日 08:51

可不可以重建TiDB节点？

QBin · 2019 年11 月 1 日 09:00

能详细描述一下整个升级过程的步骤吗？

jim.ni · 2019 年11 月 1 日 09:06

完全按下面的文档操作的
https://pingcap.com/docs-cn/v3.0/how-to/upgrade/from-previous-version/

QBin · 2019 年11 月 1 日 09:16

麻烦确认下当前的 PD 以及 TiKV 的版本。

jim.ni · 2019 年11 月 1 日 09:18

怎么看？

QBin · 2019 年11 月 1 日 09:28

确认下每个节点上面 {deploy_path}/bin/tikv-server -V , {deploy_path}/bin/pd-server -V , {ansible_path}/resource/bin/pd-ctl config show cluster-version -u http://{pd_ip}:{pd_port}

jim.ni · 2019 年11 月 1 日 09:34

[tidb@gtp1 bin]$ ./tikv-server -V TiKV Release Version: 3.0.5 Git Commit Hash: 01c872bf105dc68dda346ceda087e994c56e2702 Git Commit Branch: HEAD UTC Build Time: 2019-10-25 01:03:08 Rust Version: rustc 1.37.0-nightly (0e4a56b4b 2019-06-13)

[tidb@gtp2 bin]$ pd-server -V bash: pd-server: command not found [tidb@gtp2 bin]$ ./pd-server -V Release Version: v3.0.5 Git Commit Hash: 4934a651d2d428411a9610a5b6b9d7156a439355 Git Branch: HEAD UTC Build Time: 2019-10-25 03:24:57

[tidb@gtp2 bin]$ ./tidb-server -V Release Version: v3.0.5 Git Commit Hash: c9000abdc216b6a02efbcc578af8be1f98ba280d Git Branch: HEAD UTC Build Time: 2019-10-25 03:25:32 GoVersion: go version go1.13 linux/amd64 Race Enabled: false TiKV Min Version: v3.0.0-60965b006877ca7234adaced7890d7b029ed1306 Check Table Before Drop: false

jim.ni · 2019 年11 月 1 日 09:35

都是3.0.5，是不是PD节点数据损坏了？

jim.ni · 2019 年11 月 1 日 09:42

PD的数据好像还在

total 9968
-rw-r–r-- 1 tidb tidb 11075 Oct 17 18:15 000000000000000d-0000000000200b35.snap
-rw-r–r-- 1 tidb tidb 11075 Oct 21 05:05 000000000000000d-00000000002191d6.snap
-rw-r–r-- 1 tidb tidb 11075 Oct 24 14:19 000000000000000d-0000000000231877.snap
-rw-r–r-- 1 tidb tidb 11075 Oct 27 22:36 000000000000000d-0000000000249f18.snap
-rw-r–r-- 1 tidb tidb 11075 Oct 31 05:09 000000000000000d-00000000002625b9.snap
-rwxrwxrwx 1 tidb tidb 10141696 Nov 1 17:40 db
[tidb@gtp2 snap]$

QBin · 2019 年11 月 1 日 09:44

{ansible_path}/resource/bin/pd-ctl config show cluster-version -u http://{pd_ip}:{pd_port} 这个执行一下。

jim.ni · 2019 年11 月 1 日 09:52

[tidb@gtp2 bin]$ .//pd-ctl config show cluster-version -u http://192.168.3.62:2379
“3.0.5”

QBin · 2019 年11 月 1 日 10:05

确认下 TiKV TiDB 相关的 deploy_path 的目录对应的权限。另外麻烦给下 TiDB 的 log

jim.ni · 2019 年11 月 1 日 10:18

权限都查过没有问题，看了一下升级之前的Log，貌似已经发生错误了，全是

2019/11/01 13:43:03.864 terror.go:344: [error] lookup 47.100.23.254: Name or service not known
github.com/pingcap/errors.AddStack
        /home/jenkins/workspace/release_tidb_2.1-ga/go/pkg/mod/github.com/pingcap/errors@v0.11.1/errors.go:174
github.com/pingcap/errors.Trace
        /home/jenkins/workspace/release_tidb_2.1-ga/go/pkg/mod/github.com/pingcap/errors@v0.11.1/juju_adaptor.go:15
github.com/pingcap/tidb/session.getHostByIP
        /home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/session/session.go:1134
github.com/pingcap/tidb/session.(*session).Auth
        /home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/session/session.go:1114
github.com/pingcap/tidb/server.(*TiDBContext).Auth
        /home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/server/driver_tidb.go:274
github.com/pingcap/tidb/server.(*clientConn).openSessionAndDoAuth
        /home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/server/conn.go:414
github.com/pingcap/tidb/server.(*clientConn).readOptionalSSLRequestAndHandshakeResponse
        /home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/server/conn.go:395
github.com/pingcap/tidb/server.(*clientConn).handshake
        /home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/server/conn.go:133
github.com/pingcap/tidb/server.(*Server).onConn
        /home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/server/server.go:353
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1337

jim.ni · 2019 年11 月 1 日 10:24

然后再15:49,w我开始升级的，就持续报 Table ‘mysql.tidb’ doesn’t exist

[2019/11/01 15:49:11.169 +08:00] [INFO] [main.go:240] ["disable Prometheus push client"]
[2019/11/01 15:49:11.169 +08:00] [INFO] [tidb.go:311] ["new store"]
[2019/11/01 15:49:11.169 +08:00] [INFO] [systime_mon.go:26] ["start system time monitor"]
2019/11/01 15:49:11.169 client.go:124: [info] [pd] create pd client with endpoints [192.168.3.62:2379]
2019/11/01 15:49:11.171 client.go:226: [info] [pd] leader switches to: http://192.168.3.62:2379, previous:
2019/11/01 15:49:11.171 client.go:143: [info] [pd] init cluster id 6601270415411399747
[2019/11/01 15:49:11.178 +08:00] [INFO] [tidb.go:64] ["new domain"] [store=tikv-6601270415411399747] ["ddl lease"=45s] ["stats lease"=3s]
[2019/11/01 15:49:11.178 +08:00] [INFO] [ddl.go:369] ["[ddl] start DDL"] [ID=0f32ebce-63b0-4a5e-aca8-c8754e25b62a] [runWorker=true]
[2019/11/01 15:49:11.179 +08:00] [INFO] [ddl_worker.go:84] ["[ddl] start delRangeManager OK"] ["is a emulator"=false]
[2019/11/01 15:49:11.179 +08:00] [INFO] [ddl_worker.go:84] ["[ddl] start delRangeManager OK"] ["is a emulator"=false]
[2019/11/01 15:49:11.179 +08:00] [INFO] [ddl_worker.go:120] ["[ddl] start DDL worker"] [worker="worker 1, tp general"]
[2019/11/01 15:49:11.179 +08:00] [INFO] [ddl_worker.go:120] ["[ddl] start DDL worker"] [worker="worker 2, tp add index"]
[2019/11/01 15:49:12.929 +08:00] [INFO] [domain.go:131] ["full load InfoSchema success"] [usedSchemaVersion=0] [latestSchemaVersion=34655] ["start time"=1.739547838s]
[2019/11/01 15:49:12.930 +08:00] [INFO] [domain.go:332] ["full load and reset schema validator"]
[2019/11/01 15:49:12.934 +08:00] [WARN] [session.go:875] ["compile sql error"] [error="[schema:1146]Table 'mysql.tidb' doesn't exist"] [errorVerbose="[schema:1146]Table 'mysql.tidb' doesn't exist
github.com/pingcap/errors.AddStack
	/home/jenkins/workspace/release_tidb_2.1-ga/go/pkg/mod/github.com/pingcap/errors@v0.11.1/errors.go:174
github.com/pingcap/parser/terror.(*Error).GenWithStackByArgs
	/home/jenkins/workspace/release_tidb_2.1-ga/go/pkg/mod/github.com/pingcap/parser@v0.0.0-20190505094039-595d728571a7/terror/terror.go:233
github.com/pingcap/tidb/infoschema.(*infoSchema).TableByName
	/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/infoschema/infoschema.go:177
github.com/pingcap/tidb/planner/core.(*preprocessor).handleTableName
	/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/planner/core/preprocess.go:584
github.com/pingcap/tidb/planner/core.(*preprocessor).Leave
	/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/planner/core/preprocess.go:118
github.com/pingcap/parser/ast.(*TableName).Accept
	/home/jenkins/workspace/release_tidb_2.1-ga/go/pkg/mod/github.com/pingcap/parser@v0.0.0-20190505094039-595d728571a7/ast/dml.go:163
github.com/pingcap/parser/ast.(*TableSource).Accept
	/home/jenkins/workspace/release_tidb_2.1-ga/go/pkg/mod/github.com/pingcap/parser@v0.0.0-20190505094039-595d728571a7/ast/dml.go:232
github.com/pingcap/parser/ast.(*Join).Accept
	/home/jenkins/workspace/release_tidb_2.1-ga/go/pkg/mod/github.com/pingcap/parser@v0.0.0-20190505094039-595d728571a7/ast/dml.go:92
github.com/pingcap/parser/ast.(*TableRefsClause).Accept
	/home/jenkins/workspace/release_tidb_2.1-ga/go/pkg/mod/github.com/pingcap/parser@v0.0.0-20190505094039-595d728571a7/ast/dml.go:356
github.com/pingcap/parser/ast.(*SelectStmt).Accept
	/home/jenkins/workspace/release_tidb_2.1-ga/go/pkg/mod/github.com/pingcap/parser@v0.0.0-20190505094039-595d728571a7/ast/dml.go:512
github.com/pingcap/tidb/planner/core.Preprocess
	/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/planner/core/preprocess.go:35
github.com/pingcap/tidb/executor.(*Compiler).Compile
	/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/executor/compiler.go:45
github.com/pingcap/tidb/session.(*session).execute
	/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/session/session.go:872
github.com/pingcap/tidb/session.(*session).Execute
	/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/session/session.go:832
github.com/pingcap/tidb/session.loadSystemTZ
	/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/session/session.go:1184
github.com/pingcap/tidb/session.BootstrapSession
	/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/session/session.go:1230
main.createStoreAndDomain
	/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/tidb-server/main.go:195
main.main
	/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/tidb-server/main.go:161
runtime.main
	/usr/local/go/src/runtime/proc.go:200
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1337"] [sql="select variable_value from mysql.tidb where variable_name = 'system_tz'"]
2019/11/01 15:49:12.935 terror.go:329: [fatal] [schema:1146]Table 'mysql.tidb' doesn't exist

jim.ni · 2019 年11 月 1 日 10:27

还有没有救了？

QBin · 2019 年11 月 1 日 11:02

麻烦直接发下升级前后的 TiDB 日志文件看下。