前端应用反馈报错:iError: ER_UNKNOWN_ERROR: runtime error: invalid memory address or nil pointer dereference

为提高效率,请提供以下信息,问题描述清晰能够更快得到解决:

[TiDB 版本]
$ tiup cluster list
Found cluster newer version:

The latest version:         v1.3.1
Local installed version:    v1.2.5
Update current component:   tiup update cluster
Update all components:      tiup update --all

Starting component cluster : /home/tidb/.tiup/components/cluster/v1.2.5/tiup-cluster list
Name User Version Path PrivateKey

tidb-pro tidb v4.0.4 /home/tidb/.tiup/storage/cluster/clusters/tidb-pro /home/tidb/.tiup/storage/cluster/clusters/tidb-pro/ssh/id_rsa

[问题描述]
应用服务报错信息:
invalid memory address or nil pointer dereference

[2021-01-19T08:41:22.911] [INFO] console - { Error: ER_UNKNOWN_ERROR: runtime error: invalid memory address or nil pointer dereference
at Query.Sequence._packetToError (/home/APP/yujiang.Foil.Node.Npm/win64-node8.15.1/mysql/lib/protocol/sequences/Sequence.js:47:14)
at Query.ErrorPacket (/home/APP/yujiang.Foil.Node.Npm/win64-node8.15.1/mysql/lib/protocol/sequences/Query.js:77:18)
at Protocol._parsePacket (/home/APP/yujiang.Foil.Node.Npm/win64-node8.15.1/mysql/lib/protocol/Protocol.js:278:23)
at Parser.write (/home/APP/yujiang.Foil.Node.Npm/win64-node8.15.1/mysql/lib/protocol/Parser.js:76:12)
at Protocol.write (/home/APP/yujiang.Foil.Node.Npm/win64-node8.15.1/mysql/lib/protocol/Protocol.js:38:16)
at Socket. (/home/APP/yujiang.Foil.Node.Npm/win64-node8.15.1/mysql/lib/Connection.js:91:28)
at Socket. (/home/APP/yujiang.Foil.Node.Npm/win64-node8.15.1/mysql/lib/Connection.js:502:10)
at emitOne (events.js:116:13)
at Socket.emit (events.js:211:7)
at addChunk (_stream_readable.js:263:12)
--------------------
at Protocol._enqueue (/home/APP/yujiang.Foil.Node.Npm/win64-node8.15.1/mysql/lib/protocol/Protocol.js:144:48)
at PoolConnection.query (/home/APP/yujiang.Foil.Node.Npm/win64-node8.15.1/mysql/lib/Connection.js:200:25)
at doQuery (/home/APP/yujiang.Foil.Node/src/yjDBService.engine.mysql.js:475:24)
at success (/home/APP/yujiang.Foil.Node/src/yjDBService.engine.mysql.js:466:9)
at Object.exports.safeSuccess (/home/APP/yujiang.Foil.Node/src/yjError.js:100:13)
at Object.exports.handleResult (/home/APP/yujiang.Foil.Node/src/yjError.js:154:12)
at /home/APP/yujiang.Foil.Node/src/yjDBService.engine.mysql.js:90:11
at Ping.onOperationComplete (/home/APP/yujiang.Foil.Node.Npm/win64-node8.15.1/mysql/lib/Pool.js:110:5)
at bound (domain.js:301:14)
at Ping.runBound [as _callback] (domain.js:314:12)
code: ‘ER_UNKNOWN_ERROR’,
errno: 1105,
sqlMessage: ‘runtime error: invalid memory address or nil pointer dereference’,
sqlState: ‘HY000’,
index: 0,
sql: ’ select SQL_CALC_FOUND_ROWS p1.,(case when p3.GreenhousesSystemParentOID is null then 0 else count() end) as childrenCount,Re_SI.ReSystemIDCOID,Re_SI.IDCOID from Gre
enhousesSystem as p1 left outer join GreenhousesSystem as p3 on p1.GreenhousesSystemOID=p3.GreenhousesSystemParentOID left join Re_systemIDCOID Re_SI on Re_SI.GreenhousesSystemOID=p1.GreenhousesSystemOID where p1.GreenhousesSystemParentOID=‘4’ group by p1.GreenhousesSystemOID Order By GreenhousesSystemOID asc;SELECT FOUND_ROWS() as Total;’, url: ‘/biz/iAGS/more/morechild/systemManage/getDeviceSystem?id=4&page=1&rows=10&sort=GreenhousesSystemOID&order=asc&isLoadChildren=true&IDCOID=50’ }

注:单独执行sql语句时没有报错。

image加粗示例

麻烦看一下对应的处理业务请求的 TiDB.log 的日志是什么 ?

下面是tidb.log部分内容

[2021/01/25 13:28:00.047 +08:00] [INFO] [server.go:388] [“new connection”] [conn=7300] [remoteAddr=192.168.9.13:49900]
[2021/01/25 13:28:00.047 +08:00] [INFO] [server.go:388] [“new connection”] [conn=7301] [remoteAddr=192.168.9.13:49902]
[2021/01/25 13:28:47.443 +08:00] [ERROR] [select_result.go:242] [“invalid cop task execution summaries length”] [expected=1] [received=0]
[2021/01/25 13:28:47.458 +08:00] [INFO] [server.go:388] [“new connection”] [conn=7302] [remoteAddr=192.168.9.22:49946]
[2021/01/25 13:28:51.293 +08:00] [ERROR] [misc.go:91] [“panic in the recoverable goroutine”] [r="“invalid memory address or nil pointer dereference”"] [
“stack trace”=“github.com/pingcap/tidb/util.WithRecovery.func1\n\t/home/jenkins/agent/workspace/tidb_v4.0.4/go/src/github.com/pingcap/tidb/util/misc.go:93
\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:679\nruntime.panicmem\n\t/usr/local/go/src/runtime/panic.go:199\nruntime.sigpanic\n\t/usr/local/go
/src/runtime/signal_unix.go:394\nsync.(*Mutex).Lock\n\t/usr/local/go/src/sync/mutex.go:74\ngithub.com/pingcap/tidb/util/execdetails.(*RuntimeStatsColl).Ge
tReaderStats\n\t/home/jenkins/agent/workspace/tidb_v4.0.4/go/src/github.com/pingcap/tidb/util/execdetails/execdetails.go:452\ngithub.com/pingcap/tidb/util
/execdetails.(*RuntimeStatsColl).RecordOneReaderStats\n\t/home/jenkins/agent/workspace/tidb_v4.0.4/go/src/github.com/pingcap/tidb/util/execdetails/execdet
ails.go:430\ngithub.com/pingcap/tidb/distsql.(*selectResult).updateCopRuntimeStats\n\t/home/jenkins/agent/workspace/tidb_v4.0.4/go/src/github.com/pingcap/
tidb/distsql/select_result.go:249\ngithub.com/pingcap/tidb/distsql.(*selectResult).fetchResp\n\t/home/jenkins/agent/workspace/tidb_v4.0.4/go/src/github.co
m/pingcap/tidb/distsql/select_result.go:134\ngithub.com/pingcap/tidb/distsql.(*selectResult).Next\n\t/home/jenkins/agent/workspace/tidb_v4.0.4/go/src/gith
ub.com/pingcap/tidb/distsql/select_result.go:151\ngithub.com/pingcap/tidb/executor.(*tableResultHandler).nextChunk\n\t/home/jenkins/agent/workspace/tidb_v
4.0.4/go/src/github.com/pingcap/tidb/executor/table_reader.go:280\ngithub.com/pingcap/tidb/executor.(*TableReaderExecutor).Next\n\t/home/jenkins/agent/wor
kspace/tidb_v4.0.4/go/src/github.com/pingcap/tidb/executor/table_reader.go:166\ngithub.com/pingcap/tidb/executor.Next\n\t/home/jenkins/agent/workspace/tid
b_v4.0.4/go/src/github.com/pingcap/tidb/executor/executor.go:249\ngithub.com/pingcap/tidb/executor.(HashJoinExec).fetchBuildSideRows\n\t/home/jenkins/age
nt/workspace/tidb_v4.0.4/go/src/github.com/pingcap/tidb/executor/join.go:260\ngithub.com/pingcap/tidb/executor.(HashJoinExec).fetchAndBuildHashTable.func
1\n\t/home/jenkins/agent/workspace/tidb_v4.0.4/go/src/github.com/pingcap/tidb/executor/join.go:643\ngithub.com/pingcap/tidb/util.WithRecovery\n\t/home/jen
kins/agent/workspace/tidb_v4.0.4/go/src/github.com/pingcap/tidb/util/misc.go:96"]
[2021/01/25 13:28:51.295 +08:00] [ERROR] [conn.go:730] [“command dispatched failed”] [conn=7294] [connInfo=“id:7294, addr:192.168.9.22:49060 status:10, co
llation:utf8_general_ci, user:root”] [command=Query] [status=“inTxn:0, autocommit:1”] [sql=" select SQL_CALC_FOUND_ROWS p1.
,(case when p3.GreenhousesSys
temParentOID is null then 0 else count(
) end) as childrenCount,Re_SI.ReSystemIDCOID,Re_SI.IDCOID from GreenhousesSystem as p1 left outer join Greenhouse
sSystem as p3 on p1.GreenhousesSystemOID=p3.GreenhousesSystemParentOID left join Re_systemIDCOID Re_SI on Re_SI.GreenhousesSystemOID=p1.GreenhousesSystem
OID where p1.GreenhousesSystemParentOID=‘2’ group by p1.GreenhousesSystemOID Order By GreenhousesSystemOID asc;SELECT FOUND_ROWS() as Total;”] [txn_mo
de=PESSIMISTIC] [err=“runtime error: invalid memory address or nil pointer dereference\ngithub.com/pingcap/tidb/executor.(*HashJoinExec).fetchAndBuildHash
Table.func2\n\t/home/jenkins/agent/workspace/tidb_v4.0.4/go/src/github.com/pingcap/tidb/executor/join.go:646\ngithub.com/pingcap/tidb/util.WithRecovery.fu
nc1\n\t/home/jenkins/agent/workspace/tidb_v4.0.4/go/src/github.com/pingcap/tidb/util/misc.go:88\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:679
\nruntime.panicmem\n\t/usr/local/go/src/runtime/panic.go:199\nruntime.sigpanic\n\t/usr/local/go/src/runtime/signal_unix.go:394\nsync.(*Mutex).Lock\n\t/usr
/local/go/src/sync/mutex.go:74\ngithub.com/pingcap/tidb/util/execdetails.(*RuntimeStatsColl).GetReaderStats\n\t/home/jenkins/agent/workspace/tidb_v4.0.4/g
o/src/github.com/pingcap/tidb/util/execdetails/execdetails.go:452\ngithub.com/pingcap/tidb/util/execdetails.(*RuntimeStatsColl).RecordOneReaderStats\n\t/h
ome/jenkins/agent/workspace/tidb_v4.0.4/go/src/github.com/pingcap/tidb/util/execdetails/execdetails.go:430\ngithub.com/pingcap/tidb/distsql.(*selectResult
).updateCopRuntimeStats\n\t/home/jenkins/agent/workspace/tidb_v4.0.4/go/src/github.com/pingcap/tidb/distsql/select_result.go:249\ngithub.com/pingcap/tidb/
distsql.(*selectResult).fetchResp\n\t/home/jenkins/agent/workspace/tidb_v4.0.4/go/src/github.com/pingcap/tidb/distsql/select_result.go:134\ngithub.com/pin
gcap/tidb/distsql.(*selectResult).Next\n\t/home/jenkins/agent/workspace/tidb_v4.0.4/go/src/github.com/pingcap/tidb/distsql/select_result.go:151\ngithub.co
m/pingcap/tidb/executor.(*tableResultHandler).nextChunk\n\t/home/jenkins/agent/workspace/tidb_v4.0.4/go/src/github.com/pingcap/tidb/executor/table_reader.
go:280\ngithub.com/pingcap/tidb/executor.(*TableReaderExecutor).Next\n\t/home/jenkins/agent/workspace/tidb_v4.0.4/go/src/github.com/pingcap/tidb/executor/
table_reader.go:166\ngithub.com/pingcap/tidb/executor.Next\n\t/home/jenkins/agent/workspace/tidb_v4.0.4/go/src/github.com/pingcap/tidb/executor/executor.g
o:249\ngithub.com/pingcap/tidb/executor.(*HashJoinExec).fetchBuildSideRows\n\t/home/jenkins/agent/workspace/tidb_v4.0.4/go/src/github.com/pingcap/tidb/exe
cutor/join.go:260\ngithub.com/pingcap/tidb/executor.(*HashJoinExec).fetchAndBuildHashTable.func1\n\t/home/jenkins/agent/workspace/tidb_v4.0.4/go/src/githu
b.com/pingcap/tidb/executor/join.go:643\ngithub.com/pingcap/tidb/util.WithRecovery\n\t/home/jenkins/agent/workspace/tidb_v4.0.4/go/src/github.com/pingcap/
tidb/util/misc.go:96\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357”]
[root@tidb-server-01 log]#

另外,同样的业务,在tidb-v3.0版本上业务页面没有任何报错

这个具体的触发操作是什么?

请问下是否能够稳定复现?

系统管理中页面的查询操作

一直在特定页面中稳定出现

其他页面到没发现这种情况

@john402 请问下是否方便往集群中接入一个用于 debug 的 TiDB? 我们想打点日志排查下

如何接入debug的TiDB

我们这边提供一个二进制的 tidb,
然后运行:
./tidb-server -path “127.0.0.1:2379” -store “tikv”
-path 后面接 PD 的 IP 以及端口

好的,在哪可下载这个二进制的TiDB

稍等,我这边弄一下

@john402 tidb地址:https://drive.google.com/file/d/1_5LAFUj-dmLPc4fS2gXaijVrbvHUnLwo/view?usp=sharing

进到目录下:
./tidb-server-linux -path “127.0.0.1:2379” -store “tikv”
-path 后面接 PD 的 IP 以及端口
然后起一个 client, 连接这个 tidb,执行前面的 sql,然后查看日志。

不好意思,中国这边访问不了google地址,能否提供其他的下载连接。

你好,看一下私聊信息哈。

执行了:
./tidb-server-linux -config “/home/tidb/tidb.toml” -path “192.168.9.13:2379,192.168.9.14:2379,192.168.9.15:2379,” -store “tikv”

报错:
[2021/01/26 14:09:27.036 +08:00] [FATAL] [terror.go:348] [“unexpected error”] [error=“parse tikv://192.168.9.13:2379,192.168.9.14:2379,192.168.9.15:2379,: invalid port “:2379,” after host”]

tidb.toml配置文件中没有tikv的相关配置,怎么会报tikv这个错

指定任意一个 pd 就行了

好,我试下