dumpling备份数据时报错:error="runtime error: index out of range [1] with length 0"

为提高效率,请提供以下信息,问题描述清晰能够更快得到解决:
【 TiDB 使用环境】
tidb版本:Release Version: v5.0.1
部署架构:2tidb+3pd+3tikv独立部署

【概述】 场景 + 问题概述
备份任务失败,排查到tidb日志中报错信息:当备份数据库dbname下的表tablename1时,执行sql:SELECT id FROM dbname.tablename1 TABLESAMPLE REGIONS() ORDER BY id报错。

tablename1是个空表,没有数据。

tidb报错日志:
[2021/09/21 05:05:17.950 +08:00] [ERROR] [projection.go:448] [“projection executor panicked”] [error=“runtime error: index out of range [1] with length 0”] [stack="goroutine 4656050184…(后边的省略)

[2021/09/21 05:05:17.950 +08:00] [INFO] [conn.go:812] [“command dispatched failed”] [conn=12073577] [connInfo=“id:12073577, addr:127.0.0.1:38104 status:11, collation:utf8mb4_general_ci, user:bkUser”] [command=Query] [status=“inTxn:1, autocommit:1”] [sql="SELECT id FROM dbname.tablename1 TABLESAMPLE REGIONS() ORDER BY id"] [txn_mode=PESSIMISTIC] [err="runtime error: index out of range [1] with length 0…(后边的省略)

现象:生产和测试环境tidb集群版本一样,表结构一样;上报错的sql在测试环境执行没问题,在生产环境执行报错runtime error: index out of range [1] with length 0;使用tablename1的建表命令,改个表名,重建在生产环境create一个表,新建的表执行SELECT id FROM dbname.tablename1_new TABLESAMPLE REGIONS() ORDER BY `id;没问题。

表结构:
CREATE TABLE tablename1 (
id bigint(13) NOT NULL AUTO_INCREMENT,
created_at datetime DEFAULT CURRENT_TIMESTAMP,
updated_at datetime DEFAULT NULL,
activity_id bigint(13) NOT NULL COMMENT ‘活动ID’,
content_config_id bigint(13) NOT NULL COMMENT ‘通用活动配置主键ID’,
PRIMARY KEY (id) /*T![clustered_index] CLUSTERED */,
KEY IDX_ACTIVITY_ID (activity_id),
KEY IDX_CONFIG_ID (content_config_id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin COMMENT=‘页面装修通用活动表’

【问题】 当前遇到的问题
针对以上现象很疑惑?

【业务影响】
备份任务失败

【TiDB 版本】
Release Version: v5.0.1
Edition: Community
Git Commit Hash: 1145e347d3469d8e89f88dce86f6926ca44b3cd8
Git Branch: heads/refs/tags/v5.0.1
UTC Build Time: 2021-04-23 05:51:17
GoVersion: go1.13
Race Enabled: false
TiKV Min Version: v3.0.0-60965b006877ca7234adaced7890d7b029ed1306
Check Table Before Drop: false


若提问为性能优化、故障排查类问题,请下载脚本运行。终端输出的打印结果,请务必全选并复制粘贴上传。

3赞

…(后边的省略),麻烦把这后面的内容提供一下,不过咱们的问题,估计是个 BUG,触发条件,需要根据日志,先确认一下了

3赞

您好,下是省略的日志内容:
[2021/09/21 05:05:17.950 +08:00] [ERROR] [projection.go:448] [“projection executor panicked”] [error=“runtime error: index out of range[1] with length 0”] [stack=“goroutine 4656050184 [running]:\ngithub.com/pingcap/tidb/util.GetStack(...)\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/util/misc.go:74\ngithub.com/pingcap/tidb/executor.recoveryProjection(0xc02cceb2c0, 0x3799860, 0xc0859347a0)\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/projection.go:447 +0x90\ngithub.com/pingcap/tidb/executor.(*projectionInputFetcher).run.func1(0xc09322bf80, 0xc04cc17b80)\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/projection.go:349 +0x82\npanic(0x3799860, 0xc0859347a0)\n\t/usr/local/go/src/runtime/panic.go:679 +0x1b2\ngithub.com/pingcap/tidb/util/chunk.setMutRowBytes(...)\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/util/chunk/mutrow.go:328\ngithub.com/pingcap/tidb/util/chunk.MutRow.SetValue(0xc039d8bf40, 0x0, 0x3, 0x3438fc0, 0xc066ca2860)\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/util/chunk/mutrow.go:259 +0xb01\ngithub.com/pingcap/tidb/util/rowDecoder.(*RowDecoder).DecodeAndEvalRowWithMap(0xc072814e80, 0x405dc80, 0xc01c8c6500, 0x4040a40, 0xc043e9e760, 0xc08f12f7c0, 0x41, 0x50, 0xc000cf9740, 0x5923180, …)\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/util/rowDecoder/decoder.go:108 +0x30d\ngithub.com/pingcap/tidb/executor.(*tableRegionSampler).writeChunkFromRanges.func1(0x4040a40, 0xc043e9e760, 0xc08f12f7c0, 0x41, 0x50, 0x7f115b0bd1f0, 0x0)\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/sample.go:165 +0xd8\ngithub.com/pingcap/tidb/executor.(*sampleSyncer).sync(0xc09322bbd8, 0x0, 0x0)\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/sample.go:398 +0x164\ngithub.com/pingcap/tidb/executor.(*tableRegionSampler).scanFirstKVForEachRange(0xc02ccf3a40, 0xc0148c9290, 0x1, 0x1, 0xc09322bc60, 0xc0148c92f0, 0xc072814e80)\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/sample.go:310 +0x284\ngithub.com/pingcap/tidb/executor.(*tableRegionSampler).writeChunkFromRanges(0xc02ccf3a40, 0xc0148c9290, 0x1, 0x1, 0xc039d8b950, 0xc000e0ad10, 0x167c1b1)\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/sample.go:164 +0x196\ngithub.com/pingcap/tidb/executor.(*tableRegionSampler).writeChunk(0xc02ccf3a40, 0xc039d8b950, 0x0, 0x607511d0763800)\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/sample.go:123 +0xf3\ngithub.com/pingcap/tidb/executor.(*TableSampleExecutor).Next(0xc02ccf39a0, 0x4004660, 0xc063d51c80, 0xc039d8b950, 0x1, 0x0)\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/sample.go:69 +0x8c\ngithub.com/pingcap/tidb/executor.Next(0x4004660, 0xc063d51c80, 0x400d0a0, 0x c02ccf39a0, 0xc039d8b950, 0x0, 0x0)\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/executor.go:277 +0x225\ngithub.com/pingcap/tidb/executor.(*projectionInputFetcher).run(0xc04cc17b80, 0x4004660, 0xc063d51c80)\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/projection.go:373 +0x283\ncreated by github.com/pingcap/tidb/executor.(*ProjectionExec).prepare\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/projection.go:265 +0x753\n”]

[2021/09/21 05:05:17.950 +08:00] [INFO] [conn.go:812] [“command dispatched failed”] [conn=12073577] [connInfo=“id:12073577, addr:127.0.0.1:38104 status:11, collation:utf8mb4_general_ci, user:bkUser”] [command=Query] [status=“inTxn:1, autocommit:1”] [sql="SELECT id FROM dbname.tablename1 TABLESAMPLE REGIONS() ORDER BY id"] [txn_mode=PESSIMISTIC] [err=“runtime error: index out of range [1] with length 0\ngithub.com/pingcap/tidb/executor.recoveryProjection\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/projection.go:445\ngithub.com/pingcap/tidb/executor.(*projectionInputFetcher).run.func1\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/projection.go:349\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:679\nruntime.goPanicIndex\n\t/usr/local/go/src/runtime/panic.go:75\ngithub.com/pingcap/tidb/util/chunk.setMutRowBytes\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/util/chunk/mutrow.go:328\ngithub.com/pingcap/tidb/util/chunk.MutRow.SetValue\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/util/chunk/mutrow.go:259\ngithub.com/pingcap/tidb/util/rowDecoder.(*RowDecoder).DecodeAndEvalRowWithMap\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/util/rowDecoder/decoder.go:108\ngithub.com/pingcap/tidb/executor.(*tableRegionSampler).writeChunkFromRanges.func1\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/sample.go:165\ngithub.com/pingcap/tidb/executor.(*sampleSyncer).sync\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/sample.go:398\ngithub.com/pingcap/tidb/executor.(*tableRegionSampler).scanFirstKVForEachRange\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/sample.go:310\ngithub.com/pingcap/tidb/executor.(*tableRegionSampler).writeChunkFromRanges\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/sample.go:164\ngithub.com/pingcap/tidb/executor.(*tableRegionSampler).writeChunk\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/sample.go:123\ngithub.com/pingcap/tidb/executor.(*TableSampleExecutor).Next\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/sample.go:69\ngithub.com/pingcap/tidb/executor.Next\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/executor.go:277\ngithub.com/pingcap/tidb/executor.(*projectionInputFetcher).run\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb/executor/projection.go:373\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357”]

3赞

好滴,收到

2赞

该表最近进行过哪些操作?譬如删除数据、DDL 变更?

2赞

是否有升级过版本?
同样请求是否能稳定复现?

2赞

您好:集群是升级过,但是表应该是升级之后创建的。

备份任务每天跑一次,不是每天都报错。但是在排查问题的时候,人为每次执行下sql,都会报上边的错误。
SELECT id FROM dbname . tablename1 TABLESAMPLE REGIONS() ORDER BY id

3赞

您好:不好意思,你的两个问题无法准确给出答复。
涉及到这个表的功能还没上线,表中也没有数据,大概率是表建好就没再动过了。

3赞

@liuhuanHappyStudy. 看你上面的意思是说,每次手动这些 这个 SQL 都会报错?能提供一下这个 表的表结构不:
1、这个肯定是产品 bug 了,现在我们这边能做的是定位产生 bug 的触发条件,好后续提供不升级的临时解决办法
2、咱们可以在高版本,模拟一下这个环境,看看高版本是否还有该问题(如果没问题,升级其实是最稳妥的办法,就是可能咱们还要升级一次,动作较大)

4赞

是的。

表结构:
CREATE TABLE tablename1 (
id bigint(13) NOT NULL AUTO_INCREMENT,
created_at datetime DEFAULT CURRENT_TIMESTAMP,
updated_at datetime DEFAULT NULL,
activity_id bigint(13) NOT NULL COMMENT ‘活动ID’,
content_config_id bigint(13) NOT NULL COMMENT ‘通用活动配置主键ID’,
PRIMARY KEY ( id ) /*T![clustered_index] CLUSTERED */,
KEY IDX_ACTIVITY_ID ( activity_id ),
KEY IDX_CONFIG_ID ( content_config_id )
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin COMMENT=‘页面装修通用活动表’

我使用相同的表结构,新建表(只改了表名),然后在新建的表上执行上select 的sql不报错;然后我就把原来的表drop掉,把新建的表rename一下,临时解决了上述的问题。

非常感谢!

4赞

:smiley:收到

1赞

是否有用到过生成列,或者是 expression index 这类的 feature 呀?
如果可以拿到 ddl history 可以拿一下 https://github.com/pingcap/tidb/blob/master/docs/tidb_http_api.md
怀疑这个问题是跟生成列有关的

1赞

确认是一个 bug,在这个 pr 修复的 https://github.com/pingcap/tidb/pull/25357
修复的版本是 v5.0.3

1赞