Analyze table panicked

v-gu · 2020 年5 月 23 日 21:27

Deployed using TIDB Operator v1.1.0-rc.2 on GKE( 1.16.8-gke.15 )

[2020/05/22 14:33:29.292 +00:00] [FATAL] [lib.rs:480] ["index 3113512960 out of range for slice of length 65447"] [backtrace="stack backtrace:
   0: tikv_util::set_panic_hook::{{closure}}
             at components/tikv_util/src/lib.rs:479
   1: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:475
   2: rust_begin_unwind
             at src/libstd/panicking.rs:375
   3: core::panicking::panic_fmt
             at src/libcore/panicking.rs:84
   4: core::slice::slice_index_len_fail
             at src/libcore/slice/mod.rs:2668
   5: <tidb_query::executor::scan::ScanExecutor<S,T> as tidb_query::executor::Executor>::next
             at rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libcore/slice/mod.rs:2851
   6: tikv::coprocessor::statistics::analyze::SampleBuilder<S>::collect_columns_stats
             at home/jenkins/agent/workspace/ld_tikv_multi_branch_v4.0.0-rc.2/tikv/src/coprocessor/statistics/analyze.rs:213
      tikv::coprocessor::statistics::analyze::AnalyzeContext<S>::handle_column
             at home/jenkins/agent/workspace/ld_tikv_multi_branch_v4.0.0-rc.2/tikv/src/coprocessor/statistics/analyze.rs:58
      <tikv::coprocessor::statistics::analyze::AnalyzeContext<S> as tikv::coprocessor::RequestHandler>::handle_request::__handle_request::{{closure}}
             at home/jenkins/agent/workspace/ld_tikv_multi_branch_v4.0.0-rc.2/tikv/src/coprocessor/statistics/analyze.rs:126
   7: <std::future::GenFuture<T> as core::future::future::Future>::poll
             at rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libstd/future.rs:43
   8: <core::pin::Pin<P> as core::future::future::Future>::poll
             at rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libcore/future/future.rs:119
      <tikv::coprocessor::interceptors::tracker::Tracker<F> as core::future::future::Future>::poll

Behavior repeatable everytime.

qw4990-PingCAP · 2020 年5 月 25 日 06:45

@v-gu 感谢你的反馈，请给一下你那边 TiDB/TiKV 的版本信息。

5kbpers-PingCAP · 2020 年5 月 25 日 08:29

请问方便让我们提供一个 debug 版本，进一步查证问题吗？

v-gu · 2020 年5 月 25 日 09:55

v-gu · 2020 年5 月 25 日 09:56

可以build一个镜像，我拉一下。这个问题稳定复现

v-gu · 2020 年5 月 27 日 20:51

今天在这个表上建索引，碰到了类似的问题

2020/05/27 18:08:27.999 +00:00] [ERROR] [index.go:1138] ["[ddl] add index worker panic"]
panic="runtime error: slice bounds out of range [:29405654] with capacity 73546"] [stack="goroutine 38791048 [running]:
github.com/pingcap/tidb/util.GetStack(...)
    /home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/util/misc.go:72
github.com/pingcap/tidb/ddl.(*addIndexWorker).run.func1(0xc0003004b0)
    /home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/ddl/index.go:1137 +0x132
panic(0x2f42860, 0xc019823520)
    /usr/local/go/src/runtime/panic.go:679 +0x1b2
github.com/pingcap/tidb/util/rowcodec.(*row).getData(...)
    /home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/util/rowcodec/row.go:50
github.com/pingcap/tidb/util/rowcodec.(*DatumMapDecoder).DecodeToDatumMap(0xc019af08d0, 0xc00a162000, 0x1005d, 0x12000, 0xffffffffffffffff, 0xc001e0e8d0, 0xc001e0e8d0, 0x0, 0x0)
    /home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/util/rowcodec/decoder.go:94 +0x685
github.com/pingcap/tidb/tablecodec.DecodeRowWithMapNew(0xc00a162000, 0x1005d, 0x12000, 0xc001e0e8a0, 0x4bd6c00, 0xc001e0e8d0, 0xc001e0e8d0, 0x0, 0x0)
    /home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/tablecodec/tablecodec.go:383 +0x300
github.com/pingcap/tidb/util/rowDecoder.(*RowDecoder).DecodeAndEvalRowWithMap(0xc01383aaa0, 0x36699c0, 0xc00085a780, 0x1c0b20c57947000, 0xc00a162000, 0x1005d, 0x12000, 0x4bd6c00, 0x4bd6c00, 0xc001e0e8d0, ...)
    /home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/util/rowDecoder/decoder.go:83 +0x9a
github.com/pingcap/tidb/ddl.(*addIndexWorker).getIndexRecord(0xc0003004b0, 0x1c0b20c57947000, 0xc019823500, 0x13, 0x13, 0xc00a162000, 0x1005d, 0x12000, 0x13, 0xc019823500, ...)
    /home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/ddl/index.go:823 +0x129
github.com/pingcap/tidb/ddl.(*addIndexWorker).fetchRowColVals.func1(0x1c0b20c57947000, 0xc019823500, 0x13, 0x13, 0xc00a162000, 0x1005d, 0x12000, 0x3633101, 0x0, 0x0)
    /home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/ddl/index.go:923 +0x1c8
github.com/pingcap/tidb/ddl.iterateSnapshotRows(0x365a6c0, 0xc00024ed00, 0x1, 0x3666320, 0xc0022d6fc0, 0x5c95d4904e80003, 0x1c0b1f598147001, 0x1c0bd9aee547000, 0x0, 0xc019af1048, ...)
    /home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/ddl/index.go:1594 +0x3f6
github.com/pingcap/tidb/ddl.(*addIndexWorker).fetchRowColVals(0xc0003004b0, 0x3666e80, 0xc0198f6e70, 0x36a, 0x1c0b1f598147001, 0x1c0bd9aee547000, 0xc0198f6e00, 0x0, 0x0, 0x3627980, ...)
    /home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/ddl/index.go:907 +0x1b9
github.com/pingcap/tidb/ddl.(*addIndexWorker).backfillIndexInTxn.func2(0x3666e80, 0xc0198f6e70, 0xc0198f6e70, 0x0)
    /home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/ddl/index.go:1037 +0xf5
github.com/pingcap/tidb/kv.RunInNewTxn(0x365a6c0, 0xc00024ed00, 0xc00024ed01, 0xc019af16e0, 0xbb8, 0x0)
    /home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/kv/txn.go:47 +0xde
github.com/pingcap/tidb/ddl.(*addIndexWorker).backfillIndexInTxn(0xc0003004b0, 0x36a, 0x1c0b1f598147001, 0x1c0bd9aee547000, 0x50c5300, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/ddl/index.go:1032 +0x12e
github.com/pingcap/tidb/ddl.(*addIndexWorker).handleBackfillTask(0xc0003004b0, 0xc0000dd9a0, 0xc0054278a0, 0xc00bec5f00)
    /home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/ddl/index.go:1105 +0x192
github.com/pingcap/tidb/ddl.(*addIndexWorker).run(0xc0003004b0, 0xc0000dd9a0)
    /home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/ddl/index.go:1160 +0x230
created by github.com/pingcap/tidb/ddl.(*worker).addPhysicalTableIndex
    /home/jenkins/agent/workspace/tidb_v4.0.0-rc.2/go/src/github.com/pingcap/tidb/ddl/index.go:1416 +0x4a2
"

v-gu · 2020 年5 月 27 日 20:52

这个表是通过lightning导入的，导入之后目前就现在这个情况，并未做其他DML和DDL