TiDB v4.0.5 where 语句失效

keithyzk · 2020 年9 月 24 日 09:13

好的，已经解决了。我们拿少量数据简单测试了，没有发现问题。线上环境恢复完成也会做完整的测试之后上线。感谢~

yilong · 2020 年9 月 24 日 09:14

Cong-PingCAP · 2020 年9 月 25 日 02:32

我确认一下，使用新编译的 Lightning，Local 模式导入的吧？

keithyzk · 2020 年9 月 25 日 02:42

是的。但是从昨晚开始恢复，目前也遇到了问题，虽然恢复过程没有报错，但是总是会卡在这个界面不动

昨晚卡在48% 不动了四五个小时，直到凌晨12点的时候也一样，早上8点看到是70%多，然后现在卡在 100% 一个多个小时了 .
恢复配置文件

[lightning]

# 转换数据的并发数，默认为逻辑 CPU 数量，不需要配置。
# 混合部署的情况下可以配置为逻辑 CPU 的 75% 大小。
# region-concurrency =

# 日志
level = "info"
file = "tidb-lightning.log"

[checkpoint]
# 是否启用断点续传。
# 导入数据时，TiDB Lightning 会记录当前表导入的进度。
# 所以即使 Lightning 或其他组件异常退出，在重启时也可以避免重复再导入已完成的数据。
enable = true
# 存储断点的数据库名称。
schema = "tidb_lightning_checkpoint"
# 存储断点的方式。
#  - file：存放在本地文件系统。
#  - mysql：存放在兼容 MySQL 的数据库服务器。
driver = "file"

[tikv-importer]
# backend 设置为 local 模式
backend = "local"
# 设置本地临时存储路径
sorted-kv-dir = "/data01/tmpdir"

[mydumper]
# Mydumper 源数据目录。
data-source-dir = "/data01/edoc2v5"
filter = ['edoc2v5.*','!*.rpt_fileoperationcount']
#filter = ['edoc2v5.dms_file','edoc2v5.dms_filever']

[tidb]
# 目标集群的信息。tidb-server 的监听地址，填一个即可。
host = "127.0.0.1"
port = 4000
user = "root"
password = ""
# 表架构信息在从 TiDB 的“状态端口”获取。
status-port = 10080

非常感谢~

keithyzk · 2020 年9 月 25 日 02:48

glorvs · 2020 年9 月 25 日 03:16

麻烦搜索一下 log 里面的 ERROR 或者 WARN 看看具体报的什么问题？目前这个现象看起来像是写 tikv 的时候特别慢？要么是在不断进行失败重试，要么就是网络有问题导致写入速度过慢。

如果是重试过多，那应该是并发的冲突太剧烈导致。可以手动把 lightning 停掉。然后在 tidb-lightning.toml 里面添加如下设置：

[tikv-importer]
range-concurrency = 4

然后重启 lightning, 这样 lightning 应该会接着未完成的任务继续执行

keithyzk · 2020 年9 月 25 日 03:53

好的已经改了，现在在继续恢复

筛选了日志的 WARN 和 ERROR，出现最多的是下边的这些

[2020/09/25 08:12:14.486 +08:00] [WARN] [backend.go:334] ["import spuriously failed, going to retry again"] [engineTag=`edoc2v5`.`dms_filestorageinfo`:-1] [engineUUID=29b9440f-8862-5289-92bd-741057f3b169] [retryCnt=1] [error="split region failed: region=id:944259 start_key:\"t\\200\\000\\000\\000\\000\\000E\\377Q_i\\200\\000\\000\\000\\000\\377\\000\\000\\002\\003\\200\\000\\000\\000\\377\\000\\000\\000\\001\\003\\200\\000\\000\\377\\000\\004\\242\\210\\316\\000\\000\\000\\374\" end_key:\"t\\200\\000\\000\\000\\000\\000E\\377Q_i\\200\\000\\000\\000\\000\\377\\000\\000\\002\\003\\200\\000\\000\\000\\377\\000\\000\\000\\001\\004\\334\\230\\202\\377\\2711\\025\\000\\000\\000\\000\\000\\372\" region_epoch:<conf_ver:131 version:7438 > peers:<id:944261 store_id:14 > peers:<id:944262 store_id:2 > peers:<id:947052 store_id:1 > , err=message:\"EpochNotMatch [region 944259] 944261 epoch changed conf_ver: 131 version: 7439 != conf_ver: 131 version: 7438, retry later\" epoch_not_match:<current_regions:<id:944259 start_key:\"t\\200\\000\\000\\000\\000\\000E\\377Q_i\\200\\000\\000\\000\\000\\377\\000\\000\\002\\003\\200\\000\\000\\000\\377\\000\\000\\000\\001\\003\\200\\000\\000\\377\\000\\004\\261.\\316\\000\\000\\000\\374\" end_key:\"t\\200\\000\\000\\000\\000\\000E\\377Q_i\\200\\000\\000\\000\\000\\377\\000\\000\\002\\003\\200\\000\\000\\000\\377\\000\\000\\000\\001\\004\\334\\230\\202\\377\\2711\\025\\000\\000\\000\\000\\000\\372\" region_epoch:<conf_ver:131 version:7439 > peers:<id:944261 store_id:14 > peers:<id:944262 store_id:2 > peers:<id:947052 store_id:1 > > > "]
[2020/09/25 08:36:08.701 +08:00] [WARN] [local.go:908] ["ingest failed noretry"] [error="non retryable error: peer is not leader for region 950428, leader may None"] [errorVerbose="non retryable error: peer is not leader for region 950428, leader may None\
github.com/pingcap/tidb-lightning/lightning/backend.isIngestRetryable\
\t/data01/tidb-lightning/lightning/backend/local.go:1159\
github.com/pingcap/tidb-lightning/lightning/backend.(*local).WriteAndIngestPairs\
\t/data01/tidb-lightning/lightning/backend/local.go:902\
github.com/pingcap/tidb-lightning/lightning/backend.(*local).writeAndIngestByRange.func1\
\t/data01/tidb-lightning/lightning/backend/local.go:834\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1374"] [meta="{\"uuid\":\"yqW/SmmJTj2bwJEuuMgf7g==\",\"range\":{\"start\":\"dIAAAAAAAEX/oV9ygAAAAAT/vrMAAAAAAAD6\",\"end\":\"dIAAAAAAAEX/oV9ygAAAAAT/wR+0AAAAAAD6\"},\"cf_name\":\"default\",\"region_id\":950428,\"region_epoch\":{\"conf_ver\":122,\"version\":7215}}"][region="{\"Region\":{\"id\":950428,\"start_key\":\"dIAAAAAAAEX/oV9ygAAAAAT/vrMAAAAAAAD6\",\"end_key\":\"dIAAAAAAAEX/oV9ygAAAAAT/wR+1AAAAAAD6\",\"region_epoch\":{\"conf_ver\":122,\"version\":7215},\"peers\":[{\"id\":950429,\"store_id\":11},{\"id\":950430,\"store_id\":1},{\"id\":950431,\"store_id\":2}]},\"Leader\":{\"id\":950429,\"store_id\":11}}"]
[2020/09/25 08:36:08.701 +08:00] [WARN] [local.go:849] ["should retry this range"] [retry=0] [error="non retryable error: peer is not leader for region 950428, leader may None"] [errorVerbose="non retryable error: peer is not leader for region 950428, leader may None\
github.com/pingcap/tidb-lightning/lightning/backend.isIngestRetryable\
\t/data01/tidb-lightning/lightning/backend/local.go:1159\
github.com/pingcap/tidb-lightning/lightning/backend.(*local).WriteAndIngestPairs\
\t/data01/tidb-lightning/lightning/backend/local.go:902\
github.com/pingcap/tidb-lightning/lightning/backend.(*local).writeAndIngestByRange.func1\
\t/data01/tidb-lightning/lightning/backend/local.go:834\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1374"]
[2020/09/25 08:36:10.617 +08:00] [WARN] [local.go:908] ["ingest failed noretry"] [error="non retryable error: peer is not leader for region 950428, leader may None"] [errorVerbose="non retryable error: peer is not leader for region 950428, leader may None\
github.com/pingcap/tidb-lightning/lightning/backend.isIngestRetryable\
\t/data01/tidb-lightning/lightning/backend/local.go:1159\
github.com/pingcap/tidb-lightning/lightning/backend.(*local).WriteAndIngestPairs\
\t/data01/tidb-lightning/lightning/backend/local.go:902\
github.com/pingcap/tidb-lightning/lightning/backend.(*local).writeAndIngestByRange.func1\
\t/data01/tidb-lightning/lightning/backend/local.go:834\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1374"] [meta="{\"uuid\":\"ICrafFtTTSWmyGakjJMMXg==\",\"range\":{\"start\":\"dIAAAAAAAEX/oV9ygAAAAAT/vrMAAAAAAAD6\",\"end\":\"dIAAAAAAAEX/oV9ygAAAAAT/wR+0AAAAAAD6\"},\"cf_name\":\"default\",\"region_id\":950428,\"region_epoch\":{\"conf_ver\":122,\"version\":7215}}"][region="{\"Region\":{\"id\":950428,\"start_key\":\"dIAAAAAAAEX/oV9ygAAAAAT/vrMAAAAAAAD6\",\"end_key\":\"dIAAAAAAAEX/oV9ygAAAAAT/wR+1AAAAAAD6\",\"region_epoch\":{\"conf_ver\":122,\"version\":7215},\"peers\":[{\"id\":950429,\"store_id\":11},{\"id\":950430,\"store_id\":1},{\"id\":950431,\"store_id\":2}]},\"Leader\":{\"id\":950429,\"store_id\":11}}"]
[2020/09/25 08:36:10.617 +08:00] [WARN] [local.go:849] ["should retry this range"] [retry=1] [error="non retryable error: peer is not leader for region 950428, leader may None"] [errorVerbose="non retryable error: peer is not leader for region 950428, leader may None\
github.com/pingcap/tidb-lightning/lightning/backend.isIngestRetryable\
\t/data01/tidb-lightning/lightning/backend/local.go:1159\
github.com/pingcap/tidb-lightning/lightning/backend.(*local).WriteAndIngestPairs\
\t/data01/tidb-lightning/lightning/backend/local.go:902\
github.com/pingcap/tidb-lightning/lightning/backend.(*local).writeAndIngestByRange.func1\
\t/data01/tidb-lightning/lightning/backend/local.go:834\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1374"]
[2020/09/25 08:36:12.411 +08:00] [WARN] [local.go:908] ["ingest failed noretry"] [error="non retryable error: peer is not leader for region 950428, leader may None"] [errorVerbose="non retryable error: peer is not leader for region 950428, leader may None\
github.com/pingcap/tidb-lightning/lightning/backend.isIngestRetryable\
\t/data01/tidb-lightning/lightning/backend/local.go:1159\
github.com/pingcap/tidb-lightning/lightning/backend.(*local).WriteAndIngestPairs\
\t/data01/tidb-lightning/lightning/backend/local.go:902\
github.com/pingcap/tidb-lightning/lightning/backend.(*local).writeAndIngestByRange.func1\
\t/data01/tidb-lightning/lightning/backend/local.go:834\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1374"] [meta="{\"uuid\":\"6i/L57QKTI2srpqVvtMnMg==\",\"range\":{\"start\":\"dIAAAAAAAEX/oV9ygAAAAAT/vrMAAAAAAAD6\",\"end\":\"dIAAAAAAAEX/oV9ygAAAAAT/wR+0AAAAAAD6\"},\"cf_name\":\"default\",\"region_id\":950428,\"region_epoch\":{\"conf_ver\":122,\"version\":7215}}"][region="{\"Region\":{\"id\":950428,\"start_key\":\"dIAAAAAAAEX/oV9ygAAAAAT/vrMAAAAAAAD6\",\"end_key\":\"dIAAAAAAAEX/oV9ygAAAAAT/wR+1AAAAAAD6\",\"region_epoch\":{\"conf_ver\":122,\"version\":7215},\"peers\":[{\"id\":950429,\"store_id\":11},{\"id\":950430,\"store_id\":1},{\"id\":950431,\"store_id\":2}]},\"Leader\":{\"id\":950429,\"store_id\":11}}"]
[2020/09/25 08:36:12.411 +08:00] [WARN] [local.go:849] ["should retry this range"] [retry=2] [error="non retryable error: peer is not leader for region 950428, leader may None"] [errorVerbose="non retryable error: peer is not leader for region 950428, leader may None\
github.com/pingcap/tidb-lightning/lightning/backend.isIngestRetryable\
\t/data01/tidb-lightning/lightning/backend/local.go:1159\
github.com/pingcap/tidb-lightning/lightning/backend.(*local).WriteAndIngestPairs\
\t/data01/tidb-lightning/lightning/backend/local.go:902\
github.com/pingcap/tidb-lightning/lightning/backend.(*local).writeAndIngestByRange.func1\
\t/data01/tidb-lightning/lightning/backend/local.go:834\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1374"]
[2020/09/25 08:36:12.411 +08:00] [WARN] [local.go:952] ["write and ingest by range failed"] ["retry time"=1] [error="non retryable error: peer is not leader for region 950428, leader may None"] [errorVerbose="non retryable error: peer is not leader for region 950428, leader may None\
github.com/pingcap/tidb-lightning/lightning/backend.isIngestRetryable\
\t/data01/tidb-lightning/lightning/backend/local.go:1159\
github.com/pingcap/tidb-lightning/lightning/backend.(*local).WriteAndIngestPairs\
\t/data01/tidb-lightning/lightning/backend/local.go:902\
github.com/pingcap/tidb-lightning/lightning/backend.(*local).writeAndIngestByRange.func1\
\t/data01/tidb-lightning/lightning/backend/local.go:834\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1374"]
[2020/09/25 08:37:13.358 +08:00] [ERROR] [local.go:924] ["all retry ingest failed"] ["ingest meta"="{\"uuid\":\"CfN+qVJPS5myHFdxN++fmQ==\",\"range\":{\"start\":\"dIAAAAAAAEX/oV9ygAAAAAr/fQaDAAAAAAD6\",\"end\":\"dIAAAAAAAEX/oV9ygAAAAAr/fQaDAAAAAAD6\"},\"cf_name\":\"default\",\"region_id\":948185,\"region_epoch\":{\"conf_ver\":123,\"version\":6840}}"] [error="epoch not match: EpochNotMatch conf_ver: 123 version: 6840 != conf_ver: 125 version: 6840"] [errorVerbose="epoch not match: EpochNotMatch conf_ver: 123 version: 6840 != conf_ver: 125 version: 6840\
github.com/pingcap/tidb-lightning/lightning/backend.isIngestRetryable\
\t/data01/tidb-lightning/lightning/backend/local.go:1157\
github.com/pingcap/tidb-lightning/lightning/backend.(*local).WriteAndIngestPairs\
\t/data01/tidb-lightning/lightning/backend/local.go:902\
github.com/pingcap/tidb-lightning/lightning/backend.(*local).writeAndIngestByRange.func1\
\t/data01/tidb-lightning/lightning/backend/local.go:834\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1374"]
[2020/09/25 08:37:13.358 +08:00] [WARN] [local.go:849] ["should retry this range"] [retry=0] [error="epoch not match: EpochNotMatch conf_ver:123 version: 6840 != conf_ver: 125 version: 6840"] [errorVerbose="epoch not match: EpochNotMatch conf_ver: 123 version: 6840 != conf_ver: 125version: 6840\
github.com/pingcap/tidb-lightning/lightning/backend.isIngestRetryable\
\t/data01/tidb-lightning/lightning/backend/local.go:1157\
github.com/pingcap/tidb-lightning/lightning/backend.(*local).WriteAndIngestPairs\
\t/data01/tidb-lightning/lightning/backend/local.go:902\
github.com/pingcap/tidb-lightning/lightning/backend.(*local).writeAndIngestByRange.func1\
\t/data01/tidb-lightning/lightning/backend/local.go:834\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1374"]
[2020/09/25 08:37:14.519 +08:00] [ERROR] [local.go:924] ["all retry ingest failed"] ["ingest meta"="{\"uuid\":\"5BY888kVSyysHcknOaFzDg==\",\"range\":{\"start\":\"dIAAAAAAAEX/oV9ygAAAAAr/fQaEAAAAAAD6\",\"end\":\"dIAAAAAAAEX/oV9ygAAAAAr/f3IKAAAAAAD6\"},\"cf_name\":\"default\",\"region_id\":948185,\"region_epoch\":{\"conf_ver\":124,\"version\":6840}}"] [error="epoch not match: EpochNotMatch conf_ver: 124 version: 6840 != conf_ver: 125 version: 6840"] [errorVerbose="epoch not match: EpochNotMatch conf_ver: 124 version: 6840 != conf_ver: 125 version: 6840\
github.com/pingcap/tidb-lightning/lightning/backend.isIngestRetryable\
\t/data01/tidb-lightning/lightning/backend/local.go:1157\
github.com/pingcap/tidb-lightning/lightning/backend.(*local).WriteAndIngestPairs\
\t/data01/tidb-lightning/lightning/backend/local.go:902\
github.com/pingcap/tidb-lightning/lightning/backend.(*local).writeAndIngestByRange.func1\
\t/data01/tidb-lightning/lightning/backend/local.go:834\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1374"]
[2020/09/25 08:37:14.519 +08:00] [WARN] [local.go:849] ["should retry this range"] [retry=0] [error="epoch not match: EpochNotMatch conf_ver:124 version: 6840 != conf_ver: 125 version: 6840"] [errorVerbose="epoch not match: EpochNotMatch conf_ver: 124 version: 6840 != conf_ver: 125version: 6840\
github.com/pingcap/tidb-lightning/lightning/backend.isIngestRetryable\
\t/data01/tidb-lightning/lightning/backend/local.go:1157\
github.com/pingcap/tidb-lightning/lightning/backend.(*local).WriteAndIngestPairs\
\t/data01/tidb-lightning/lightning/backend/local.go:902\
github.com/pingcap/tidb-lightning/lightning/backend.(*local).writeAndIngestByRange.func1\
\t/data01/tidb-lightning/lightning/backend/local.go:834\
runtime.goexit\
\t/usr/local/go/src/runtime/asm_amd64.s:1374"]
[2020/09/25 08:52:00.321 +08:00] [WARN] [local.go:399] ["keys within region is empty, skip ingest"]
[2020/09/25 08:52:00.321 +08:00] [WARN] [local.go:399] ["keys within region is empty, skip ingest"]
[2020/09/25 08:52:00.322 +08:00] [WARN] [local.go:399] ["keys within region is empty, skip ingest"]
[2020/09/25 08:52:00.322 +08:00] [WARN] [local.go:399] ["keys within region is empty, skip ingest"]
[2020/09/25 08:52:00.323 +08:00] [WARN] [local.go:399] ["keys within region is empty, skip ingest"]
[2020/09/25 08:52:00.323 +08:00] [WARN] [local.go:399] ["keys within region is empty, skip ingest"]

glorvs · 2020 年9 月 25 日 04:16

看起来确实是 ingest 的时候，冲突过于剧烈导致，应该吧 range-concurrency 调低之后会改善不少

keithyzk · 2020 年9 月 25 日 06:56

你好，速度还是很慢😭。我用的是上边发的 tidb-lightning 的 new-collation 分支编译安装的。和这个有关系吗？

glorvs · 2020 年9 月 25 日 06:58

应该没有，new collation 解决的是 kv 编码的问题，这个是因为 ingest 的问题。那麻烦是一下吧 range-concurrenty 改成 1 看看

system · 2022 年10 月 31 日 19:04

此话题已在最后回复的 1 分钟后被自动关闭。不再允许新回复。