查询单个数据表时出现 other error: default not found: key maybe read truncated/dropped table data

【 TiDB 使用环境`】生产环境
【 TiDB 版本】v5.4.0
【遇到的问题】查询该表数据的时候 出现key xxxxxxxxxxxxx maybe read truncated/dropped table data
image

出现这个问题之前,有没有做什么操作?
或者有没有什么背景和环境信息,可以共享出来?

表结构,表数据量,查询条件,执行计划这些,看看。结果集很大,快照保留时间这些参数。

一个月之前做过一次升级 升级到最新的这个版本
根据 pd-ctl 查询这个 key的情况如下

[sinyang@9-44-32-123 ~/ctl]$ ./pd-ctl -u http://9.22.58.223:15120 region key 7480000000000000405F728000000019128722
{
“id”: 1067551817,
“start_key”: “6D44444C4A6F6248FF69FF73746F727900FF0000FC0000000000FF0000680000000000FF00A5DFFF00000000FF00000000F7000000FC”,
“end_key”: “7480000000000000FF0D5F728000000000FF0000740000000000FA”,
“epoch”: {
“conf_ver”: 26174,
“version”: 2626
},
“peers”: [
{
“id”: 1067551819,
“store_id”: 1054234453,
“role_name”: “Voter”
},
{
“id”: 1067551820,
“store_id”: 3063008,
“role_name”: “Voter”
},
{
“id”: 1067552010,
“store_id”: 1056792949,
“role_name”: “Voter”
}
],
“leader”: {
“id”: 1067552010,
“store_id”: 1056792949,
“role_name”: “Voter”
},
“written_bytes”: 0,
“read_bytes”: 210972,
“written_keys”: 0,
“read_keys”: 233,
“approximate_size”: 94,
“approximate_keys”: 69675
}

表结构如下
CREATE TABLE cube_kd_all_2020_new (
dt int(11) NOT NULL,
group_key varchar(45) NOT NULL,
utype int(11) NOT NULL,
bucket_id int(11) NOT NULL,
gender int(11) NOT NULL,
age_bin int(11) NOT NULL,
city_level int(11) NOT NULL,
week_act int(11) NOT NULL,
act_month int(11) NOT NULL,
act_30 int(11) NOT NULL,
plat int(11) NOT NULL,
article_new int(11) NOT NULL,
last_week_article_new int(11) NOT NULL,
last_month_article_new int(11) NOT NULL,
last_week_duration_type int(11) NOT NULL,
last_month_duration_type int(11) NOT NULL,
article_active int(11) DEFAULT NULL,
dim_back1 int(11) DEFAULT NULL,
dim_back2 int(11) DEFAULT NULL,
qqbase_week_act int(11) NOT NULL COMMENT ‘手Q上周活跃度’,
video_new_click int(11) NOT NULL,
last_week_video_new_click int(11) NOT NULL,
last_month_video_new_click int(11) NOT NULL,
dim_back3 int(11) DEFAULT NULL,
dim_back4 int(11) DEFAULT NULL,
pic_uv bigint(20) DEFAULT NULL,
pic_pv bigint(20) DEFAULT NULL,
video_uv bigint(20) DEFAULT NULL,
video_pv bigint(20) DEFAULT NULL,
big_pic_pv bigint(20) DEFAULT NULL,
all_uv bigint(20) DEFAULT NULL,
expose_uv_dedup bigint(20) DEFAULT NULL,
expose_pv_dedup bigint(20) DEFAULT NULL,
click_uv_dedup bigint(20) DEFAULT NULL,
click_pv_dedup bigint(20) DEFAULT NULL,
up_flush_uv bigint(20) DEFAULT NULL,
up_flush_pv bigint(20) DEFAULT NULL,
down_flush_uv bigint(20) DEFAULT NULL,
down_flush_pv bigint(20) DEFAULT NULL,
duration_uv bigint(20) DEFAULT NULL,
duration_sum bigint(20) DEFAULT NULL,
push_expose_uv_dedup bigint(20) DEFAULT NULL,
push_expose_pv_dedup bigint(20) DEFAULT NULL,
push_click_uv_dedup bigint(20) DEFAULT NULL,
push_click_pv_dedup bigint(20) DEFAULT NULL,
dau3a_uv bigint(20) DEFAULT NULL,
north_expose bigint(20) DEFAULT NULL,
click_and_duration_1min_uv bigint(20) DEFAULT NULL,
click_and_interact_uv bigint(20) DEFAULT NULL,
cost_micros bigint(20) DEFAULT NULL,
qqbase_dau bigint(20) DEFAULT NULL,
qqbase_dau_day1 bigint(20) DEFAULT NULL,
qqbase_back1 bigint(20) DEFAULT NULL,
qqbase_total_duration bigint(20) DEFAULT NULL,
cut_front_cnt bigint(20) DEFAULT NULL,
watch_duration bigint(20) DEFAULT NULL,
watch_uv bigint(20) DEFAULT NULL,
small_duration bigint(20) DEFAULT NULL,
small_dur_uv bigint(20) DEFAULT NULL,
short_duration bigint(20) DEFAULT NULL,
short_dur_uv bigint(20) DEFAULT NULL,
small_vv bigint(20) DEFAULT NULL,
short_vv bigint(20) DEFAULT NULL,
d56_7408_dur bigint(20) DEFAULT NULL,
d56_7408_dur_uv bigint(20) DEFAULT NULL,
d56_7409_dur bigint(20) DEFAULT NULL,
d56_7409_dur_uv bigint(20) DEFAULT NULL,
channel0_expose_pv bigint(20) DEFAULT NULL,
channel0_expose_small_pv bigint(20) DEFAULT NULL,
channel0_expose_short_pv bigint(20) DEFAULT NULL,
channel56_expose_pv bigint(20) DEFAULT NULL,
channel0_click_pv bigint(20) DEFAULT NULL,
channel0_click_small_pv bigint(20) DEFAULT NULL,
channel0_click_short_pv bigint(20) DEFAULT NULL,
channel56_click_pv bigint(20) DEFAULT NULL,
first_watch_duration bigint(20) DEFAULT NULL,
first_watch_uv bigint(20) DEFAULT NULL,
pic7626_pv bigint(20) DEFAULT NULL,
pic7625_pv bigint(20) DEFAULT NULL,
d56_7408_vv bigint(20) DEFAULT NULL,
d56_7409_vv bigint(20) DEFAULT NULL,
d56_7408_short0_vv bigint(20) DEFAULT NULL,
d56_7408_short1_vv bigint(20) DEFAULT NULL,
d56_7409_short0_vv bigint(20) DEFAULT NULL,
d56_7409_short1_vv bigint(20) DEFAULT NULL,
channel0_expose_uv bigint(20) DEFAULT NULL,
channel0_expose_small_uv bigint(20) DEFAULT NULL,
channel0_expose_short_uv bigint(20) DEFAULT NULL,
channel0_click_uv bigint(20) DEFAULT NULL,
channel0_click_small_uv bigint(20) DEFAULT NULL,
channel0_click_short_uv bigint(20) DEFAULT NULL,
ls_short_content_duration_total bigint(20) DEFAULT NULL,
ls_valid_follow_act_cnt bigint(20) DEFAULT NULL,
ls_unfollow_act_cnt bigint(20) DEFAULT NULL,
ls_valid_follow_7_cnt_0_to_1 bigint(20) DEFAULT NULL,
ls_valid_follow_7_cnt_1 bigint(20) DEFAULT NULL,
ls_valid_follow_7_cnt_5 bigint(20) DEFAULT NULL,
ls_valid_follow_7_cnt_10 bigint(20) DEFAULT NULL,
ls_valid_follow_7_cnt_30 bigint(20) DEFAULT NULL,
ls_article_duration bigint(20) DEFAULT NULL,
ls_short_video_duration bigint(20) DEFAULT NULL,
ls_small_video_duration bigint(20) DEFAULT NULL,
ls_short_content_duration bigint(20) DEFAULT NULL,
pic_dur bigint(20) DEFAULT NULL,
pic_dur_uv bigint(20) DEFAULT NULL,
etl_stamp varchar(26) DEFAULT NULL,
kd_tab_dau3a bigint(20) DEFAULT NULL,
kd_tab_pic_pv bigint(20) DEFAULT NULL,
kd_tab_video_pv bigint(20) DEFAULT NULL,
kd_tab_feeds_exp_pv bigint(20) DEFAULT NULL,
kd_tab_feeds_exp_uv bigint(20) DEFAULT NULL,
kd_tab_feeds_clk_pv bigint(20) DEFAULT NULL,
kd_tab_feeds_clk_uv bigint(20) DEFAULT NULL,
kd_tab_duration bigint(20) DEFAULT NULL,
kd_tab_duration_uv bigint(20) DEFAULT NULL,
kd_tab_dau3a_yes bigint(20) DEFAULT NULL,
kd_tab_dau3a_back bigint(20) DEFAULT NULL,
kd_tab_duration_valid bigint(20) DEFAULT NULL,
kd_tab_duration_valid_uv bigint(20) DEFAULT NULL,
week_qq_register bigint(20) DEFAULT ‘-1’,
month_qq_register bigint(20) DEFAULT ‘-1’,
daily_activity_in7 bigint(20) DEFAULT ‘-1’,
daily_activity_in14 bigint(20) DEFAULT ‘-1’,
dau3a_yes bigint(20) DEFAULT NULL,
dau3a_back bigint(20) DEFAULT NULL,
exposure_cnt bigint(20) DEFAULT NULL,
click_cnt bigint(20) DEFAULT NULL,
total_cost_micros bigint(20) DEFAULT NULL,
p0_exposure_cnt bigint(20) DEFAULT NULL,
p0_click_cnt bigint(20) DEFAULT NULL,
p0_real_cost bigint(20) DEFAULT NULL,
p1_exposure_cnt bigint(20) DEFAULT NULL,
p1_click_cnt bigint(20) DEFAULT NULL,
p1_real_cost bigint(20) DEFAULT NULL,
p2_exposure_cnt bigint(20) DEFAULT NULL,
p2_click_cnt bigint(20) DEFAULT NULL,
p2_real_cost bigint(20) DEFAULT NULL,
p3_exposure_cnt bigint(20) DEFAULT NULL,
p3_click_cnt bigint(20) DEFAULT NULL,
p3_real_cost bigint(20) DEFAULT NULL,
p4_exposure_cnt bigint(20) DEFAULT NULL,
p4_click_cnt bigint(20) DEFAULT NULL,
p4_real_cost bigint(20) DEFAULT NULL,
p5_exposure_cnt bigint(20) DEFAULT NULL,
p5_click_cnt bigint(20) DEFAULT NULL,
p5_real_cost bigint(20) DEFAULT NULL,
p6_exposure_cnt bigint(20) DEFAULT NULL,
p6_click_cnt bigint(20) DEFAULT NULL,
p6_real_cost bigint(20) DEFAULT NULL,
p7_exposure_cnt bigint(20) DEFAULT NULL,
p7_click_cnt bigint(20) DEFAULT NULL,
p7_real_cost bigint(20) DEFAULT NULL,
p8_exposure_cnt bigint(20) DEFAULT NULL,
p8_click_cnt bigint(20) DEFAULT NULL,
p8_real_cost bigint(20) DEFAULT NULL,
p9_exposure_cnt bigint(20) DEFAULT NULL,
p9_click_cnt bigint(20) DEFAULT NULL,
p9_real_cost bigint(20) DEFAULT NULL,
valid_follow_cnt_7 bigint(20) DEFAULT ‘-1’,
zan_pv bigint(20) DEFAULT NULL,
zan_uv bigint(20) DEFAULT NULL,
valid_follow_cnt_30 bigint(20) DEFAULT ‘-1’,
friend_enter_uv bigint(20) DEFAULT NULL COMMENT ‘好友频道-频道总进入’,
friend_dau bigint(20) DEFAULT NULL COMMENT ‘好友频道-dau’,
friend_feeds_all_dur bigint(20) DEFAULT NULL COMMENT ‘好友频道-时长’,
friend_dianzan_click_pv bigint(20) DEFAULT NULL COMMENT ‘好友频道-点赞按钮点击pv’,
friend_dianzan_click_uv bigint(20) DEFAULT NULL COMMENT ‘好友频道-点赞按钮点击uv’,
friend_guanzhu_click_pv bigint(20) DEFAULT NULL COMMENT ‘好友频道-关注按钮点击pv’,
friend_guanzhu_click_uv bigint(20) DEFAULT NULL COMMENT ‘好友频道-关注按钮点击uv’,
friend_video_play_vv bigint(20) DEFAULT NULL COMMENT ‘好友频道-视频播放vv’,
friend_video_play_uv bigint(20) DEFAULT NULL COMMENT ‘好友频道-视频播放uv’,
friend_feeds_video_click_pv bigint(20) DEFAULT NULL COMMENT ‘好友频道-视频feeds点击pv’,
friend_feeds_video_click_uv bigint(20) DEFAULT NULL COMMENT ‘好友频道-视频feeds点击uv’,
content_zan_pv bigint(20) DEFAULT NULL,
content_zan_uv bigint(20) DEFAULT NULL,
comment_zan_pv bigint(20) DEFAULT NULL,
comment_zan_uv bigint(20) DEFAULT NULL,
focus_pv bigint(20) DEFAULT NULL,
focus_uv bigint(20) DEFAULT NULL,
all_channel_share_pv bigint(20) DEFAULT NULL,
all_channel_share_uv bigint(20) DEFAULT NULL,
cmm_biu_card_expose_pv bigint(20) DEFAULT NULL,
cmm_biu_card_expose_uv bigint(20) DEFAULT NULL,
cmm_biu_card_click_pv bigint(20) DEFAULT NULL,
cmm_biu_card_click_uv bigint(20) DEFAULT NULL,
comment_expose_pv bigint(20) DEFAULT NULL,
comment_expose_uv bigint(20) DEFAULT NULL,
first_comment_expose_pv bigint(20) DEFAULT NULL,
first_comment_expose_uv bigint(20) DEFAULT NULL,
first_comment_pv bigint(20) DEFAULT NULL,
first_comment_uv bigint(20) DEFAULT NULL,
second_comment_pv bigint(20) DEFAULT NULL,
second_comment_uv bigint(20) DEFAULT NULL,
comment_duration bigint(20) DEFAULT NULL,
comment_jubao_pv bigint(20) DEFAULT NULL,
biu_expose_pv bigint(20) DEFAULT NULL,
biu_expose_uv bigint(20) DEFAULT NULL,
biu_pv bigint(20) DEFAULT NULL,
biu_uv bigint(20) DEFAULT NULL,
pic_cmm_biu_uv bigint(20) DEFAULT NULL,
video_cmm_biu_uv bigint(20) DEFAULT NULL,
cmm_biu_card_valid_expose_pv bigint(20) DEFAULT NULL,
cmm_biu_card_valid_expose_uv bigint(20) DEFAULT NULL,
cmm_card_duration bigint(20) DEFAULT NULL,
cmm_card_zan_pv bigint(20) DEFAULT NULL,
cmm_card_second_zan_pv bigint(20) DEFAULT NULL,
cmm_card_page_pv bigint(20) DEFAULT NULL,
cmm_card_page_uv bigint(20) DEFAULT NULL,
cmm_card_page_duration bigint(20) DEFAULT NULL,
cmm_card_page_click_pv bigint(20) DEFAULT NULL,
cmm_card_page_click_uv bigint(20) DEFAULT NULL,
cmm_card_page_reply_pv bigint(20) DEFAULT NULL,
cmm_card_page_expose_pv bigint(20) DEFAULT NULL,
cmm_card_page_expose_uv bigint(20) DEFAULT NULL,
cmm_card_share_click_pv bigint(20) DEFAULT NULL,
cmm_card_share_clicK_uv bigint(20) DEFAULT NULL,
cmm_page_share_click_pv bigint(20) DEFAULT NULL,
cmm_page_share_click_uv bigint(20) DEFAULT NULL,
cmm_card2_share_click_pv bigint(20) DEFAULT NULL,
cmm_card2_share_click_uv bigint(20) DEFAULT NULL,
comment_first_zan_pv bigint(20) DEFAULT NULL,
comment_first_zan_uv bigint(20) DEFAULT NULL,
comment_second_zan_pv bigint(20) DEFAULT NULL,
comment_second_zan_uv bigint(20) DEFAULT NULL,
cancel_focus_pv bigint(20) DEFAULT NULL,
cancel_focus_uv bigint(20) DEFAULT NULL,
video_share_pv bigint(20) DEFAULT NULL,
video_share_uv bigint(20) DEFAULT NULL,
pic_share_pv bigint(20) DEFAULT NULL,
pic_share_uv bigint(20) DEFAULT NULL,
new_video_share_pv bigint(20) DEFAULT NULL,
new_video_share_uv bigint(20) DEFAULT NULL,
new_pic_share_pv bigint(20) DEFAULT NULL,
new_pic_share_uv bigint(20) DEFAULT NULL,
new_all_channel_share_pv bigint(20) DEFAULT NULL,
new_all_channel_share_uv bigint(20) DEFAULT NULL,
video_share_page_expose_pv bigint(20) DEFAULT NULL,
video_share_page_expose_uv bigint(20) DEFAULT NULL,
first_expose_pv bigint(20) DEFAULT NULL,
first_expose_uv bigint(20) DEFAULT NULL,
new_comment_pv bigint(20) DEFAULT NULL,
new_comment_uv bigint(20) DEFAULT NULL,
first_new_comment_pv bigint(20) DEFAULT NULL,
first_new_comment_uv bigint(20) DEFAULT NULL,
second_new_comment_pv bigint(20) DEFAULT NULL,
second_new_comment_uv bigint(20) DEFAULT NULL,
read_time bigint(20) DEFAULT NULL,
first_comment_zan_pv bigint(20) DEFAULT NULL,
first_comment_zan_uv bigint(20) DEFAULT NULL,
first_comment_biu_pv bigint(20) DEFAULT NULL,
first_comment_biu_uv bigint(20) DEFAULT NULL,
home_detail_exposure_cnt bigint(20) DEFAULT NULL,
home_detail_click_cnt bigint(20) DEFAULT NULL,
home_detail_cost bigint(20) DEFAULT NULL,
heyue_expose_pv bigint(20) DEFAULT ‘0’,
jingpin_expose_pv bigint(20) DEFAULT ‘0’,
ziying_expose_pv bigint(20) DEFAULT ‘0’,
ieg_expose_pv bigint(20) DEFAULT ‘0’,
friend_chann_exposure_cnt bigint(20) DEFAULT ‘0’,
friend_chann_click_cnt bigint(20) DEFAULT ‘0’,
friend_chann_cost bigint(20) DEFAULT ‘0’,
KEY etl_stamp_index (etl_stamp),
KEY query_index_0 (group_key, utype, bucket_id, dt),
KEY dt_index (dt, group_key)
) ENGINE = InnoDB DEFAULT CHARSET = utf8mb4 COLLATE = utf8mb4_bin

表数据量在 35亿左右

GC 时间

是一个大宽表

不太明白 这个startkey 和 endkey为啥长度不一样 一般长的应该是索引的key吧 短的是数据的key

具体导出的错误信息如下
{“level”:“INFO”,“time”:“2022/05/13 13:53:52.746 +08:00”,“caller”:“conn.go:1115”,“message”:“command dispatched failed”,“conn”:89,“connInfo”:“id:89, addr:9.37.143.215:56347 status:11, collation:utf8mb4_general_ci, user:mdbroot”,“command”:“Query”,“status”:“inTxn:1, autocommit:1”,“sql”:"SELECT * FROM drilldown_data.cube_kd_all_2020_new ORDER BY _tidb_rowid",“txn_mode”:“OPTIMISTIC”,“err”:“other error: default not found: key:7480000000000000405F728000000019128722, maybe read truncated/dropped table data?\ngithub.com/pingcap/tidb/store/copr.(*copIteratorWorker).handleCopResponse\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/store/copr/coprocessor.go:969\ngithub.com/pingcap/tidb/store/copr.(*copIteratorWorker).handleTaskOnce\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/store/copr/coprocessor.go:784\ngithub.com/pingcap/tidb/store/copr.(*copIteratorWorker).handleTask\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/store/copr/coprocessor.go:668\ngithub.com/pingcap/tidb/store/copr.(*copIteratorWorker).run\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/store/copr/coprocessor.go:410\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371”}

之前是什么版本?


参考下这个帖子吧,感觉和你的问题差不多

但是,我不知道是什么操作导致这个问题生产的,按照分析就是缺 版本数据了,丢数据了…

嗯嗯 好的

v4.0的

字段设置了默认值,插入的时候是不是赋值成NULL了。