JSON查询异常，部分数据查询不到

🌞 建议反馈 产品缺陷
41

在有问题的表上执行TiFlash数据校验，校验失败，完整的输出如下：

 /data01/deploy/bin/tiflash/tiflash dttool inspect --workdir /data01/tiflash/data/data/t_138302/stable --file-id 63154 --check --imitative
[2024/09/13 11:10:12.102 +08:00] [INFO] [ComputeLabelHolder.cpp:47] ["get cluster id: unknown"] [thread_id=1]
[2024/09/13 11:10:12.102 +08:00] [INFO] [ComputeLabelHolder.cpp:55] ["get process id: unknown"] [thread_id=1]
[2024/09/13 11:10:12.103 +08:00] [INFO] [ComputeLabelHolder.cpp:47] ["get cluster id: unknown"] [thread_id=1]
[2024/09/13 11:10:12.103 +08:00] [INFO] [ComputeLabelHolder.cpp:55] ["get process id: unknown"] [thread_id=1]
[2024/09/13 11:10:12.103 +08:00] [INFO] [ComputeLabelHolder.cpp:47] ["get cluster id: unknown"] [thread_id=1]
[2024/09/13 11:10:12.103 +08:00] [INFO] [ComputeLabelHolder.cpp:55] ["get process id: unknown"] [thread_id=1]
[2024/09/13 11:10:12.103 +08:00] [INFO] [ComputeLabelHolder.cpp:47] ["get cluster id: unknown"] [thread_id=1]
[2024/09/13 11:10:12.103 +08:00] [INFO] [ComputeLabelHolder.cpp:55] ["get process id: unknown"] [thread_id=1]
[2024/09/13 11:10:12.103 +08:00] [INFO] [BackgroundProcessingPool.cpp:90] ["Create BackgroundProcessingPool, prefix=bg- n_threads=16"] [thread_id=1]
[2024/09/13 11:10:12.106 +08:00] [INFO] [BackgroundProcessingPool.cpp:90] ["Create BackgroundProcessingPool, prefix=bg-block- n_threads=16"] [thread_id=1]
[2024/09/13 11:10:12.110 +08:00] [INFO] [PathCapacityMetrics.cpp:93] ["Init capacity [path=/data01/tiflash/data/data/t_138302/stable] [capacity=0.00 B]"] [thread_id=1]
[2024/09/13 11:10:12.110 +08:00] [INFO] [PageStorageImpl.cpp:54] ["PageStorageImpl start. Config{ PageStorageConfig {blob_file_limit_size: 268435456, blob_spacemap_type: 2, blob_heavy_gc_valid_rate: 0.500, blob_heavy_gc_valid_rate_raft_data: 0.050, blob_block_alignment_bytes: 0, wal_roll_size: 2097152, wal_max_persisted_log_files: 4} }"] [source=__global__.log] [thread_id=1]
[2024/09/13 11:10:12.110 +08:00] [INFO] [PageStorageImpl.cpp:54] ["PageStorageImpl start. Config{ PageStorageConfig {blob_file_limit_size: 268435456, blob_spacemap_type: 2, blob_heavy_gc_valid_rate: 0.500, blob_heavy_gc_valid_rate_raft_data: 0.050, blob_block_alignment_bytes: 0, wal_roll_size: 2097152, wal_max_persisted_log_files: 4} }"] [source=__global__.data] [thread_id=1]
[2024/09/13 11:10:12.110 +08:00] [INFO] [PageStorageImpl.cpp:54] ["PageStorageImpl start. Config{ PageStorageConfig {blob_file_limit_size: 268435456, blob_spacemap_type: 2, blob_heavy_gc_valid_rate: 0.500, blob_heavy_gc_valid_rate_raft_data: 0.050, blob_block_alignment_bytes: 0, wal_roll_size: 2097152, wal_max_persisted_log_files: 4} }"] [source=__global__.meta] [thread_id=1]
[2024/09/13 11:10:12.111 +08:00] [INFO] [PageDirectoryFactory.cpp:68] ["PageDirectory restored, max_page_id=0 max_applied_ver=0"] [source=__global__.log] [thread_id=1]
[2024/09/13 11:10:12.111 +08:00] [INFO] [PageDirectoryFactory.cpp:68] ["PageDirectory restored, max_page_id=0 max_applied_ver=0"] [source=__global__.data] [thread_id=1]
[2024/09/13 11:10:12.111 +08:00] [INFO] [PageDirectoryFactory.cpp:68] ["PageDirectory restored, max_page_id=0 max_applied_ver=0"] [source=__global__.meta] [thread_id=1]
[2024/09/13 11:10:12.111 +08:00] [INFO] [KVStore.cpp:70] ["KVStore inited, eager_raft_log_gc_enabled=false"] [thread_id=1]
[2024/09/13 11:10:12.111 +08:00] [INFO] [MinTSOScheduler.cpp:73] ["thread_hard_limit is 10000, thread_soft_limit is 5000, and active_set_soft_limit is 128 in MinTSOScheduler."] [thread_id=1]
[2024/09/13 11:10:12.111 +08:00] [INFO] [PageStorageImpl.cpp:54] ["PageStorageImpl start. Config{ PageStorageConfig {blob_file_limit_size: 268435456, blob_spacemap_type: 2, blob_heavy_gc_valid_rate: 0.500, blob_heavy_gc_valid_rate_raft_data: 0.050, blob_block_alignment_bytes: 0, wal_roll_size: 2097152, wal_max_persisted_log_files: 4} }"] [source=RegionPersister] [thread_id=1]
[2024/09/13 11:10:12.111 +08:00] [INFO] [PageDirectoryFactory.cpp:68] ["PageDirectory restored, max_page_id=0 max_applied_ver=0"] [source=RegionPersister] [thread_id=1]
[2024/09/13 11:10:12.111 +08:00] [INFO] [RegionPersister.cpp:381] ["RegionPersister running. Current Run Mode is ONLY_V3"] [thread_id=1]
[2024/09/13 11:10:12.112 +08:00] [INFO] [KVStore.cpp:95] ["Restored 0 regions"] [thread_id=1]
[2024/09/13 11:10:12.112 +08:00] [INFO] [RegionTable.cpp:122] ["RegionTable restore start"] [thread_id=1]
[2024/09/13 11:10:12.112 +08:00] [INFO] [RegionTable.cpp:127] ["RegionTable restore end, n_tables=0"] [thread_id=1]
[2024/09/13 11:10:12.112 +08:00] [INFO] [DTToolInspect.cpp:51] ["bytes on disk: 1525267"] [source=DTToolInspect] [thread_id=1]
[2024/09/13 11:10:12.112 +08:00] [INFO] [DTToolInspect.cpp:57] ["with new checksum: true"] [source=DTToolInspect] [thread_id=1]
[2024/09/13 11:10:12.112 +08:00] [INFO] [DTToolInspect.cpp:73] ["checksum algorithm: xxh3"] [source=DTToolInspect] [thread_id=1]
[2024/09/13 11:10:12.112 +08:00] [INFO] [DTToolInspect.cpp:78] ["creation_build_time: 2024-05-24 06:40:59"] [source=DTToolInspect] [thread_id=1]
[2024/09/13 11:10:12.112 +08:00] [INFO] [DTToolInspect.cpp:78] ["creation_commit_hash: c1838001167c8ba83af759085a71ad61e6c2a5af"] [source=DTToolInspect] [thread_id=1]
[2024/09/13 11:10:12.112 +08:00] [INFO] [DTToolInspect.cpp:78] ["creation_edition: Community"] [source=DTToolInspect] [thread_id=1]
[2024/09/13 11:10:12.112 +08:00] [INFO] [DTToolInspect.cpp:78] ["creation_release_version: v8.1.0"] [source=DTToolInspect] [thread_id=1]
[2024/09/13 11:10:12.112 +08:00] [INFO] [DTToolInspect.cpp:78] ["creation_version: 8.1.0"] [source=DTToolInspect] [thread_id=1]
[2024/09/13 11:10:12.112 +08:00] [INFO] [DTToolInspect.cpp:95] ["checking full_path is /data01/tiflash/data/data/t_138302/stable/dmf_63154/20.dat: "] [source=DTToolInspect] [thread_id=1]
[2024/09/13 11:10:12.113 +08:00] [INFO] [DTToolInspect.cpp:114] ["[success]"] [source=DTToolInspect] [thread_id=1]
[2024/09/13 11:10:12.113 +08:00] [INFO] [DTToolInspect.cpp:95] ["checking full_path is /data01/tiflash/data/data/t_138302/stable/dmf_63154/7.dat: "] [source=DTToolInspect] [thread_id=1]
[2024/09/13 11:10:12.114 +08:00] [INFO] [DTToolInspect.cpp:114] ["[success]"] [source=DTToolInspect] [thread_id=1]
[2024/09/13 11:10:12.115 +08:00] [INFO] [DTToolInspect.cpp:95] ["checking full_path is /data01/tiflash/data/data/t_138302/stable/dmf_63154/18.dat: "] [source=DTToolInspect] [thread_id=1]
[2024/09/13 11:10:12.115 +08:00] [INFO] [DTToolInspect.cpp:114] ["[success]"] [source=DTToolInspect] [thread_id=1]
[2024/09/13 11:10:12.115 +08:00] [INFO] [DTToolInspect.cpp:95] ["checking full_path is /data01/tiflash/data/data/t_138302/stable/dmf_63154/meta: "] [source=DTToolInspect] [thread_id=1]
[2024/09/13 11:10:13.111 +08:00] [INFO] [KVStore.cpp:438] ["Destroy KVStore"] [thread_id=1]
[2024/09/13 11:10:13.111 +08:00] [INFO] [KVStore.cpp:615] ["KVStore shutdown, wait thread alloc monitor join"] [thread_id=1]
[2024/09/13 11:10:13.112 +08:00] [INFO] [ReadIndex.cpp:371] ["KVStore shutdown, deleting read index worker"] [thread_id=1]
[2024/09/13 11:10:13.112 +08:00] [INFO] [KVStore.cpp:441] ["Destroy KVStore Finished"] [thread_id=1]
[2024/09/13 11:10:16.827 +08:00] [ERROR] [Exception.cpp:91] ["Code: 0, e.displayText() = DB::TiFlashException: checksum mismatch for /data01/tiflash/data/data/t_138302/stable/dmf_63154/meta, e.what() = DB::TiFlashException, Stack trace:\n\n\n       0x1efe7e1\tDB::TiFlashException::TiFlashException(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::TiFlashError const&) [tiflash+32499681]\n                \tdbms/src/Common/TiFlashException.h:263\n       0x78becb7\tDB::FramedChecksumReadBuffer<DB::Digest::XXH3>::readBig(char*, unsigned long) [tiflash+126610615]\n                \tdbms/src/IO/Checksum/ChecksumBuffer.h:305\n       0x77a7495\tDTTool::Inspect::inspectServiceMain(DB::Context&, DTTool::Inspect::InspectArgs const&) [tiflash+125465749]\n                \tdbms/src/Server/DTTool/DTToolInspect.cpp:98\n       0x77aa678\tDTTool::Inspect::inspectEntry(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, void (*)(int, char const* const*, DB::EngineStoreServerHelper const*)) [tiflash+125478520]\n                \tdbms/src/Server/DTTool/DTToolInspect.cpp:267\n       0x208a671\tDTTool::mainEntryTiFlashDTTool(int, char**) [tiflash+34121329]\n                \tdbms/src/Server/DTTool/DTTool.cpp:84\n       0x1ecef5c\tmain [tiflash+32304988]\n                \tdbms/src/Server/main.cpp:172\n  0x7f4a01856555\t__libc_start_main [libc.so.6+140629]\n       0x1bae269\t<unknown symbol> [tiflash+29024873]"] [source=DTToolInspect] [thread_id=1]
42

建议把 tiflash 数据清理再重建一下副本试一试看。
另外 tiflash 做过什么非法操作吗？如果只是 tiflash 重启，不会造成 tiflash 数据异常的，如果无非法操作，麻烦收集一下相关时间段的 metric 和 log，我们排查一下 tiflash 数据异常的原因。

43

没做过什么非法操作，只是我们的一个TiFlash节点硬件故障强制下线过。 另外我们发现有问题的表都是包含json字段的表

44

请问有相关时间段的 metric 和 log，我们排查一下 tiflash 数据异常的原因。