用自带的importer工具导入数据到tikv集群后，查不到数据

🪐 TiDB 技术问题 备份&数据迁移
1

使用tikv-importer工具将数据导入到tikv集群后，通过tikv查询导入的数据，发现找不到：
其导入的流程如下：，其中client是一个实现https://github.com/pingcap/kvproto/blob/ecdbf1f8c130089392a9bb5f86f7577deddfbed5/proto/import_kvpb.proto 这个proto中的grpc客户端

722×732 79.6 KB

2

通过tikv cli 的命令行扫描keys 试试

3

也没有找到

4

哪估计没写进去… :face_with_spiral_eyes:

5

image
image1417×41 4.51 KB

我看是有sst文件的

8

[2023/09/21 17:46:19.781 +08:00] [INFO] [kv_importer.rs:69] [“open engine completed”] [engine=“EngineFile { uuid: b59be8f0-0eb6-414e-84d2-97836ceb0af6, path: EnginePath { save: “./b59be8f0-0eb6-414e-84d2-97836ceb0af6”, temp: “./.temp/b59be8f0-0eb6-414e-84d2-97836ceb0af6” } }”]

[2023/09/21 17:46:19.800 +08:00] [INFO] [kv_importer.rs:69] [“open engine completed”] [engine=“EngineFile { uuid: 69e7ea77-fb5c-471b-a0cb-2ab8b030ef42, path: EnginePath { save: “./69e7ea77-fb5c-471b-a0cb-2ab8b030ef42”, temp: “./.temp/69e7ea77-fb5c-471b-a0cb-2ab8b030ef42” } }”]

[2023/09/21 17:46:19.815 +08:00] [INFO] [kv_importer.rs:69] [“open engine completed”] [engine=“EngineFile { uuid: 2d55cf1a-f8eb-4846-a08b-6635c3d50e6a, path: EnginePath { save: “./2d55cf1a-f8eb-4846-a08b-6635c3d50e6a”, temp: “./.temp/2d55cf1a-f8eb-4846-a08b-6635c3d50e6a” } }”]

[2023/09/21 17:46:19.835 +08:00] [INFO] [kv_importer.rs:69] [“open engine completed”] [engine=“EngineFile { uuid: 911b2062-f386-438d-8361-ad9c0fd1a8cf, path: EnginePath { save: “./911b2062-f386-438d-8361-ad9c0fd1a8cf”, temp: “./.temp/911b2062-f386-438d-8361-ad9c0fd1a8cf” } }”]

[2023/09/21 17:46:19.850 +08:00] [INFO] [kv_importer.rs:69] [“open engine completed”] [engine=“EngineFile { uuid: 0bee1b57-64af-4c34-b0cc-e761b6d1bec8, path: EnginePath { save: “./0bee1b57-64af-4c34-b0cc-e761b6d1bec8”, temp: “./.temp/0bee1b57-64af-4c34-b0cc-e761b6d1bec8” } }”]

[2023/09/21 17:46:19.866 +08:00] [INFO] [kv_importer.rs:69] [“open engine completed”] [engine=“EngineFile { uuid: 856f1ab1-758d-4520-96cf-40636cb5791f, path: EnginePath { save: “./856f1ab1-758d-4520-96cf-40636cb5791f”, temp: “./.temp/856f1ab1-758d-4520-96cf-40636cb5791f” } }”]

[2023/09/21 17:46:19.896 +08:00] [INFO] [kv_importer.rs:69] [“open engine completed”] [engine=“EngineFile { uuid: 797512a9-31f3-452b-8364-7c35eef847e4, path: EnginePath { save: “./797512a9-31f3-452b-8364-7c35eef847e4”, temp: “./.temp/797512a9-31f3-452b-8364-7c35eef847e4” } }”]

[2023/09/21 17:46:19.915 +08:00] [INFO] [kv_importer.rs:69] [“open engine completed”] [engine=“EngineFile { uuid: f6c08aa7-c1e0-4f67-ac9f-64d99dc00748, path: EnginePath { save: “./f6c08aa7-c1e0-4f67-ac9f-64d99dc00748”, temp: “./.temp/f6c08aa7-c1e0-4f67-ac9f-64d99dc00748” } }”]

[2023/09/21 17:46:19.923 +08:00] [INFO] [kv_importer.rs:109] [“close engine completed”] [engine=“EngineFile { uuid: b59be8f0-0eb6-414e-84d2-97836ceb0af6, path: EnginePath { save: “./b59be8f0-0eb6-414e-84d2-97836ceb0af6”, temp: “./.temp/b59be8f0-0eb6-414e-84d2-97836ceb0af6” } }”]

[2023/09/21 17:46:19.940 +08:00] [INFO] [kv_importer.rs:109] [“close engine completed”] [engine=“EngineFile { uuid: 69e7ea77-fb5c-471b-a0cb-2ab8b030ef42, path: EnginePath { save: “./69e7ea77-fb5c-471b-a0cb-2ab8b030ef42”, temp: “./.temp/69e7ea77-fb5c-471b-a0cb-2ab8b030ef42” } }”]

[2023/09/21 17:46:19.943 +08:00] [INFO] [kv_importer.rs:109] [“close engine completed”] [engine=“EngineFile { uuid: 2d55cf1a-f8eb-4846-a08b-6635c3d50e6a, path: EnginePath { save: “./2d55cf1a-f8eb-4846-a08b-6635c3d50e6a”, temp: “./.temp/2d55cf1a-f8eb-4846-a08b-6635c3d50e6a” } }”]

[2023/09/21 17:46:19.945 +08:00] [INFO] [kv_importer.rs:109] [“close engine completed”] [engine=“EngineFile { uuid: 911b2062-f386-438d-8361-ad9c0fd1a8cf, path: EnginePath { save: “./911b2062-f386-438d-8361-ad9c0fd1a8cf”, temp: “./.temp/911b2062-f386-438d-8361-ad9c0fd1a8cf” } }”]

[2023/09/21 17:46:19.959 +08:00] [INFO] [kv_importer.rs:109] [“close engine completed”] [engine=“EngineFile { uuid: 0bee1b57-64af-4c34-b0cc-e761b6d1bec8, path: EnginePath { save: “./0bee1b57-64af-4c34-b0cc-e761b6d1bec8”, temp: “./.temp/0bee1b57-64af-4c34-b0cc-e761b6d1bec8” } }”]

[2023/09/21 17:46:19.961 +08:00] [INFO] [kv_importer.rs:109] [“close engine completed”] [engine=“EngineFile { uuid: 856f1ab1-758d-4520-96cf-40636cb5791f, path: EnginePath { save: “./856f1ab1-758d-4520-96cf-40636cb5791f”, temp: “./.temp/856f1ab1-758d-4520-96cf-40636cb5791f” } }”]

[2023/09/21 17:46:19.963 +08:00] [INFO] [kv_importer.rs:109] [“close engine completed”] [engine=“EngineFile { uuid: 797512a9-31f3-452b-8364-7c35eef847e4, path: EnginePath { save: “./797512a9-31f3-452b-8364-7c35eef847e4”, temp: “./.temp/797512a9-31f3-452b-8364-7c35eef847e4” } }”]

[2023/09/21 17:46:19.978 +08:00] [INFO] [kv_importer.rs:109] [“close engine completed”] [engine=“EngineFile { uuid: f6c08aa7-c1e0-4f67-ac9f-64d99dc00748, path: EnginePath { save: “./f6c08aa7-c1e0-4f67-ac9f-64d99dc00748”, temp: “./.temp/f6c08aa7-c1e0-4f67-ac9f-64d99dc00748” } }”]

[2023/09/21 17:46:19.978 +08:00] [INFO] [util.rs:403] [“connecting to PD endpoint”] []

[2023/09/21 17:46:19.978 +08:00] [INFO] [] [“New connected subchannel at 0x7f6808a39180 for subchannel 0x7f681be1c1c0”]

[2023/09/21 17:46:19.979 +08:00] [INFO] [util.rs:403] [“connecting to PD endpoint”] []

[2023/09/21 17:46:19.979 +08:00] [INFO] [] [“New connected subchannel at 0x7f6808a39240 for subchannel 0x7f681be1c1c0”]

[2023/09/21 17:46:19.980 +08:00] [INFO] [util.rs:403] [“connecting to PD endpoint”] []

[2023/09/21 17:46:19.980 +08:00] [INFO] [] [“New connected subchannel at 0x7f6808a39300 for subchannel 0x7f681be1c1c0”]

[2023/09/21 17:46:19.980 +08:00] [INFO] [util.rs:462] [“connected to PD leader”] []

[2023/09/21 17:46:19.980 +08:00] [INFO] [util.rs:391] [“all PD endpoints are consistent”] []

[2023/09/21 17:46:19.981 +08:00] [INFO] [] [“New connected subchannel at 0x7f6587e3e150 for subchannel 0x7f681be1c700”]

[2023/09/21 17:46:19.981 +08:00] [INFO] [] [“New connected subchannel at 0x7f6587e3e1e0 for subchannel 0x7f681be1ca80”]

[2023/09/21 17:46:19.982 +08:00] [INFO] [] [“New connected subchannel at 0x7f6587e3e270 for subchannel 0x7f681be1ce00”]

[2023/09/21 17:46:20.011 +08:00] [INFO] [kv_service.rs:82] [“switch cluster”] [req=“SwitchModeRequest { mode: Import }”]

[2023/09/21 17:46:20.012 +08:00] [INFO] [util.rs:403] [“connecting to PD endpoint”] []

[2023/09/21 17:46:20.012 +08:00] [INFO] [] [“New connected subchannel at 0x7f6808a39390 for subchannel 0x7f681b61d1c0”]

[2023/09/21 17:46:20.013 +08:00] [INFO] [util.rs:403] [“connecting to PD endpoint”] []

[2023/09/21 17:46:20.013 +08:00] [INFO] [] [“New connected subchannel at 0x7f6808a39450 for subchannel 0x7f681b61d1c0”]

[2023/09/21 17:46:20.013 +08:00] [INFO] [util.rs:403] [“connecting to PD endpoint”] []

[2023/09/21 17:46:20.014 +08:00] [INFO] [] [“New connected subchannel at 0x7f6808a39510 for subchannel 0x7f681b61d1c0”]

[2023/09/21 17:46:20.014 +08:00] [INFO] [util.rs:462] [“connected to PD leader”] []

[2023/09/21 17:46:20.014 +08:00] [INFO] [util.rs:391] [“all PD endpoints are consistent”] []

[2023/09/21 17:46:20.034 +08:00] [INFO] [import.rs:56] [“import engine”] [tag=“[ImportJob b59be8f0-0eb6-414e-84d2-97836ceb0af6]”]

[2023/09/21 17:46:20.034 +08:00] [INFO] [prepare.rs:76] [“split and scatter”] [tag=“[PrepareJob b59be8f0-0eb6-414e-84d2-97836ceb0af6]”]

[2023/09/21 17:46:20.034 +08:00] [INFO] [prepare.rs:80] [“get size properties”] [size=0] [tag=“[PrepareJob b59be8f0-0eb6-414e-84d2-97836ceb0af6]”]

[2023/09/21 17:46:20.034 +08:00] [INFO] [prepare.rs:138] [“scatter all regions finished”] [takes=150ns] [tag=“[PrepareJob b59be8f0-0eb6-414e-84d2-97836ceb0af6]”]

[2023/09/21 17:46:20.034 +08:00] [INFO] [prepare.rs:94] [“split and scatter completed”] [takes=133.791µs] [ranges=0] [tag=“[PrepareJob b59be8f0-0eb6-414e-84d2-97836ceb0af6]”]

[2023/09/21 17:46:20.034 +08:00] [INFO] [import.rs:194] [“run import threads”] [num_import_jobs=24] [tag=“[ImportJob b59be8f0-0eb6-414e-84d2-97836ceb0af6]”]

[2023/09/21 17:46:20.055 +08:00] [INFO] [import.rs:99] [“import engine completed”] [takes=21.953706ms] [tag=“[ImportJob b59be8f0-0eb6-414e-84d2-97836ceb0af6]”]

[2023/09/21 17:46:20.056 +08:00] [INFO] [kv_importer.rs:145] [“import completed”] [uuid=b59be8f0-0eb6-414e-84d2-97836ceb0af6]

[2023/09/21 17:46:20.059 +08:00] [INFO] [kv_importer.rs:179] [“cleanup completed”] [uuid=b59be8f0-0eb6-414e-84d2-97836ceb0af6]

[2023/09/21 17:46:20.059 +08:00] [INFO] [util.rs:403] [“connecting to PD endpoint”] []

[2023/09/21 17:46:20.060 +08:00] [INFO] [] [“New connected subchannel at 0x7f6808a395a0 for subchannel 0x7f681a81c1c0”]

[2023/09/21 17:46:20.060 +08:00] [INFO] [util.rs:403] [“connecting to PD endpoint”] []

[2023/09/21 17:46:20.060 +08:00] [INFO] [] [“New connected subchannel at 0x7f6808a39660 for subchannel 0x7f681a81c1c0”]

[2023/09/21 17:46:20.061 +08:00] [INFO] [util.rs:403] [“connecting to PD endpoint”] []

[2023/09/21 17:46:20.061 +08:00] [INFO] [] [“New connected subchannel at 0x7f6808a39720 for subchannel 0x7f681a81c1c0”]

[2023/09/21 17:46:20.061 +08:00] [INFO] [util.rs:462] [“connected to PD leader”] []

[2023/09/21 17:46:20.061 +08:00] [INFO] [util.rs:391] [“all PD endpoints are consistent”] []

[2023/09/21 17:46:20.078 +08:00] [INFO] [import.rs:56] [“import engine”] [tag=“[ImportJob 69e7ea77-fb5c-471b-a0cb-2ab8b030ef42]”]

[2023/09/21 17:46:20.078 +08:00] [INFO] [prepare.rs:76] [“split and scatter”] [tag=“[PrepareJob 69e7ea77-fb5c-471b-a0cb-2ab8b030ef42]”]

[2023/09/21 17:46:20.078 +08:00] [INFO] [prepare.rs:80] [“get size properties”] [size=200] [tag=“[PrepareJob 69e7ea77-fb5c-471b-a0cb-2ab8b030ef42]”]

[2023/09/21 17:46:20.078 +08:00] [INFO] [prepare.rs:138] [“scatter all regions finished”] [takes=161ns] [tag=“[PrepareJob 69e7ea77-fb5c-471b-a0cb-2ab8b030ef42]”]

[2023/09/21 17:46:20.078 +08:00] [INFO] [prepare.rs:94] [“split and scatter completed”] [takes=13.095µs] [ranges=0] [tag=“[PrepareJob 69e7ea77-fb5c-471b-a0cb-2ab8b030ef42]”]

[2023/09/21 17:46:20.078 +08:00] [INFO] [import.rs:194] [“run import threads”] [num_import_jobs=24] [tag=“[ImportJob 69e7ea77-fb5c-471b-a0cb-2ab8b030ef42]”]

[2023/09/21 17:46:20.081 +08:00] [INFO] [import.rs:182] [“dispatch-job done”] [ranges_handled=1] [id=36] [tag=“[ImportJob 69e7ea77-fb5c-471b-a0cb-2ab8b030ef42]”]

[2023/09/21 17:46:20.081 +08:00] [INFO] [import.rs:320] [“import sst”] [sst=“SSTFile { uuid: ce8288ec-d402-4722-9073-24cb805ec46b, range: 3030303030303044FF3030303161343034FF3230463030313030FF3030303030303030FF3030303030303030FF3030334538000000FCFFFFFFFFFFFFFFFF{…}, length: 1619, cf_name: “write” }”] [tag=“[ImportSSTJob 69e7ea77-fb5c-471b-a0cb-2ab8b030ef42:0:1]”]

[2023/09/21 17:46:20.081 +08:00] [INFO] [import.rs:363] [“upload and ingest sst”] [region=“RegionInfo { region: Region { id: 4, range: {…}, region_epoch: RegionEpoch { conf_ver: 5, version: 1 }, peers: [Peer { id: 7, store_id: 3, is_learner: false }, Peer { id: 10, store_id: 2, is_learner: false }, Peer { id: 11, store_id: 1, is_learner: false }] }, leader: Some(Peer { id: 7, store_id: 3, is_learner: false }) }”] [tag=“[ImportSSTJob 69e7ea77-fb5c-471b-a0cb-2ab8b030ef42:0:1]”]

[2023/09/21 17:46:20.082 +08:00] [INFO] [] [“New connected subchannel at 0x7f6587e3e300 for subchannel 0x7f681e1271c0”]

[2023/09/21 17:46:20.094 +08:00] [INFO] [] [“New connected subchannel at 0x7f6587a46150 for subchannel 0x7f681e127540”]

[2023/09/21 17:46:20.099 +08:00] [INFO] [] [“New connected subchannel at 0x7f6586e0f150 for subchannel 0x7f681e1278c0”]

[2023/09/21 17:46:20.120 +08:00] [INFO] [import.rs:478] [“ingest completed”] [region_id=4] [store=3] [takes=16.238892ms] [tag=“[ImportSSTJob 69e7ea77-fb5c-471b-a0cb-2ab8b030ef42:0:1]”]

[2023/09/21 17:46:20.120 +08:00] [INFO] [import.rs:346] [“import sst completed”] [takes=38.529469ms] [tag=“[ImportSSTJob 69e7ea77-fb5c-471b-a0cb-2ab8b030ef42:0:1]”]

[2023/09/21 17:46:20.120 +08:00] [INFO] [engine.rs:220] [“cleanup SST completed”] [file_path=““./69e7ea77-fb5c-471b-a0cb-2ab8b030ef42/.70a5ef1f-e221-486b-9b70-f67a7e13b1a9:write””]

[2023/09/21 17:46:20.120 +08:00] [INFO] [import.rs:99] [“import engine completed”] [takes=42.726626ms] [tag=“[ImportJob 69e7ea77-fb5c-471b-a0cb-2ab8b030ef42]”]

[2023/09/21 17:46:20.120 +08:00] [INFO] [kv_importer.rs:145] [“import completed”] [uuid=69e7ea77-fb5c-471b-a0cb-2ab8b030ef42]

[2023/09/21 17:46:20.125 +08:00] [INFO] [kv_importer.rs:179] [“cleanup completed”] [uuid=69e7ea77-fb5c-471b-a0cb-2ab8b030ef42]

[2023/09/21 17:46:20.125 +08:00] [INFO] [util.rs:403] [“connecting to PD endpoint”] []

[2023/09/21 17:46:20.125 +08:00] [INFO] [] [“New connected subchannel at 0x7f6808a397b0 for subchannel 0x7f681921d1c0”]

[2023/09/21 17:46:20.126 +08:00] [INFO] [util.rs:403] [“connecting to PD endpoint”] []

[2023/09/21 17:46:20.126 +08:00] [INFO] [] [“New connected subchannel at 0x7f6808a39870 for subchannel 0x7f681921d1c0”]

[2023/09/21 17:46:20.126 +08:00] [INFO] [util.rs:403] [“connecting to PD endpoint”] []

[2023/09/21 17:46:20.126 +08:00] [INFO] [] [“New connected subchannel at 0x7f6808a39930 for subchannel 0x7f681921d1c0”]

[2023/09/21 17:46:20.127 +08:00] [INFO] [util.rs:462] [“connected to PD leader”] []

[2023/09/21 17:46:20.127 +08:00] [INFO] [util.rs:391] [“all PD endpoints are consistent”] []

[2023/09/21 17:46:20.147 +08:00] [INFO] [import.rs:56] [“import engine”] [tag=“[ImportJob 2d55cf1a-f8eb-4846-a08b-6635c3d50e6a]”]

[2023/09/21 17:46:20.147 +08:00] [INFO] [prepare.rs:76] [“split and scatter”] [tag=“[PrepareJob 2d55cf1a-f8eb-4846-a08b-6635c3d50e6a]”]

[2023/09/21 17:46:20.147 +08:00] [INFO] [prepare.rs:80] [“get size properties”] [size=0] [tag=“[PrepareJob 2d55cf1a-f8eb-4846-a08b-6635c3d50e6a]”]

[2023/09/21 17:46:20.147 +08:00] [INFO] [prepare.rs:138] [“scatter all regions finished”] [takes=130ns] [tag=“[PrepareJob 2d55cf1a-f8eb-4846-a08b-6635c3d50e6a]”]

[2023/09/21 17:46:20.147 +08:00] [INFO] [prepare.rs:94] [“split and scatter completed”] [takes=113.282µs] [ranges=0] [tag=“[PrepareJob 2d55cf1a-f8eb-4846-a08b-6635c3d50e6a]”]

[2023/09/21 17:46:20.147 +08:00] [INFO] [import.rs:194] [“run import threads”] [num_import_jobs=24] [tag=“[ImportJob 2d55cf1a-f8eb-4846-a08b-6635c3d50e6a]”]

[2023/09/21 17:46:20.151 +08:00] [INFO] [import.rs:99] [“import engine completed”] [takes=4.668911ms] [tag=“[ImportJob 2d55cf1a-f8eb-4846-a08b-6635c3d50e6a]”]

[2023/09/21 17:46:20.151 +08:00] [INFO] [kv_importer.rs:145] [“import completed”] [uuid=2d55cf1a-f8eb-4846-a08b-6635c3d50e6a]

[2023/09/21 17:46:20.156 +08:00] [INFO] [kv_importer.rs:179] [“cleanup completed”] [uuid=2d55cf1a-f8eb-4846-a08b-6635c3d50e6a]

[2023/09/21 17:46:20.156 +08:00] [INFO] [util.rs:403] [“connecting to PD endpoint”] []

[2023/09/21 17:46:20.157 +08:00] [INFO] [] [“New connected subchannel at 0x7f6808a399f0 for subchannel 0x7f682025a540”]

[2023/09/21 17:46:20.157 +08:00] [INFO] [util.rs:403] [“connecting to PD endpoint”] []

[2023/09/21 17:46:20.157 +08:00] [INFO] [] [“New connected subchannel at 0x7f6808a39ab0 for subchannel 0x7f682025a540”]

[2023/09/21 17:46:20.158 +08:00] [INFO] [util.rs:403] [“connecting to PD endpoint”] []

[2023/09/21 17:46:20.158 +08:00] [INFO] [] [“New connected subchannel at 0x7f6808a39b70 for subchannel 0x7f682025a540”]

[2023/09/21 17:46:20.158 +08:00] [INFO] [util.rs:462] [“connected to PD leader”] []

[2023/09/21 17:46:20.158 +08:00] [INFO] [util.rs:391] [“all PD endpoints are consistent”] []

[2023/09/21 17:46:20.171 +08:00] [INFO] [import.rs:56] [“import engine”] [tag=“[ImportJob 911b2062-f386-438d-8361-ad9c0fd1a8cf]”]

[2023/09/21 17:46:20.171 +08:00] [INFO] [prepare.rs:76] [“split and scatter”] [tag=“[PrepareJob 911b2062-f386-438d-8361-ad9c0fd1a8cf]”]

[2023/09/21 17:46:20.171 +08:00] [INFO] [prepare.rs:80] [“get size properties”] [size=0] [tag=“[PrepareJob 911b2062-f386-438d-8361-ad9c0fd1a8cf]”]

[2023/09/21 17:46:20.171 +08:00] [INFO] [prepare.rs:138] [“scatter all regions finished”] [takes=140ns] [tag=“[PrepareJob 911b2062-f386-438d-8361-ad9c0fd1a8cf]”]

[2023/09/21 17:46:20.171 +08:00] [INFO] [prepare.rs:94] [“split and scatter completed”] [takes=82.545µs] [ranges=0] [tag=“[PrepareJob 911b2062-f386-438d-8361-ad9c0fd1a8cf]”]

[2023/09/21 17:46:20.171 +08:00] [INFO] [import.rs:194] [“run import threads”] [num_import_jobs=24] [tag=“[ImportJob 911b2062-f386-438d-8361-ad9c0fd1a8cf]”]

[2023/09/21 17:46:20.195 +08:00] [INFO] [import.rs:99] [“import engine completed”] [takes=23.93328ms] [tag=“[ImportJob 911b2062-f386-438d-8361-ad9c0fd1a8cf]”]

[2023/09/21 17:46:20.195 +08:00] [INFO] [kv_importer.rs:145] [“import completed”] [uuid=911b2062-f386-438d-8361-ad9c0fd1a8cf]

[2023/09/21 17:46:20.203 +08:00] [INFO] [kv_importer.rs:179] [“cleanup completed”] [uuid=911b2062-f386-438d-8361-ad9c0fd1a8cf]

[2023/09/21 17:46:20.203 +08:00] [INFO] [util.rs:403] [“connecting to PD endpoint”] []

[2023/09/21 17:46:20.204 +08:00] [INFO] [] [“New connected subchannel at 0x7f6808a39c30 for subchannel 0x7f681fc5a1c0”]

[2023/09/21 17:46:20.204 +08:00] [INFO] [util.rs:403] [“connecting to PD endpoint”] []

[2023/09/21 17:46:20.204 +08:00] [INFO] [] [“New connected subchannel at 0x7f6808a39cf0 for subchannel 0x7f681fc5a1c0”]

[2023/09/21 17:46:20.205 +08:00] [INFO] [util.rs:403] [“connecting to PD endpoint”] []

[2023/09/21 17:46:20.205 +08:00] [INFO] [] [“New connected subchannel at 0x7f6808a39db0 for subchannel 0x7f681fc5a1c0”]

[2023/09/21 17:46:20.205 +08:00] [INFO] [util.rs:462] [“connected to PD leader”] []

[2023/09/21 17:46:20.205 +08:00] [INFO] [util.rs:391] [“all PD endpoints are consistent”] []

[2023/09/21 17:46:20.223 +08:00] [INFO] [import.rs:56] [“import engine”] [tag=“[ImportJob 0bee1b57-64af-4c34-b0cc-e761b6d1bec8]”]

[2023/09/21 17:46:20.224 +08:00] [INFO] [prepare.rs:76] [“split and scatter”] [tag=“[PrepareJob 0bee1b57-64af-4c34-b0cc-e761b6d1bec8]”]

[2023/09/21 17:46:20.224 +08:00] [INFO] [prepare.rs:80] [“get size properties”] [size=209] [tag=“[PrepareJob 0bee1b57-64af-4c34-b0cc-e761b6d1bec8]”]

[2023/09/21 17:46:20.224 +08:00] [INFO] [prepare.rs:138] [“scatter all regions finished”] [takes=211ns] [tag=“[PrepareJob 0bee1b57-64af-4c34-b0cc-e761b6d1bec8]”]

[2023/09/21 17:46:20.224 +08:00] [INFO] [prepare.rs:94] [“split and scatter completed”] [takes=15.288µs] [ranges=0] [tag=“[PrepareJob 0bee1b57-64af-4c34-b0cc-e761b6d1bec8]”]

[2023/09/21 17:46:20.224 +08:00] [INFO] [import.rs:194] [“run import threads”] [num_import_jobs=24] [tag=“[ImportJob 0bee1b57-64af-4c34-b0cc-e761b6d1bec8]”]

[2023/09/21 17:46:20.246 +08:00] [INFO] [import.rs:182] [“dispatch-job done”] [ranges_handled=1] [id=46] [tag=“[ImportJob 0bee1b57-64af-4c34-b0cc-e761b6d1bec8]”]

[2023/09/21 17:46:20.246 +08:00] [INFO] [import.rs:320] [“import sst”] [sst=“SSTFile { uuid: 2cc2cadc-eb26-4c18-99be-f11b3c416bde, range: 3030303030303044FF3030303461343034FF3230463030313030FF3030303030303030FF3030303030303030FF3030334135000000FCFFFFFFFFFFFFFFFF{…}, length: 1628, cf_name: “write” }”] [tag=“[ImportSSTJob 0bee1b57-64af-4c34-b0cc-e761b6d1bec8:0:1]”]

[2023/09/21 17:46:20.246 +08:00] [INFO] [import.rs:363] [“upload and ingest sst”] [region=“RegionInfo { region: Region { id: 4, range: {…}, region_epoch: RegionEpoch { conf_ver: 5, version: 1 }, peers: [Peer { id: 7, store_id: 3, is_learner: false }, Peer { id: 10, store_id: 2, is_learner: false }, Peer { id: 11, store_id: 1, is_learner: false }] }, leader: Some(Peer { id: 7, store_id: 3, is_learner: false }) }”] [tag=“[ImportSSTJob 0bee1b57-64af-4c34-b0cc-e761b6d1bec8:0:1]”]

[2023/09/21 17:46:20.247 +08:00] [INFO] [] [“New connected subchannel at 0x7f681e002a00 for subchannel 0x7f681e13a1c0”]

[2023/09/21 17:46:20.253 +08:00] [INFO] [] [“New connected subchannel at 0x7f6587a51150 for subchannel 0x7f681e13a540”]

[2023/09/21 17:46:20.260 +08:00] [INFO] [] [“New connected subchannel at 0x7f6586e0f150 for subchannel 0x7f681e13a8c0”]

[2023/09/21 17:46:20.284 +08:00] [INFO] [import.rs:478] [“ingest completed”] [region_id=4] [store=3] [takes=20.081252ms] [tag=“[ImportSSTJob 0bee1b57-64af-4c34-b0cc-e761b6d1bec8:0:1]”]

[2023/09/21 17:46:20.284 +08:00] [INFO] [import.rs:346] [“import sst completed”] [takes=37.969468ms] [tag=“[ImportSSTJob 0bee1b57-64af-4c34-b0cc-e761b6d1bec8:0:1]”]

[2023/09/21 17:46:20.284 +08:00] [INFO] [engine.rs:220] [“cleanup SST completed”] [file_path=““./0bee1b57-64af-4c34-b0cc-e761b6d1bec8/.9ab3dbe7-43a7-4a33-900d-8f54da86dd6b:write””]

[2023/09/21 17:46:20.285 +08:00] [INFO] [import.rs:99] [“import engine completed”] [takes=61.131241ms] [tag=“[ImportJob 0bee1b57-64af-4c34-b0cc-e761b6d1bec8]”]

[2023/09/21 17:46:20.285 +08:00] [INFO] [kv_importer.rs:145] [“import completed”] [uuid=0bee1b57-64af-4c34-b0cc-e761b6d1bec8]

[2023/09/21 17:46:20.289 +08:00] [INFO] [kv_importer.rs:179] [“cleanup completed”] [uuid=0bee1b57-64af-4c34-b0cc-e761b6d1bec8]

[2023/09/21 17:46:20.290 +08:00] [INFO] [util.rs:403] [“connecting to PD endpoint”] []

[2023/09/21 17:46:20.290 +08:00] [INFO] [] [“New connected subchannel at 0x7f6808a39c60 for subchannel 0x7f681ec591c0”]

[2023/09/21 17:46:20.290 +08:00] [INFO] [util.rs:403] [“connecting to PD endpoint”] []

[2023/09/21 17:46:20.291 +08:00] [INFO] [] [“New connected subchannel at 0x7f6808a39d20 for subchannel 0x7f681ec591c0”]

[2023/09/21 17:46:20.291 +08:00] [INFO] [util.rs:403] [“connecting to PD endpoint”] []

[2023/09/21 17:46:20.291 +08:00] [INFO] [] [“New connected subchannel at 0x7f6808a39de0 for subchannel 0x7f681ec591c0”]

[2023/09/21 17:46:20.291 +08:00] [INFO] [util.rs:462] [“connected to PD leader”] []

[2023/09/21 17:46:20.292 +08:00] [INFO] [util.rs:391] [“all PD endpoints are consistent”] []

[2023/09/21 17:46:20.306 +08:00] [INFO] [import.rs:56] [“import engine”] [tag=“[ImportJob 856f1ab1-758d-4520-96cf-40636cb5791f]”]

[2023/09/21 17:46:20.306 +08:00] [INFO] [prepare.rs:76] [“split and scatter”] [tag=“[PrepareJob 856f1ab1-758d-4520-96cf-40636cb5791f]”]

[2023/09/21 17:46:20.306 +08:00] [INFO] [prepare.rs:80] [“get size properties”] [size=0] [tag=“[PrepareJob 856f1ab1-758d-4520-96cf-40636cb5791f]”]

[2023/09/21 17:46:20.306 +08:00] [INFO] [prepare.rs:138] [“scatter all regions finished”] [takes=151ns] [tag=“[PrepareJob 856f1ab1-758d-4520-96cf-40636cb5791f]”]

[2023/09/21 17:46:20.306 +08:00] [INFO] [prepare.rs:94] [“split and scatter completed”] [takes=130.986µs] [ranges=0] [tag=“[PrepareJob 856f1ab1-758d-4520-96cf-40636cb5791f]”]

[2023/09/21 17:46:20.306 +08:00] [INFO] [import.rs:194] [“run import threads”] [num_import_jobs=24] [tag=“[ImportJob 856f1ab1-758d-4520-96cf-40636cb5791f]”]

[2023/09/21 17:46:20.321 +08:00] [INFO] [import.rs:99] [“import engine completed”] [takes=15.406821ms] [tag=“[ImportJob 856f1ab1-758d-4520-96cf-40636cb5791f]”]

[2023/09/21 17:46:20.321 +08:00] [INFO] [kv_importer.rs:145] [“import completed”] [uuid=856f1ab1-758d-4520-96cf-40636cb5791f]

[2023/09/21 17:46:20.324 +08:00] [INFO] [kv_importer.rs:179] [“cleanup completed”] [uuid=856f1ab1-758d-4520-96cf-40636cb5791f]

[2023/09/21 17:46:20.324 +08:00] [INFO] [util.rs:403] [“connecting to PD endpoint”] []

[2023/09/21 17:46:20.324 +08:00] [INFO] [] [“New connected subchannel at 0x7f6808a39ea0 for subchannel 0x7f681e0591c0”]

[2023/09/21 17:46:20.325 +08:00] [INFO] [util.rs:403] [“connecting to PD endpoint”] []

[2023/09/21 17:46:20.325 +08:00] [INFO] [] [“New connected subchannel at 0x7f6808a39f60 for subchannel 0x7f681e0591c0”]

[2023/09/21 17:46:20.325 +08:00] [INFO] [util.rs:403] [“connecting to PD endpoint”] []

[2023/09/21 17:46:20.325 +08:00] [INFO] [] [“New connected subchannel at 0x7f6808a3a020 for subchannel 0x7f681e0591c0”]

[2023/09/21 17:46:20.326 +08:00] [INFO] [util.rs:462] [“connected to PD leader”] []

[2023/09/21 17:46:20.326 +08:00] [INFO] [util.rs:391] [“all PD endpoints are consistent”] []

[2023/09/21 17:46:20.339 +08:00] [INFO] [import.rs:56] [“import engine”] [tag=“[ImportJob 797512a9-31f3-452b-8364-7c35eef847e4]”]

[2023/09/21 17:46:20.339 +08:00] [INFO] [prepare.rs:76] [“split and scatter”] [tag=“[PrepareJob 797512a9-31f3-452b-8364-7c35eef847e4]”]

[2023/09/21 17:46:20.339 +08:00] [INFO] [prepare.rs:80] [“get size properties”] [size=0] [tag=“[PrepareJob 797512a9-31f3-452b-8364-7c35eef847e4]”]

[2023/09/21 17:46:20.339 +08:00] [INFO] [prepare.rs:138] [“scatter all regions finished”] [takes=250ns] [tag=“[PrepareJob 797512a9-31f3-452b-8364-7c35eef847e4]”]

[2023/09/21 17:46:20.339 +08:00] [INFO] [prepare.rs:94] [“split and scatter completed”] [takes=105.598µs] [ranges=0] [tag=“[PrepareJob 797512a9-31f3-452b-8364-7c35eef847e4]”]

[2023/09/21 17:46:20.339 +08:00] [INFO] [import.rs:194] [“run import threads”] [num_import_jobs=24] [tag=“[ImportJob 797512a9-31f3-452b-8364-7c35eef847e4]”]

[2023/09/21 17:46:20.361 +08:00] [INFO] [import.rs:99] [“import engine completed”] [takes=21.704819ms] [tag=“[ImportJob 797512a9-31f3-452b-8364-7c35eef847e4]”]

[2023/09/21 17:46:20.361 +08:00] [INFO] [kv_importer.rs:145] [“import completed”] [uuid=797512a9-31f3-452b-8364-7c35eef847e4]

[2023/09/21 17:46:20.369 +08:00] [INFO] [kv_importer.rs:179] [“cleanup completed”] [uuid=797512a9-31f3-452b-8364-7c35eef847e4]

[2023/09/21 17:46:20.369 +08:00] [INFO] [util.rs:403] [“connecting to PD endpoint”] []

[2023/09/21 17:46:20.370 +08:00] [INFO] [] [“New connected subchannel at 0x7f6808a3a0e0 for subchannel 0x7f681ca591c0”]

[2023/09/21 17:46:20.370 +08:00] [INFO] [util.rs:403] [“connecting to PD endpoint”] []

[2023/09/21 17:46:20.370 +08:00] [INFO] [] [“New connected subchannel at 0x7f6808a3a1a0 for subchannel 0x7f681ca591c0”]

[2023/09/21 17:46:20.371 +08:00] [INFO] [util.rs:403] [“connecting to PD endpoint”] []

[2023/09/21 17:46:20.371 +08:00] [INFO] [] [“New connected subchannel at 0x7f6808a3a260 for subchannel 0x7f681ca591c0”]

[2023/09/21 17:46:20.371 +08:00] [INFO] [util.rs:462] [“connected to PD leader”] []

[2023/09/21 17:46:20.371 +08:00] [INFO] [util.rs:391] [“all PD endpoints are consistent”] []

[2023/09/21 17:46:20.385 +08:00] [INFO] [import.rs:56] [“import engine”] [tag=“[ImportJob f6c08aa7-c1e0-4f67-ac9f-64d99dc00748]”]

[2023/09/21 17:46:20.385 +08:00] [INFO] [prepare.rs:76] [“split and scatter”] [tag=“[PrepareJob f6c08aa7-c1e0-4f67-ac9f-64d99dc00748]”]

[2023/09/21 17:46:20.385 +08:00] [INFO] [prepare.rs:80] [“get size properties”] [size=204] [tag=“[PrepareJob f6c08aa7-c1e0-4f67-ac9f-64d99dc00748]”]

[2023/09/21 17:46:20.385 +08:00] [INFO] [prepare.rs:138] [“scatter all regions finished”] [takes=90ns] [tag=“[PrepareJob f6c08aa7-c1e0-4f67-ac9f-64d99dc00748]”]

[2023/09/21 17:46:20.385 +08:00] [INFO] [prepare.rs:94] [“split and scatter completed”] [takes=112.882µs] [ranges=0] [tag=“[PrepareJob f6c08aa7-c1e0-4f67-ac9f-64d99dc00748]”]

[2023/09/21 17:46:20.385 +08:00] [INFO] [import.rs:194] [“run import threads”] [num_import_jobs=24] [tag=“[ImportJob f6c08aa7-c1e0-4f67-ac9f-64d99dc00748]”]

[2023/09/21 17:46:20.389 +08:00] [INFO] [import.rs:182] [“dispatch-job done”] [ranges_handled=1] [id=45] [tag=“[ImportJob f6c08aa7-c1e0-4f67-ac9f-64d99dc00748]”]

[2023/09/21 17:46:20.389 +08:00] [INFO] [import.rs:320] [“import sst”] [sst=“SSTFile { uuid: 79042a21-1427-41ea-b445-81408d6082c8, range: 3030303030303044FF3030303761343034FF3230463030313030FF3030303030303030FF3030303030303030FF3030374430000000FCFFFFFFFFFFFFFFFF{…}, length: 1623, cf_name: “write” }”] [tag=“[ImportSSTJob f6c08aa7-c1e0-4f67-ac9f-64d99dc00748:1:1]”]

[2023/09/21 17:46:20.390 +08:00] [INFO] [import.rs:363] [“upload and ingest sst”] [region=“RegionInfo { region: Region { id: 4, range: {…}, region_epoch: RegionEpoch { conf_ver: 5, version: 1 }, peers: [Peer { id: 7, store_id: 3, is_learner: false }, Peer { id: 10, store_id: 2, is_learner: false }, Peer { id: 11, store_id: 1, is_learner: false }] }, leader: Some(Peer { id: 7, store_id: 3, is_learner: false }) }”] [tag=“[ImportSSTJob f6c08aa7-c1e0-4f67-ac9f-64d99dc00748:1:1]”]

[2023/09/21 17:46:20.390 +08:00] [INFO] [] [“New connected subchannel at 0x7f6586e0f150 for subchannel 0x7f681cd261c0”]

[2023/09/21 17:46:20.394 +08:00] [INFO] [] [“New connected subchannel at 0x7f6587a41150 for subchannel 0x7f681cd26540”]

[2023/09/21 17:46:20.398 +08:00] [INFO] [] [“New connected subchannel at 0x7f6586a46150 for subchannel 0x7f681cd268c0”]

[2023/09/21 17:46:20.418 +08:00] [INFO] [import.rs:478] [“ingest completed”] [region_id=4] [store=3] [takes=16.571957ms] [tag=“[ImportSSTJob f6c08aa7-c1e0-4f67-ac9f-64d99dc00748:1:1]”]

[2023/09/21 17:46:20.418 +08:00] [INFO] [import.rs:346] [“import sst completed”] [takes=29.017ms] [tag=“[ImportSSTJob f6c08aa7-c1e0-4f67-ac9f-64d99dc00748:1:1]”]

[2023/09/21 17:46:20.418 +08:00] [INFO] [engine.rs:220] [“cleanup SST completed”] [file_path=““./f6c08aa7-c1e0-4f67-ac9f-64d99dc00748/.c1814990-e44a-4188-b0d0-d603161a9ffd:write””]

[2023/09/21 17:46:20.419 +08:00] [INFO] [import.rs:99] [“import engine completed”] [takes=34.284435ms] [tag=“[ImportJob f6c08aa7-c1e0-4f67-ac9f-64d99dc00748]”]

[2023/09/21 17:46:20.419 +08:00] [INFO] [kv_importer.rs:145] [“import completed”] [uuid=f6c08aa7-c1e0-4f67-ac9f-64d99dc00748]

[2023/09/21 17:46:20.426 +08:00] [INFO] [kv_importer.rs:179] [“cleanup completed”] [uuid=f6c08aa7-c1e0-4f67-ac9f-64d99dc00748]

[2023/09/21 17:46:20.426 +08:00] [INFO] [util.rs:403] [“connecting to PD endpoint”] []

[2023/09/21 17:46:20.426 +08:00] [INFO] [] [“New connected subchannel at 0x7f6808a3a110 for subchannel 0x7f681b61d540”]

[2023/09/21 17:46:20.427 +08:00] [INFO] [util.rs:403] [“connecting to PD endpoint”] []

[2023/09/21 17:46:20.427 +08:00] [INFO] [] [“New connected subchannel at 0x7f6808a3a1d0 for subchannel 0x7f681b61d540”]

[2023/09/21 17:46:20.428 +08:00] [INFO] [util.rs:403] [“connecting to PD endpoint”] []

[2023/09/21 17:46:20.428 +08:00] [INFO] [] [“New connected subchannel at 0x7f6808a3a290 for subchannel 0x7f681b61d540”]

[2023/09/21 17:46:20.428 +08:00] [INFO] [util.rs:462] [“connected to PD leader”] []

[2023/09/21 17:46:20.428 +08:00] [INFO] [util.rs:391] [“all PD endpoints are consistent”] []

[2023/09/21 17:46:20.429 +08:00] [INFO] [] [“New connected subchannel at 0x7f6587e3e300 for subchannel 0x7f681b61d1c0”]

[2023/09/21 17:46:20.429 +08:00] [INFO] [] [“New connected subchannel at 0x7f6587e3e390 for subchannel 0x7f681b61da80”]

[2023/09/21 17:46:20.429 +08:00] [INFO] [] [“New connected subchannel at 0x7f6587e3e420 for subchannel 0x7f681b61de00”]

[2023/09/21 17:46:20.430 +08:00] [INFO] [kv_service.rs:320] [“compact cluster”] [req=“CompactRequest { range: None, output_level: 0 }”]

[2023/09/21 17:46:20.430 +08:00] [INFO] [util.rs:403] [“connecting to PD endpoint”] []

[2023/09/21 17:46:20.430 +08:00] [INFO] [] [“New connected subchannel at 0x7f6808a3a350 for subchannel 0x7f681b4231c0”]

[2023/09/21 17:46:20.431 +08:00] [INFO] [util.rs:403] [“connecting to PD endpoint”] []

[2023/09/21 17:46:20.431 +08:00] [INFO] [] [“New connected subchannel at 0x7f6808a3a410 for subchannel 0x7f681b4231c0”]

[2023/09/21 17:46:20.431 +08:00] [INFO] [util.rs:403] [“connecting to PD endpoint”] []

[2023/09/21 17:46:20.432 +08:00] [INFO] [] [“New connected subchannel at 0x7f6808a3a4d0 for subchannel 0x7f681b4231c0”]

[2023/09/21 17:46:20.432 +08:00] [INFO] [util.rs:462] [“connected to PD leader”] []

[2023/09/21 17:46:20.432 +08:00] [INFO] [util.rs:391] [“all PD endpoints are consistent”] []

[2023/09/21 17:46:20.432 +08:00] [INFO] [] [“New connected subchannel at 0x7f6587e3e4b0 for subchannel 0x7f681b423540”]

[2023/09/21 17:46:20.433 +08:00] [INFO] [] [“New connected subchannel at 0x7f6587e3e540 for subchannel 0x7f681b4238c0”]

[2023/09/21 17:46:20.433 +08:00] [INFO] [] [“New connected subchannel at 0x7f6587e3e5d0 for subchannel 0x7f681b423c40”]

[2023/09/21 17:46:20.463 +08:00] [INFO] [kv_service.rs:82] [“switch cluster”] [req=“SwitchModeRequest { mode: Normal }”]
这是importer的日志

9

但是 keys 都没有，咋能查出 value 呢…

是不是用的版本不对…

你用 PD 查一下region，看看具体region 的数据试试

10

image
image1505×255 24.5 KB

image

现在运行这个会报错

11

image
image1499×151 7.32 KB

region显示是有的， 3个key-value

12

版本的话是用的4.0.2的，tikv是6.1.0的

13

版本不对吧，，，

image
image1499×151 7.32 KB

这个里头的key 都无范围的，哪怎么读？ :rofl:

14

没有范围应该是这个region的问题，但是数据是有范围的，不知道为什么没有更新这个region的startkey

15

推荐啥版本，我试试 :joy:

16

我预先分区后，scan这个分区，还是有问题。

image
image1507×652 128 KB

17

不过keys 的范围有了…

image
image1669×866 74.9 KB

这工具都没更新了…
和 tidb 的版本完全对不上… :rofl:

不好意思，完全帮不上忙…

18

这个范围是我先分区得到的

19

那我如果不用这个工具，有没有这这样一个工具呢，就是我会生成kv，kv的顺序是未知的，然后通过batchWrite的方法写到这个工具，写完后，通过这个工具将生成的sst文件上传到tikv

20

这里我主要用的是tikv，做存储引擎，没有用tidb

21

哪可以用官方推荐的client 工具，TiKV | Java Client

有很多种语言，
image

22

但是我用的是C++，并且需要load大量数据。。。。。。