juice写入元数据到tidb报错

🪐 TiDB 技术问题
1

【 TiDB 使用环境】生产环境 /测试/ Poc
【 TiDB 版本】
【复现路径】
测试juicefs的元数据性能， ```
hadoop jar juicefs-hadoop.jar nnbench open -files 10000 -baseDir jfs://{JFS_NAME}/tmp/benchmarks/NNBench -local

在线程达到一定数量的情况下，tikv报错
【遇到的问题：问题现象及影响】
tikv报错：
2023-09-01 17:44:21,244 ERROR juicefs.JuiceFileSystemImpl: 2023/09/01 17:44:21.191927 juicefs[29229] <ERROR>: error: newRegion's range key is not encoded: id:16 start_key:"jfs-tikv-oss1\375Aq\000\000\000\000\000\000\000I" end_key:"jfs-tikv\377-oss1\375Az\377v\000\000\000\000\000\000D\37716\000\000\000\000\000\000\371" region_epoch:<conf_ver:5 version:15 > peers:<id:17 store_id:1 > peers:<id:18 store_id:4 > peers:<id:19 store_id:5 > , invalid marker byte, group bytes "jfs-tikv-"
goroutine 1341 [running, locked to thread]:
runtime/debug.Stack()
	/home/work/go/src/runtime/debug/stack.go:24 +0x65
github.com/juicedata/juicefs/pkg/meta.errno({0x7f9d8626c2a0, 0xc001a3eb58})
	/home/work/juicefs/pkg/meta/utils.go:122 +0xad
github.com/juicedata/juicefs/pkg/meta.(*kvMeta).doMknod(0xc000838a00, {0x7f9d8629ef48?, 0xc0036a2cc0}, 0x13b2d, {0xc0038b06b3, 0x8}, 0x1, 0x1b6, 0x12, 0x0, ...)
	/home/work/juicefs/pkg/meta/tkv.go:1120 +0x312
github.com/juicedata/juicefs/pkg/meta.(*baseMeta).Mknod(0xc0013ea000, {0x7f9d8629ef48, 0xc0036a2cc0}, 0x13b2d, {0xc0038b06b3, 0x8}, 0x90?, 0x38b?, 0xc0?, 0x0, ...)
	/home/work/juicefs/pkg/meta/base.go:1344 +0x2d5
github.com/juicedata/juicefs/pkg/meta.(*baseMeta).Create(0xc0013ea000, {0x7f9d8629ef48?, 0xc0036a2cc0?}, 0xc0038b0690?, {0xc0038b06b3?, 0x7f9d857f3e38?}, 0x4101?, 0x82e1?, 0x80, 0xc002a98648, ...)
	/home/work/juicefs/pkg/meta/base.go:1357 +0x109
github.com/juicedata/juicefs/pkg/fs.(*FileSystem).Create(0xc000a7e0a0, {0x7f9d8629ef48?, 0xc0036a2cc0}, {0xc0038b0690, 0x2b}, 0x1b6, 0x5f1?)
	/home/work/juicefs/pkg/fs/fs.go:790 +0x436
main.jfs_create(0x7f9d82d697f8?, 0xc000068060?, 0x0?, 0x6901?, 0x428?)
	/home/work/juicefs/sdk/java/libjfs/main.go:741 +0xb0 [utils.go:122]

【资源配置】*进入到 TiDB Dashboard -集群信息 (Cluster Info) -主机(Hosts) 截图此页面*
【附件：截图/日志/监控】
![image|690x279](upload://98ahB5yKoeY99TKRj2Di7IUMUvy.png)
2

这是 juicefs 的错误，请去 juicefs 社区寻找一下帮助

如果有类似 tikv，tidb，pd 节点实例的错误日志，或者关键错误信息，可以提供出来，这个可以帮助排查…

4

这个错误应该是 TiKV 的 client-go 检测到 start_key 的编码不对。key 应该以 “Memory Comparable Encoding” 方式编码，参考 TiKV 源码解析系列文章（十三）MVCC 数据读取 | PingCAP。
end_key 的编码是对的。

造成这个问题的原因可能是错误的使用了 ravwkv api 写入数据，或者不正确的 split region。

1 个赞
5

嗯。根据错误日志，发生了一个编码错误，JuiceFS 中的某个关键字未正确编码。可能是由于配置或环境问题导致的

6

感谢回复。
我这边搭建的tikv是一个测试集群，没有往进写其它数据，只做为存储juicefs的元数据使用。之前在juicefs社区了解到juicefs不会使用rowkv api,您指出的split region不正确，是有什么split的规则么？有配置可以调整么？
我刚接触tikv，不太懂，再次感谢

7

感谢回复。
我是按照tikv的社区文档，按照二进制文件的形式搭的服务，没有加其它配置，您说的配置错误有没有具体的错误类型或者可以修改的配置？刚开始接触tikv，不太懂。谢谢。

8

最开始写失败的报错是这样的

image
image1818×431 124 KB 

2023/09/05 18:50:32.608902 juicefs[182686] <ERROR>: error: newRegion's range key is not encoded: id:20 start_key:"0904\375Aa|\000\000\000\000\000\000I" end_key:"0904\375A\254Z\377\000\000\000\000\000\000C\000\377\000\000\002\000\000\000\000\000\372" region_epoch:<conf_ver:5 version:27 > peers:<id:21 store_id:1 > peers:<id:22 store_id:4 > peers:<id:23 store_id:5 > , invalid marker byte, group bytes "0904\xfdAa|\x00"
goroutine 365 [running, locked to thread]:
runtime/debug.Stack()
	/home/work/go/src/runtime/debug/stack.go:24 +0x65
github.com/juicedata/juicefs/pkg/meta.errno({0x7fdc5dd722a0, 0xc0058af3f8})
	/home/work/juicefs/pkg/meta/utils.go:122 +0xad
github.com/juicedata/juicefs/pkg/meta.(*kvMeta).doLookup(0xc00031ee80, {0xc00236c330?, 0x7c3a?}, 0xc0043b4240?, {0xc00575a426, 0xa}, 0xc0035e0310, 0xc006420370)
	/home/work/juicefs/pkg/meta/tkv.go:853 +0x378
github.com/juicedata/juicefs/pkg/meta.(*baseMeta).Lookup(0xc002400000, {0x7fdc5dda4f48, 0xc003b90780}, 0x7c3a, {0xc00575a426, 0xa}, 0xc0035e0310, 0xc006420370, 0x0)
	/home/work/juicefs/pkg/meta/base.go:1114 +0x4b7
github.com/juicedata/juicefs/pkg/fs.(*FileSystem).lookup(0xc001a3a000, {0x7fdc5dda4f48, 0xc003b90780}, 0x1?, {0xc00575a426, 0xa}, 0xc0035e0310, 0xc006420370)
	/home/work/juicefs/pkg/fs/fs.go:660 +0x4db
github.com/juicedata/juicefs/pkg/fs.(*FileSystem).doResolve(0xc001a3a000, {0x7fdc5dda4f48, 0xc003b90780}, {0xc00575a420, 0x22}, 0x1, 0x0)
	/home/work/juicefs/pkg/fs/fs.go:734 +0x40d
github.com/juicedata/juicefs/pkg/fs.(*FileSystem).resolve(...)
	/home/work/juicefs/pkg/fs/fs.go:684
github.com/juicedata/juicefs/pkg/fs.(*FileSystem).Create(0xc001a3a000, {0x7fdc5dda4f48?, 0xc003b90780}, {0xc00575a420, 0x2b}, 0x1b6, 0x6fd?)
	/home/work/juicefs/pkg/fs/fs.go:786 +0x2e7
main.jfs_create(0x7fdc5a86f7f8?, 0xc000068060?, 0x0?, 0x7d01?, 0x113?)
	/home/work/juicefs/sdk/java/libjfs/main.go:741 +0xb0 [utils.go:122]

再次访问就报错如下

image
image1765×272 73.2 KB 

2023/09/05 19:13:07.337504 juicefs[154644] <ERROR>: error: failed to decode region range key, key: "30393034FD41010000000000000044", err: invalid marker byte, group bytes "0904\xfdAa|\x00"
goroutine 17 [running, locked to thread]:
runtime/debug.Stack()
	/home/work/go/src/runtime/debug/stack.go:24 +0x65
github.com/juicedata/juicefs/pkg/meta.errno({0x7fa3874f02a0, 0xc001b9a678})
	/home/work/juicefs/pkg/meta/utils.go:122 +0xad
github.com/juicedata/juicefs/pkg/meta.(*kvMeta).doReaddir(0xc002590060, {0x7fa387522f48?, 0xc001544680?}, 0x59?, 0x1, 0xc0010b4558, 0x7fa386edda20?)
	/home/work/juicefs/pkg/meta/tkv.go:1781 +0x29c
github.com/juicedata/juicefs/pkg/meta.(*baseMeta).Readdir(0xc0025aa000, {0x7fa387522f48, 0xc001544680}, 0x1, 0x1, 0xc0010b4558)
	/home/work/juicefs/pkg/meta/base.go:1759 +0x484
github.com/juicedata/juicefs/pkg/fs.(*File).ReaddirPlus(0xc000f02b40, {0x7fa387522f48?, 0xc001544680}, 0x0)
	/home/work/juicefs/pkg/fs/fs.go:1105 +0x289
main.jfs_listdir(0xc000943e28?, 0xc000943e28?, 0x7fa383fed7f8?, 0x0, 0x7fa3fdb48880, 0x8000)
	/home/work/juicefs/sdk/java/libjfs/main.go:1054 +0x1f4 [utils.go:122]
ls: jfs://jfs/