br增量备份数据量爆增300倍，请教下原理，还是触发了BUG？

老龚caiji · 2022 年10 月 31 日 02:53

【 TiDB 使用环境】生产
【 TiDB 版本】
Cluster version: v4.0.9
【问题现象及影响】
环境是：每月1号全量备份，1号后每天按1号的全量进行增量备份。
正常情况下每天增量的数据大概为5G不到。
突然有一天，数据量爆增了非常多，正常情况下应该1小时左右就备份完成。但今天备份了一天一夜还没完成。备份的数据量达到了3.5T（备份日志进度才72%），时间太久现在进程终结了。
请问这是发生了什么？或者说什么情况下会导致这样。因为我们的数据修改量不会这么大的想不明白。
还有看BR备份的日志都是正常的，真有这么大的数据吗？为啥过了一天，我再重新增量备份，速度就快了很多。

【附件】

备份日志文件太大了，截取了最后一小部分日志如下：
[2022/10/31 10:54:15.956 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e2b841983] [EndKey=7480000000000008b65f728000000e2b8872a8]
[2022/10/31 10:54:15.982 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e3c8421ba] [EndKey=7480000000000008b65f728000000e3c867304]
[2022/10/31 10:54:16.010 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000ddc683a87] [EndKey=7480000000000008b65f728000000ddc6ab9c9]
[2022/10/31 10:54:16.010 +08:00] [INFO] [rtree.go:136] [“delete overlapping range”] [StartKey=7480000000000008b65f728000000ddc683a87] [EndKey=7480000000000008b65f728000000ddc6ab9c9]
[2022/10/31 10:54:16.131 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000ddd10baef] [EndKey=7480000000000008b65f728000000ddd13400f]
[2022/10/31 10:54:16.171 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e04465be1] [EndKey=7480000000000008b65f728000000e0449f706]
[2022/10/31 10:54:16.179 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e305724aa] [EndKey=7480000000000008b65f728000000e305ee961]
[2022/10/31 10:54:16.203 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000ddb94a591] [EndKey=7480000000000008b65f728000000ddb9a81e3]
[2022/10/31 10:54:16.205 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e3d99fbae] [EndKey=7480000000000008b65f728000000e3d9bc210]
[2022/10/31 10:54:16.279 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e06f74a6a] [EndKey=7480000000000008b65f728000000e06fd5e73]
[2022/10/31 10:54:16.302 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e4339ac84] [EndKey=7480000000000008b65f728000000e433e87df]
[2022/10/31 10:54:16.303 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e0ca19883] [EndKey=7480000000000008b65f728000000e0ca4d1e9]
[2022/10/31 10:54:16.305 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000dfa79fdf7] [EndKey=7480000000000008b65f728000000dfa855e78]
[2022/10/31 10:54:16.309 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e2c5ab32d] [EndKey=7480000000000008b65f728000000e2c5d70b8]
[2022/10/31 10:54:16.334 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e07a61d42] [EndKey=7480000000000008b65f728000000e07aa05e6]
[2022/10/31 10:54:16.337 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e404f346a] [EndKey=7480000000000008b65f728000000e4057f4a7]
[2022/10/31 10:54:16.354 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e3e82ae44] [EndKey=7480000000000008b65f728000000e3e855d8c]
[2022/10/31 10:54:16.368 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e2e1592db] [EndKey=7480000000000008b65f728000000e2e198fab]
[2022/10/31 10:54:16.381 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e4691b5f6] [EndKey=7480000000000008b65f728000000e469417b5]
[2022/10/31 10:54:16.386 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e0366a4a6] [EndKey=7480000000000008b65f728000000e03694a76]
[2022/10/31 10:54:16.393 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000ddfd61e13] [EndKey=7480000000000008b65f728000000ddfde9c7f]
[2022/10/31 10:54:16.393 +08:00] [INFO] [rtree.go:136] [“delete overlapping range”] [StartKey=7480000000000008b65f728000000ddfd61e13] [EndKey=7480000000000008b65f728000000ddfde9c7f]
[2022/10/31 10:54:16.411 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e05ed2151] [EndKey=7480000000000008b65f728000000e05ee8a07]
[2022/10/31 10:54:16.425 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000ddb48ae17] [EndKey=7480000000000008b65f728000000ddb4bf69d]
[2022/10/31 10:54:16.450 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e45d771f9] [EndKey=7480000000000008b65f728000000e45d9f35a]
[2022/10/31 10:54:16.460 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000de06f9be6] [EndKey=7480000000000008b65f728000000de0794584]
[2022/10/31 10:54:16.476 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e06af1e41] [EndKey=7480000000000008b65f728000000e06b41f67]
[2022/10/31 10:54:16.485 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e2ef529a9] [EndKey=7480000000000008b65f728000000e2f01c67f]
[2022/10/31 10:54:16.494 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e06c011d9] [EndKey=7480000000000008b65f728000000e06c23727]
[2022/10/31 10:54:16.506 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e2f01c67f] [EndKey=7480000000000008b65f728000000e2f097137]
[2022/10/31 10:54:16.510 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000de084c908] [EndKey=7480000000000008b65f728000000de086d3b7]
[2022/10/31 10:54:16.542 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e41c49a6f] [EndKey=7480000000000008b65f728000000e41c7e842]
[2022/10/31 10:54:16.545 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e07762a06] [EndKey=7480000000000008b65f728000000e077902b7]
[2022/10/31 10:54:16.569 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000f4dfd3950] [EndKey=7480000000000008b65f728000000f4dfe9061]
[2022/10/31 10:54:16.584 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e11afe229] [EndKey=7480000000000008b65f728000000e11b4ffda]
[2022/10/31 10:54:16.594 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000f510b054b] [EndKey=7480000000000008b65f728000000f510d9903]
[2022/10/31 10:54:16.600 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e3eabe032] [EndKey=7480000000000008b65f728000000e3eae5c76]
[2022/10/31 10:54:16.637 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000dd9368f6a] [EndKey=7480000000000008b65f728000000dd9377bd3]
[2022/10/31 10:54:16.645 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000dfcb8619d] [EndKey=7480000000000008b65f728000000dfccaf49c]
[2022/10/31 10:54:16.658 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000f4d95fb13] [EndKey=7480000000000008b65f728000000f4d9734df]
[2022/10/31 10:54:16.688 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e0ee25d53] [EndKey=7480000000000008b65f728000000e0ee6f81e]
[2022/10/31 10:54:16.700 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e01ce4963] [EndKey=7480000000000008b65f728000000e01d24be9]
[2022/10/31 10:54:16.702 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000dd95aee39] [EndKey=7480000000000008b65f728000000dd95ce4b2]
[2022/10/31 10:54:16.704 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000ddecf9ed3] [EndKey=7480000000000008b65f728000000dded3420f]
[2022/10/31 10:54:16.707 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e0eed88e3] [EndKey=7480000000000008b65f728000000e0ef06969]
[2022/10/31 10:54:16.717 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000f4eb7e704] [EndKey=7480000000000008b65f728000000f4eb9cd6d]
[2022/10/31 10:54:16.755 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000ddf6c522c] [EndKey=7480000000000008b65f728000000ddf6dbdcb]
[2022/10/31 10:54:16.762 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e41d81649] [EndKey=7480000000000008b65f728000000e41d9c7b7]
[2022/10/31 10:54:16.774 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000de12d6c99] [EndKey=7480000000000008b65f728000000de130da88]
[2022/10/31 10:54:16.780 +08:00] [INFO] [coprocessor.go:1034] [“[TIME_COP_WAIT] resp_time:571.753461ms txnStartTS:437042506932682754 region_id:21 store_addr:192.168.241.61:20160 kv_wait_ms:571”]
[2022/10/31 10:54:16.804 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e09fd3df0] [EndKey=7480000000000008b65f728000000e0a02a2a3]
[2022/10/31 10:54:16.820 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e03721eef] [EndKey=7480000000000008b65f728000000e0379a9fd]
[2022/10/31 10:54:16.825 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e01eb81f8] [EndKey=7480000000000008b65f728000000e01f4817b]
[2022/10/31 10:54:16.870 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e2d54227e] [EndKey=7480000000000008b65f728000000e2d593731]
[2022/10/31 10:54:16.871 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e020bb4b0] [EndKey=7480000000000008b65f728000000e021ed59b]
[2022/10/31 10:54:17.016 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e464077f4] [EndKey=7480000000000008b65f728000000e4644bb8d]
[2022/10/31 10:54:17.028 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000f4b8f7742] [EndKey=7480000000000008b65f728000000f4b922b82]
[2022/10/31 10:54:17.028 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000f4fd2d865] [EndKey=7480000000000008b65f728000000f4fd8efa7]
[2022/10/31 10:54:17.035 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000f4ce8a472] [EndKey=7480000000000008b65f728000000f4cea42ee]
[2022/10/31 10:54:17.038 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e3e2f7b8a] [EndKey=7480000000000008b65f728000000e3e32aa00]
[2022/10/31 10:54:17.054 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000dfc10fad9] [EndKey=7480000000000008b65f728000000dfc14ee98]
[2022/10/31 10:54:17.054 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e03d39954] [EndKey=7480000000000008b65f728000000e03d5de0b]
[2022/10/31 10:54:17.058 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000dd9938f5c] [EndKey=7480000000000008b65f728000000dd997d556]
[2022/10/31 10:54:17.077 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e04d278d9] [EndKey=7480000000000008b65f728000000e04d8da43]
[2022/10/31 10:54:17.110 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e336d5b43] [EndKey=7480000000000008b65f728000000e336f7adb]
[2022/10/31 10:54:17.113 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e475bed20] [EndKey=7480000000000008b65f728000000e475f072f]
[2022/10/31 10:54:17.161 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e2f822f14] [EndKey=7480000000000008b65f728000000e2f8ceb74]
[2022/10/31 10:54:17.198 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000f4ad55844] [EndKey=7480000000000008b65f728000000f4ad80589]
[2022/10/31 10:54:17.202 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e392833dc] [EndKey=7480000000000008b65f728000000e392faa18]
[2022/10/31 10:54:17.227 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e2ad0e8a2] [EndKey=7480000000000008b65f728000000e2ad43709]
[2022/10/31 10:54:17.231 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000dfe9fbadd] [EndKey=7480000000000008b65f728000000dfea458c6]
[2022/10/31 10:54:17.232 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e133cf774] [EndKey=7480000000000008b65f728000000e134128d3]
[2022/10/31 10:54:17.232 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000de19c2702] [EndKey=7480000000000008b65f728000000de19f9909]
[2022/10/31 10:54:17.275 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000f4f0dbb38] [EndKey=7480000000000008b65f728000000f4f138dee]
[2022/10/31 10:54:17.295 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e42424113] [EndKey=7480000000000008b65f728000000e424a7d93]
[2022/10/31 10:54:17.312 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e039faf79] [EndKey=7480000000000008b65f728000000e03a0ccbf]
[2022/10/31 10:54:17.315 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000f4d1ba61a] [EndKey=7480000000000008b65f728000000f4d1cc3d5]
[2022/10/31 10:54:17.322 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e4859af7b] [EndKey=7480000000000008b65f728000000e485c8f57]
[2022/10/31 10:54:17.324 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e012600a0] [EndKey=7480000000000008b65f728000000e0128521a]
[2022/10/31 10:54:17.336 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000f4e73f58b] [EndKey=7480000000000008b65f728000000f4e753b69]
[2022/10/31 10:54:17.336 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000f4d4c663d] [EndKey=7480000000000008b65f728000000f4d4f11a4]
[2022/10/31 10:54:17.363 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e08d9a4fe] [EndKey=7480000000000008b65f728000000e08e45d29]
[2022/10/31 10:54:17.363 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000dfee33269] [EndKey=7480000000000008b65f728000000dfee84e72]
[2022/10/31 10:54:17.374 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000dfe87515c] [EndKey=7480000000000008b65f728000000dfe89e906]
[2022/10/31 10:54:17.385 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e0d9fa89c] [EndKey=7480000000000008b65f728000000e0da6d18f]
[2022/10/31 10:54:17.392 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e27efac38] [EndKey=7480000000000008b65f728000000e27f10bbb]
[2022/10/31 10:54:17.404 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e415432b6] [EndKey=7480000000000008b65f728000000e41568a62]
[2022/10/31 10:54:17.407 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e0909fe73] [EndKey=7480000000000008b65f728000000e090e03d3]
[2022/10/31 10:54:17.423 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e0915082f] [EndKey=7480000000000008b65f728000000e0918adfd]
[2022/10/31 10:54:17.447 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e344eda5e] [EndKey=7480000000000008b65f728000000e34515843]
[2022/10/31 10:54:17.463 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e4cf651ec] [EndKey=7480000000000008b65f728000000e4cf86854]
[2022/10/31 10:54:17.468 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e34690e97] [EndKey=7480000000000008b65f728000000e346aee68]
[2022/10/31 10:54:17.497 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e07fd3902] [EndKey=7480000000000008b65f728000000e0804960d]
[2022/10/31 10:54:17.497 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e05fc39c2] [EndKey=7480000000000008b65f728000000e0601bcc5]
[2022/10/31 10:54:17.510 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e0ec7121e] [EndKey=7480000000000008b65f728000000e0ec99014]
[2022/10/31 10:54:17.510 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e05ea92a3] [EndKey=7480000000000008b65f728000000e05ed2151]
[2022/10/31 10:54:17.510 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e3f1dd998] [EndKey=7480000000000008b65f728000000e3f20542a]
[2022/10/31 10:54:17.519 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e090e03d3] [EndKey=7480000000000008b65f728000000e0915082f]
[2022/10/31 10:54:17.522 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e0615656c] [EndKey=7480000000000008b65f728000000e06179b52]
[2022/10/31 10:54:17.563 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e10807f54] [EndKey=7480000000000008b65f728000000e10878aa8]
[2022/10/31 10:54:17.600 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e322190e0] [EndKey=7480000000000008b65f728000000e322733d2]
[2022/10/31 10:54:17.600 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e32a89cfd] [EndKey=7480000000000008b65f728000000e32ac89da]
[2022/10/31 10:54:17.623 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e32420be1] [EndKey=7480000000000008b65f728000000e32457c79]
[2022/10/31 10:54:17.624 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f728000000e464c1f38] [EndKey=7480000000000008b65f728000000e464d7f08]
[2022/10/31 10:54:17.656 +08:00] [INFO] [client.go:878] [“range backuped”] [StartKey=7480000000000008b65f6980000000000000010380000000000495ec01687474703a2f2f6dff702e77656978696eff2e71712e636f6d2fff733f5f5f62697a3dff4d7a41784e544131ff4e6a63344d773d3dff266d69643d323635ff3038363538393826ff6964783d3126736eff3d63313932623031ff3835316534353631ff3231613233343334ff3765633238373037ff3200000000000000f8] [EndKey=7480000000000008b65f6980000000000000010380000000000495ec01687474703a2f2f6dff702e77656978696eff2e71712e636f6d2fff733f5f5f62697a3dff4d7a49324f54417aff4d6a63774f413d3dff266d69643d323635ff3130313138373426ff6964783d3126736eff3d37396565633162ff3433313239663930ff3034653161323064ff3332666164613434ff3000000000000000f8]
^C
[tidb@b49 ~]$
[tidb@b49 ~]$
[tidb@b49 ~]$ tailf tidb-toolkit-v4.0.9-linux-amd64/logs/backup-bsppr-incr-2022-10-30.log
[2022/10/31 10:39:11.647 +08:00] [INFO] [client.go:828] [“try backup”] [StartKey=7480000000000008b65f728000000fdf5c9c5f] [EndKey=7480000000000008b65f728000000fdf5da3f3] [storeID=113105576] [“retry time”=1]
[2022/10/31 10:39:11.647 +08:00] [INFO] [conn.go:223] [“Reset backup client”] [storeID=113105576]
[2022/10/31 10:39:11.648 +08:00] [INFO] [client.go:828] [“try backup”] [StartKey=7480000000000008b65f69800000000000000403800000000040d317038000000f8d5b9ebb] [EndKey=7480000000000008b65f69800000000000000403800000000040d32f038000000f8f2f0eb3] [storeID=113105576] [“retry time”=1]
[2022/10/31 10:39:11.648 +08:00] [INFO] [conn.go:223] [“Reset backup client”] [storeID=113105576]
[2022/10/31 10:39:11.650 +08:00] [INFO] [client.go:828] [“try backup”] [StartKey=7480000000000008b65f728000000fdf2d848e] [EndKey=7480000000000008b65f728000000fdf2f44f7] [storeID=113105576] [“retry time”=1]
[2022/10/31 10:39:11.650 +08:00] [INFO] [conn.go:223] [“Reset backup client”] [storeID=113105576]
[2022/10/31 10:39:11.651 +08:00] [INFO] [client.go:828] [“try backup”] [StartKey=7480000000000008b65f728000000fdf5a425c] [EndKey=7480000000000008b65f728000000fdf5afacb] [storeID=113105576] [“retry time”=1]
[2022/10/31 10:39:11.651 +08:00] [INFO] [conn.go:223] [“Reset backup client”] [storeID=113105576]
[2022/10/31 10:39:11.652 +08:00] [INFO] [client.go:828] [“try backup”] [StartKey=7480000000000008b65f728000000fdf4b53ed] [EndKey=7480000000000008b65f728000000fdf4c6848] [storeID=113105576] [“retry time”=1]
[2022/10/31 10:39:11.923 +08:00] [INFO] [coprocessor.go:1034] [“[TIME_COP_WAIT] resp_time:317.020322ms txnStartTS:437042269797220354 region_id:164891523 store_addr:192.168.241.61:20160 kv_wait_ms:317”]
^C
[tidb@b49 ~]$ tail -100f tidb-toolkit-v4.0.9-linux-amd64/logs/backup-bsppr-incr-2022-10-30.log
[2022/10/31 10:38:59.946 +08:00] [INFO] [conn.go:223] [“Reset backup client”] [storeID=113105576]
[2022/10/31 10:38:59.948 +08:00] [INFO] [client.go:828] [“try backup”] [StartKey=7480000000000008b65f69800000000000000301687474703a2f2f77ff77772e646f757969ff6e2e636f6d2f7669ff64656f2f37313534ff3935333932363233ff3034343533353200fe038000000fafaef1b2] [EndKey=7480000000000008b65f69800000000000000301687474703a2f2f77ff77772e646f757969ff6e2e636f6d2f7669ff64656f2f37313535ff3034323331373030ff3936383537363800fe038000000fb127fafa] [storeID=113105576] [“retry time”=3]
[2022/10/31 10:39:02.219 +08:00] [WARN] [client.go:806] [“reset the connection in handleFineGrained”] [storeID=113105576]
[2022/10/31 10:39:02.219 +08:00] [INFO] [conn.go:223] [“Reset backup client”] [storeID=113105576]
[2022/10/31 10:39:02.220 +08:00] [WARN] [client.go:806] [“reset the connection in handleFineGrained”] [storeID=113105576]
[2022/10/31 10:39:02.223 +08:00] [WARN] [client.go:806] [“reset the connection in handleFineGrained”] [storeID=113105576]
[2022/10/31 10:39:02.224 +08:00] [WARN] [client.go:806] [“reset the connection in handleFineGrained”] [storeID=113105576]
[2022/10/31 10:39:02.228 +08:00] [INFO] [client.go:828] [“try backup”] [StartKey=7480000000000008b65f69800000000000000403800000000040d317038000000f8d5b9ebb] [EndKey=7480000000000008b65f69800000000000000403800000000040d32f038000000f8f2f0eb3] [storeID=113105576] [“retry time”=3]
[2022/10/31 10:39:02.228 +08:00] [INFO] [conn.go:223] [“Reset backup client”] [storeID=113105576]
[2022/10/31 10:39:02.237 +08:00] [INFO] [client.go:828] [“try backup”] [StartKey=7480000000000008b65f728000000fe178ef09] [EndKey=7480000000000008b65f728000000fe17bdfc0] [storeID=113105576] [“retry time”=3]
[2022/10/31 10:39:02.237 +08:00] [INFO] [conn.go:223] [“Reset backup client”] [storeID=113105576]
[2022/10/31 10:39:02.245 +08:00] [INFO] [client.go:828] [“try backup”] [StartKey=7480000000000008b65f69800000000000000103800000000004ab6c01687474703a2f2f77ff77772e736f68752eff636f6d2f612f3538ff393833383039395fff3131353336323f6dff785f636f6d6d656eff745f69643d663362ff6131373437316438ff6233386530386230ff3165366166663262ff6162656461346662ff6661346262363038ff6537363339336165ff3265323336353333ff3339323536000000fc] [EndKey=7480000000000008b65f69800000000000000103800000000004ab6c01687474703a2f2f77ff77772e7869616f68ff6f6e677368752e63ff6f6d2f646973636fff766572792f697465ff6d2f363333393136ff3630303030303030ff3030316430313364ff6165000000000000f9] [storeID=113105576] [“retry time”=3]
[2022/10/31 10:39:02.245 +08:00] [INFO] [conn.go:223] [“Reset backup client”] [storeID=113105576]
[2022/10/31 10:39:02.251 +08:00] [WARN] [client.go:806] [“reset the connection in handleFineGrained”] [storeID=113105576]
[2022/10/31 10:39:02.254 +08:00] [INFO] [client.go:828] [“try backup”] [StartKey=7480000000000008b65f69800000000000000103800000000004ab6b01687474703a2f2f77ff77772e7869616f68ff6f6e677368752e63ff6f6d2f646973636fff766572792f697465ff6d2f363333323961ff6131303030303030ff3030313730313763ff6137000000000000f9] [EndKey=7480000000000008b65f69800000000000000103800000000004ab6c01687474703a2f2f70ff6f73742e6d2e736dff7a646d2e636f6d2fff702f616771363767ff7064000000000000f9] [storeID=113105576] [“retry time”=3]
[2022/10/31 10:39:02.254 +08:00] [INFO] [conn.go:223] [“Reset backup client”] [storeID=113105576]
[2022/10/31 10:39:02.262 +08:00] [INFO] [client.go:828] [“try backup”] [StartKey=7480000000000008b65f728000000fe14ddcc7] [EndKey=7480000000000008b65f728000000fe14ed036] [storeID=113105576] [“retry time”=4]
[2022/10/31 10:39:02.643 +08:00] [WARN] [client.go:806] [“reset the connection in handleFineGrained”] [storeID=113105576]
[2022/10/31 10:39:02.643 +08:00] [INFO] [conn.go:223] [“Reset backup client”] [storeID=113105576]
[2022/10/31 10:39:02.645 +08:00] [INFO] [client.go:828] [“try backup”] [StartKey=7480000000000008b65f69800000000000000103800000000004ab6601687474703a2f2f77ff77772e646f757969ff6e2e636f6d2f7669ff64656f2f37313533ff3431353335363733ff343838353135303fff6d785f636f6d6d65ff6e745f69643d3162ff3538613762636536ff6462643166336163ff3631316330346632ff3334326637393636ff3135336561396238ff6231316464323666ff6532616364636263ff6264616437340000fd] [EndKey=7480000000000008b65f69800000000000000103800000000004ab6601687474703a2f2f77ff77772e646f757969ff6e2e636f6d2f7669ff64656f2f37313534ff3330343435313235ff383337313333393fff6d785f636f6d6d65ff6e745f69643d3666ff3932346161623161ff3632623561363535ff3336323266346162ff3564643164646561ff3230393036396335ff3138393637393066ff3361316165326530ff3933343763320000fd] [storeID=113105576] [“retry time”=3]
[2022/10/31 10:39:02.740 +08:00] [INFO] [coprocessor.go:1034] [“[TIME_COP_WAIT] resp_time:424.220843ms txnStartTS:437042267372388353 region_id:163845902 store_addr:192.168.241.61:20160 kv_wait_ms:387”]
[2022/10/31 10:39:02.740 +08:00] [INFO] [coprocessor.go:1034] [“[TIME_COP_WAIT] resp_time:423.612766ms txnStartTS:437042267372388353 region_id:158953886 store_addr:192.168.241.61:20160 kv_wait_ms:388”]
[2022/10/31 10:39:02.946 +08:00] [WARN] [client.go:806] [“reset the connection in handleFineGrained”] [storeID=113105576]
[2022/10/31 10:39:02.946 +08:00] [INFO] [conn.go:223] [“Reset backup client”] [storeID=113105576]
[2022/10/31 10:39:02.948 +08:00] [INFO] [client.go:828] [“try backup”] [StartKey=7480000000000008b65f728000000fe158ed2f] [EndKey=7480000000000008b65f728000000fe15b98fd] [storeID=113105576] [“retry time”=4]
[2022/10/31 10:39:05.219 +08:00] [WARN] [client.go:806] [“reset the connection in handleFineGrained”] [storeID=113105576]
[2022/10/31 10:39:05.219 +08:00] [INFO] [conn.go:223] [“Reset backup client”] [storeID=113105576]
[2022/10/31 10:39:05.224 +08:00] [INFO] [client.go:828] [“try backup”] [StartKey=7480000000000008b65f69800000000000000301687474703a2f2f77ff77772e646f757969ff6e2e636f6d2f7669ff64656f2f37313534ff3935333932363233ff3034343533353200fe038000000fafaef1b2] [EndKey=7480000000000008b65f69800000000000000301687474703a2f2f77ff77772e646f757969ff6e2e636f6d2f7669ff64656f2f37313535ff3034323331373030ff3936383537363800fe038000000fb127fafa] [storeID=113105576] [“retry time”=4]
[2022/10/31 10:39:05.228 +08:00] [WARN] [client.go:806] [“reset the connection in handleFineGrained”] [storeID=113105576]
[2022/10/31 10:39:05.228 +08:00] [INFO] [conn.go:223] [“Reset backup client”] [storeID=113105576]
[2022/10/31 10:39:05.229 +08:00] [INFO] [client.go:828] [“try backup”] [StartKey=7480000000000008b65f69800000000000000403800000000040d317038000000f8d5b9ebb] [EndKey=7480000000000008b65f69800000000000000403800000000040d32f038000000f8f2f0eb3] [storeID=113105576] [“retry time”=4]
[2022/10/31 10:39:05.237 +08:00] [WARN] [client.go:806] [“reset the connection in handleFineGrained”] [storeID=113105576]
[2022/10/31 10:39:05.237 +08:00] [INFO] [conn.go:223] [“Reset backup client”] [storeID=113105576]
[2022/10/31 10:39:05.239 +08:00] [INFO] [client.go:828] [“try backup”] [StartKey=7480000000000008b65f728000000fe178ef09] [EndKey=7480000000000008b65f728000000fe17bdfc0] [storeID=113105576] [“retry time”=4]
[2022/10/31 10:39:05.246 +08:00] [WARN] [client.go:806] [“reset the connection in handleFineGrained”] [storeID=113105576]
[2022/10/31 10:39:05.246 +08:00] [INFO] [conn.go:223] [“Reset backup client”] [storeID=113105576]
[2022/10/31 10:39:05.247 +08:00] [INFO] [client.go:828] [“try backup”] [StartKey=7480000000000008b65f69800000000000000103800000000004ab6c01687474703a2f2f77ff77772e736f68752eff636f6d2f612f3538ff393833383039395fff3131353336323f6dff785f636f6d6d656eff745f69643d663362ff6131373437316438ff6233386530386230ff3165366166663262ff6162656461346662ff6661346262363038ff6537363339336165ff3265323336353333ff3339323536000000fc] [EndKey=7480000000000008b65f69800000000000000103800000000004ab6c01687474703a2f2f77ff77772e7869616f68ff6f6e677368752e63ff6f6d2f646973636fff766572792f697465ff6d2f363333393136ff3630303030303030ff3030316430313364ff6165000000000000f9] [storeID=113105576] [“retry time”=4]
[2022/10/31 10:39:05.254 +08:00] [WARN] [client.go:806] [“reset the connection in handleFineGrained”] [storeID=113105576]
[2022/10/31 10:39:05.254 +08:00] [INFO] [conn.go:223] [“Reset backup client”] [storeID=113105576]
[2022/10/31 10:39:05.256 +08:00] [INFO] [client.go:828] [“try backup”] [StartKey=7480000000000008b65f69800000000000000103800000000004ab6b01687474703a2f2f77ff77772e7869616f68ff6f6e677368752e63ff6f6d2f646973636fff766572792f697465ff6d2f363333323961ff6131303030303030ff3030313730313763ff6137000000000000f9] [EndKey=7480000000000008b65f69800000000000000103800000000004ab6c01687474703a2f2f70ff6f73742e6d2e736dff7a646d2e636f6d2fff702f616771363767ff7064000000000000f9] [storeID=113105576] [“retry time”=4]
[2022/10/31 10:39:05.643 +08:00] [WARN] [client.go:806] [“reset the connection in handleFineGrained”] [storeID=113105576]
[2022/10/31 10:39:05.644 +08:00] [INFO] [conn.go:223] [“Reset backup client”] [storeID=113105576]
[2022/10/31 10:39:05.947 +08:00] [WARN] [client.go:806] [“reset the connection in handleFineGrained”] [storeID=113105576]
[2022/10/31 10:39:05.947 +08:00] [INFO] [conn.go:223] [“Reset backup client”] [storeID=113105576]
[2022/10/31 10:39:06.131 +08:00] [INFO] [client.go:828] [“try backup”] [StartKey=7480000000000008b65f69800000000000000103800000000004ab6601687474703a2f2f77ff77772e646f757969ff6e2e636f6d2f7669ff64656f2f37313533ff3431353335363733ff343838353135303fff6d785f636f6d6d65ff6e745f69643d3162ff3538613762636536ff6462643166336163ff3631316330346632ff3334326637393636ff3135336561396238ff6231316464323666ff6532616364636263ff6264616437340000fd] [EndKey=7480000000000008b65f69800000000000000103800000000004ab6601687474703a2f2f77ff77772e646f757969ff6e2e636f6d2f7669ff64656f2f37313534ff3330343435313235ff383337313333393fff6d785f636f6d6d65ff6e745f69643d3666ff3932346161623161ff3632623561363535ff3336323266346162ff3564643164646561ff3230393036396335ff3138393637393066ff3361316165326530ff3933343763320000fd] [storeID=113105576] [“retry time”=4]
[2022/10/31 10:39:08.220 +08:00] [WARN] [client.go:806] [“reset the connection in handleFineGrained”] [storeID=113105576]
[2022/10/31 10:39:08.220 +08:00] [INFO] [conn.go:223] [“Reset backup client”] [storeID=113105576]
[2022/10/31 10:39:08.229 +08:00] [WARN] [client.go:806] [“reset the connection in handleFineGrained”] [storeID=113105576]
[2022/10/31 10:39:08.238 +08:00] [WARN] [client.go:806] [“reset the connection in handleFineGrained”] [storeID=113105576]
[2022/10/31 10:39:08.246 +08:00] [WARN] [client.go:806] [“reset the connection in handleFineGrained”] [storeID=113105576]
[2022/10/31 10:39:08.255 +08:00] [WARN] [client.go:806] [“reset the connection in handleFineGrained”] [storeID=113105576]
[2022/10/31 10:39:08.265 +08:00] [INFO] [conn.go:223] [“Reset backup client”] [storeID=113105576]
[2022/10/31 10:39:08.282 +08:00] [INFO] [conn.go:223] [“Reset backup client”] [storeID=113105576]
[2022/10/31 10:39:08.287 +08:00] [INFO] [client.go:603] [“start fine grained backup”] [incomplete=1]
[2022/10/31 10:39:08.288 +08:00] [INFO] [client.go:573] [“find leader”] [Leader=“{"id":169601194,"store_id":113105576}”] [Key=7480000000000008ffb65f698000000000ff0000030168747470ff3a2f2f77ff77772eff646f757969ff6e2eff636f6d2f7669ff64ff656f2f37313534ffff3935333932363233ffff30343435333532ff00fe038000000fafffaef1b20000000000fa]
[2022/10/31 10:39:08.289 +08:00] [INFO] [conn.go:223] [“Reset backup client”] [storeID=113105576]
[2022/10/31 10:39:08.291 +08:00] [INFO] [client.go:603] [“start fine grained backup”] [incomplete=1]
[2022/10/31 10:39:08.291 +08:00] [INFO] [client.go:573] [“find leader”] [Leader=“{"id":169913863,"store_id":113105576}”] [Key=7480000000000008ffb65f698000000000ff0000040380000000ff0040d31703800000ff0f8d5b9ebb000000fc]
[2022/10/31 10:39:08.296 +08:00] [INFO] [conn.go:223] [“Reset backup client”] [storeID=113105576]
[2022/10/31 10:39:08.301 +08:00] [INFO] [client.go:828] [“try backup”] [StartKey=7480000000000008b65f69800000000000000301687474703a2f2f77ff77772e646f757969ff6e2e636f6d2f7669ff64656f2f37313534ff3935333932363233ff3034343533353200fe038000000fafaef1b2] [EndKey=7480000000000008b65f69800000000000000301687474703a2f2f77ff77772e646f757969ff6e2e636f6d2f7669ff64656f2f37313535ff3034323331373030ff3936383537363800fe038000000fb127fafa] [storeID=113105576] [“retry time”=0]
[2022/10/31 10:39:08.301 +08:00] [INFO] [client.go:828] [“try backup”] [StartKey=7480000000000008b65f69800000000000000403800000000040d317038000000f8d5b9ebb] [EndKey=7480000000000008b65f69800000000000000403800000000040d32f038000000f8f2f0eb3] [storeID=113105576] [“retry time”=0]
[2022/10/31 10:39:08.315 +08:00] [INFO] [client.go:603] [“start fine grained backup”] [incomplete=31]
[2022/10/31 10:39:08.316 +08:00] [INFO] [client.go:573] [“find leader”] [Leader=“{"id":171232986,"store_id":113105576}”] [Key=7480000000000008ffb65f728000000fdfff2d848e0000000000fa]
[2022/10/31 10:39:08.316 +08:00] [INFO] [client.go:573] [“find leader”] [Leader=“{"id":171242741,"store_id":113105576}”] [Key=7480000000000008ffb65f728000000fdfff5a425c0000000000fa]
[2022/10/31 10:39:08.316 +08:00] [INFO] [client.go:828] [“try backup”] [StartKey=7480000000000008b65f728000000fdf2d848e] [EndKey=7480000000000008b65f728000000fdf2f44f7] [storeID=113105576] [“retry time”=0]
[2022/10/31 10:39:08.316 +08:00] [INFO] [client.go:828] [“try backup”] [StartKey=7480000000000008b65f728000000fdf5a425c] [EndKey=7480000000000008b65f728000000fdf5afacb] [storeID=113105576] [“retry time”=0]
[2022/10/31 10:39:08.316 +08:00] [INFO] [client.go:573] [“find leader”] [Leader=“{"id":171329619,"store_id":113105576}”] [Key=7480000000000008ffb65f728000000fdfff5c9c5f0000000000fa]
[2022/10/31 10:39:08.316 +08:00] [INFO] [client.go:828] [“try backup”] [StartKey=7480000000000008b65f728000000fdf5c9c5f] [EndKey=7480000000000008b65f728000000fdf5da3f3] [storeID=113105576] [“retry time”=0]
[2022/10/31 10:39:08.316 +08:00] [INFO] [client.go:573] [“find leader”] [Leader=“{"id":171238569,"store_id":113105576}”] [Key=7480000000000008ffb65f728000000fdfff4b53ed0000000000fa]
[2022/10/31 10:39:08.316 +08:00] [INFO] [client.go:828] [“try backup”] [StartKey=7480000000000008b65f728000000fdf4b53ed] [EndKey=7480000000000008b65f728000000fdf4c6848] [storeID=113105576] [“retry time”=0]
[2022/10/31 10:39:08.644 +08:00] [WARN] [client.go:806] [“reset the connection in handleFineGrained”] [storeID=113105576]
[2022/10/31 10:39:08.644 +08:00] [INFO] [conn.go:223] [“Reset backup client”] [storeID=113105576]
[2022/10/31 10:39:11.220 +08:00] [WARN] [client.go:806] [“reset the connection in handleFineGrained”] [storeID=113105576]
[2022/10/31 10:39:11.220 +08:00] [INFO] [conn.go:223] [“Reset backup client”] [storeID=113105576]
[2022/10/31 10:39:11.227 +08:00] [INFO] [client.go:603] [“start fine grained backup”] [incomplete=3]
[2022/10/31 10:39:11.228 +08:00] [INFO] [client.go:573] [“find leader”] [Leader=“{"id":168265049,"store_id":113105576}”] [Key=7480000000000008ffb65f698000000000ff0000010380000000ff0004ab6b01687474ff703a2f2f77ff7777ff2e7869616f68ff6fff6e677368752e63ffff6f6d2f646973636fffff766572792f6974ff65ff6d2f36333332ff3961ff6131303030ff303030ff30303137ff30313763ff613700ff0000000000f90000fd]
[2022/10/31 10:39:11.228 +08:00] [INFO] [client.go:828] [“try backup”] [StartKey=7480000000000008b65f69800000000000000103800000000004ab6b01687474703a2f2f77ff77772e7869616f68ff6f6e677368752e63ff6f6d2f646973636fff766572792f697465ff6d2f363333323961ff6131303030303030ff3030313730313763ff6137000000000000f9] [EndKey=7480000000000008b65f69800000000000000103800000000004ab6c01687474703a2f2f70ff6f73742e6d2e736dff7a646d2e636f6d2fff702f616771363767ff7064000000000000f9] [storeID=113105576] [“retry time”=0]
[2022/10/31 10:39:11.228 +08:00] [INFO] [client.go:573] [“find leader”] [Leader=“{"id":170278481,"store_id":113105576}”] [Key=7480000000000008ffb65f698000000000ff0000010380000000ff0004ab6c01687474ff703a2f2f77ff7777ff2e736f68752eff63ff6f6d2f612f3538ffff393833383039395fffff3131353336323fff6dff785f636f6d6dff656eff745f69643dff663362ff61313734ff37316438ff623338ff6530386230ff3165ff366166663262ff61ff62656461346662ffff6661346262363038ffff65373633393361ff65ff326532333635ff3333ff3339323536ff000000fc00000000fb]
[2022/10/31 10:39:11.228 +08:00] [INFO] [client.go:828] [“try backup”] [StartKey=7480000000000008b65f69800000000000000103800000000004ab6c01687474703a2f2f77ff77772e736f68752eff636f6d2f612f3538ff393833383039395fff3131353336323f6dff785f636f6d6d656eff745f69643d663362ff6131373437316438ff6233386530386230ff3165366166663262ff6162656461346662ff6661346262363038ff6537363339336165ff3265323336353333ff3339323536000000fc] [EndKey=7480000000000008b65f69800000000000000103800000000004ab6c01687474703a2f2f77ff77772e7869616f68ff6f6e677368752e63ff6f6d2f646973636fff766572792f697465ff6d2f363333393136ff3630303030303030ff3030316430313364ff6165000000000000f9] [storeID=113105576] [“retry time”=0]
[2022/10/31 10:39:11.228 +08:00] [INFO] [client.go:573] [“find leader”] [Leader=“{"id":170280997,"store_id":113105576}”] [Key=7480000000000008ffb65f698000000000ff0000010380000000ff0004ab6601687474ff703a2f2f77ff7777ff2e646f757969ff6eff2e636f6d2f7669ffff64656f2f37313533ffff34313533353637ff33ff343838353135ff303fff6d785f636fff6d6d65ff6e745f69ff643d3162ff353861ff3762636536ff6462ff643166336163ff36ff31316330346632ffff3334326637393636ffff31353365613962ff38ff623131646432ff3666ff6532616364ff636263ff62646164ff37340000fd000000fc]
[2022/10/31 10:39:11.228 +08:00] [INFO] [client.go:828] [“try backup”] [StartKey=7480000000000008b65f69800000000000000103800000000004ab6601687474703a2f2f77ff77772e646f757969ff6e2e636f6d2f7669ff64656f2f37313533ff3431353335363733ff343838353135303fff6d785f636f6d6d65ff6e745f69643d3162ff3538613762636536ff6462643166336163ff3631316330346632ff3334326637393636ff3135336561396238ff6231316464323666ff6532616364636263ff6264616437340000fd] [EndKey=7480000000000008b65f69800000000000000103800000000004ab6601687474703a2f2f77ff77772e646f757969ff6e2e636f6d2f7669ff64656f2f37313534ff3330343435313235ff383337313333393fff6d785f636f6d6d65ff6e745f69643d3666ff3932346161623161ff3632623561363535ff3336323266346162ff3564643164646561ff3230393036396335ff3138393637393066ff3361316165326530ff3933343763320000fd] [storeID=113105576] [“retry time”=0]
[2022/10/31 10:39:11.644 +08:00] [WARN] [client.go:806] [“reset the connection in handleFineGrained”] [storeID=113105576]
[2022/10/31 10:39:11.644 +08:00] [WARN] [client.go:806] [“reset the connection in handleFineGrained”] [storeID=113105576]
[2022/10/31 10:39:11.644 +08:00] [INFO] [conn.go:223] [“Reset backup client”] [storeID=113105576]
[2022/10/31 10:39:11.644 +08:00] [WARN] [client.go:806] [“reset the connection in handleFineGrained”] [storeID=113105576]
[2022/10/31 10:39:11.644 +08:00] [WARN] [client.go:806] [“reset the connection in handleFineGrained”] [storeID=113105576]
[2022/10/31 10:39:11.644 +08:00] [WARN] [client.go:806] [“reset the connection in handleFineGrained”] [storeID=113105576]
[2022/10/31 10:39:11.644 +08:00] [WARN] [client.go:806] [“reset the connection in handleFineGrained”] [storeID=113105576]
[2022/10/31 10:39:11.646 +08:00] [INFO] [client.go:828] [“try backup”] [StartKey=7480000000000008b65f69800000000000000301687474703a2f2f77ff77772e646f757969ff6e2e636f6d2f7669ff64656f2f37313534ff3935333932363233ff3034343533353200fe038000000fafaef1b2] [EndKey=7480000000000008b65f69800000000000000301687474703a2f2f77ff77772e646f757969ff6e2e636f6d2f7669ff64656f2f37313535ff3034323331373030ff3936383537363800fe038000000fb127fafa] [storeID=113105576] [“retry time”=1]
[2022/10/31 10:39:11.646 +08:00] [INFO] [conn.go:223] [“Reset backup client”] [storeID=113105576]
[2022/10/31 10:39:11.647 +08:00] [INFO] [client.go:828] [“try backup”] [StartKey=7480000000000008b65f728000000fdf5c9c5f] [EndKey=7480000000000008b65f728000000fdf5da3f3] [storeID=113105576] [“retry time”=1]
[2022/10/31 10:39:11.647 +08:00] [INFO] [conn.go:223] [“Reset backup client”] [storeID=113105576]
[2022/10/31 10:39:11.648 +08:00] [INFO] [client.go:828] [“try backup”] [StartKey=7480000000000008b65f69800000000000000403800000000040d317038000000f8d5b9ebb] [EndKey=7480000000000008b65f69800000000000000403800000000040d32f038000000f8f2f0eb3] [storeID=113105576] [“retry time”=1]
[2022/10/31 10:39:11.648 +08:00] [INFO] [conn.go:223] [“Reset backup client”] [storeID=113105576]
[2022/10/31 10:39:11.650 +08:00] [INFO] [client.go:828] [“try backup”] [StartKey=7480000000000008b65f728000000fdf2d848e] [EndKey=7480000000000008b65f728000000fdf2f44f7] [storeID=113105576] [“retry time”=1]
[2022/10/31 10:39:11.650 +08:00] [INFO] [conn.go:223] [“Reset backup client”] [storeID=113105576]
[2022/10/31 10:39:11.651 +08:00] [INFO] [client.go:828] [“try backup”] [StartKey=7480000000000008b65f728000000fdf5a425c] [EndKey=7480000000000008b65f728000000fdf5afacb] [storeID=113105576] [“retry time”=1]
[2022/10/31 10:39:11.651 +08:00] [INFO] [conn.go:223] [“Reset backup client”] [storeID=113105576]
[2022/10/31 10:39:11.652 +08:00] [INFO] [client.go:828] [“try backup”] [StartKey=7480000000000008b65f728000000fdf4b53ed] [EndKey=7480000000000008b65f728000000fdf4c6848] [storeID=113105576] [“retry time”=1]
[2022/10/31 10:39:11.923 +08:00] [INFO] [coprocessor.go:1034] [“[TIME_COP_WAIT] resp_time:317.020322ms txnStartTS:437042269797220354 region_id:164891523 store_addr:192.168.241.61:20160 kv_wait_ms:317”]

h5n1 · 2022 年10 月 31 日 02:57

你的GC设置了多久，能一个月内按1号的全备备份？增量备份现在还是实验特性

老龚caiji · 2022 年10 月 31 日 03:02

我GC时间设置的1个月744h
update mysql.tidb set VARIABLE_VALUE=“744h” where VARIABLE_NAME=“tikv_gc_life_time”;

h5n1 · 2022 年10 月 31 日 03:03

最近是不是有大数据量变更

老龚caiji · 2022 年10 月 31 日 03:07

是有变更，但是每个月的数据量也就200G左右，怎么会增量出来3.5T呀？（3.5T数据备份进度才72%让我结束了），数据基数就200G不到，我是想不通怎么会这样？遇到BUG了？

老龚caiji · 2022 年11 月 1 日 02:15

有没有官方的老师来解答一下呀？很郁闷是BUG还是啥问题
BR是官方推荐的备份工具，应该不是非稳定版本软件吧？

forever · 2022 年11 月 1 日 03:12

比如200g的数据每条都有10个旧版本九四2t了

老龚caiji · 2022 年11 月 1 日 03:16

应该不会这么多的，即使是10个旧版本也才2T，实际得备份4T多

大鱼海棠 · 2022 年11 月 1 日 09:21

mvcc数据
gc这么久，查询抗的住吗

老龚caiji · 2022 年11 月 2 日 02:10

没感觉到慢，我们现在单表50亿+数据，没感觉啥异常
可能是我们数据场景就是增和查吧，对修改删除很少吧
关键是就那1天数据那样，第二天我再增量备份数据量就很少了，如果是MVCC的话，那第二天应该也很大才对呀

大鱼海棠 · 2022 年11 月 2 日 03:56

第二天增量备份的是 backup_ts, current_ts之间的数据

近墨者zyl · 2022 年11 月 2 日 04:33

还是用逻辑备份吧。。br的物理备份现在不好用

大鱼海棠 · 2022 年11 月 2 日 07:02

可以问题业务侧，有没有做什么重操作
2.可以用set tidb_snapshot=xxxx ，比对两天的数据量差异，有可能能发现一些线索，就是比较麻烦
3.比对一些慢查询，去看下有没有遇到的版本数明显增加的情况

老龚caiji · 2022 年11 月 2 日 07:47

我每天的增量都是基于月初的全量备份，包含前一天的数据的

大鱼海棠 · 2022 年11 月 2 日 08:48

等大佬来看下吧，不知道那里的问题了

近墨者zyl · 2022 年11 月 3 日 06:11

关注进展

老龚caiji · 2022 年11 月 7 日 09:14

应该是没有进展了

yilong · 2022 年11 月 8 日 03:09

有没有看过备份前几天的 tikv 容量变化？可以上传下当时 tikv-detail 监控信息和 BR 备份监控。

alfred · 2022 年11 月 8 日 14:49

全量备份有多大呢？是不是跑成全量了

Leavrth · 2022 年11 月 10 日 02:35

这个日志表明了可能存在重复备份的情况

[2022/10/31 10:54:16.010 +08:00] [INFO] [rtree.go:136] [“delete overlapping range”] [StartKey=7480000000000008b65f728000000ddc683a87] [EndKey=7480000000000008b65f728000000ddc6ab9c9]

可以打印一下有 7480000000000008b65f728000000ddc683a87 相关记录的日志嘛。

另外，这个日志有点奇怪，似乎这是 checksum 阶段才会出现的日志

[2022/10/31 10:39:11.923 +08:00] [INFO] [coprocessor.go:1034] [“[TIME_COP_WAIT] resp_time:317.020322ms txnStartTS:437042269797220354 region_id:164891523 store_addr:192.168.241.61:20160 kv_wait_ms:317”]

可以打印一下带有 backup time range 字段的日志嘛，确认一下 StartVersion 的值