pd主节点中报大量的remove operator because region disappeared？

🪐 TiDB 技术问题 部署&运维管理
1

tidb version：

v5.3.0

pd主节点报大量的remove operator because region disappeared日志：

[2022/01/18 18:38:25.554 +08:00] [WARN] [operator_controller.go:224] ["remove operator because region disappeared"] [region-id=124272413] [operator="merge-region {merge: region 124272413 to 124272417} (kind:merge, region:124272413(42153,2279), createAt:2022-01-18 18:38:25.397470596 +0800 CST m=+525129.283309999, startAt:2022-01-18 18:38:25.397512263 +0800 CST m=+525129.283351662, currentStep:0, steps:[merge region 124272413 into region 124272417])"]
[2022/01/18 18:38:25.554 +08:00] [WARN] [operator_controller.go:224] ["remove operator because region disappeared"] [region-id=124272425] [operator="merge-region {merge: region 124272425 to 124272421} (kind:merge, region:124272425(42153,2279), createAt:2022-01-18 18:38:25.397910671 +0800 CST m=+525129.283750110, startAt:2022-01-18 18:38:25.397981999 +0800 CST m=+525129.283821437, currentStep:0, steps:[merge region 124272425 into region 124272421])"]
[2022/01/18 18:38:26.554 +08:00] [WARN] [operator_controller.go:224] ["remove operator because region disappeared"] [region-id=124272405] [operator="merge-region {merge: region 124272405 to 124272409} (kind:merge, region:124272405(46950,2428), createAt:2022-01-18 18:38:26.073978929 +0800 CST m=+525129.959818397, startAt:2022-01-18 18:38:26.07403712 +0800 CST m=+525129.959876550, currentStep:0, steps:[merge region 124272405 into region 124272409])"]
[2022/01/18 18:38:27.053 +08:00] [WARN] [operator_controller.go:224] ["remove operator because region disappeared"] [region-id=124272429] [operator="merge-region {merge: region 124272429 to 124272434} (kind:merge, region:124272429(3278,2664), createAt:2022-01-18 18:38:26.661664122 +0800 CST m=+525130.547503540, startAt:2022-01-18 18:38:26.661711899 +0800 CST m=+525130.547551334, currentStep:0, steps:[merge region 124272429 into region 124272434])"]
[2022/01/18 18:38:27.054 +08:00] [WARN] [operator_controller.go:224] ["remove operator because region disappeared"] [region-id=124272444] [operator="merge-region {merge: region 124272444 to 124272449} (kind:merge, region:124272444(3278,2664), createAt:2022-01-18 18:38:26.662224658 +0800 CST m=+525130.548064096, startAt:2022-01-18 18:38:26.6622476 +0800 CST m=+525130.548087003, currentStep:0, steps:[merge region 124272444 into region 124272449])"]
[2022/01/18 18:38:27.054 +08:00] [WARN] [operator_controller.go:224] ["remove operator because region disappeared"] [region-id=124272454] [operator="merge-region {merge: region 124272454 to 124209715} (kind:merge, region:124272454(3278,2664), createAt:2022-01-18 18:38:26.662393829 +0800 CST m=+525130.548233250, startAt:2022-01-18 18:38:26.662411764 +0800 CST m=+525130.548251169, currentStep:0, steps:[merge region 124272454 into region 124209715])"]
[2022/01/18 18:38:27.054 +08:00] [WARN] [operator_controller.go:224] ["remove operator because region disappeared"] [region-id=124272417] [operator="merge-region {merge: region 124272417 to 124272421} (kind:merge, region:124272417(42155,2279), createAt:2022-01-18 18:38:26.683971207 +0800 CST m=+525130.569810653, startAt:2022-01-18 18:38:26.684027161 +0800 CST m=+525130.569866611, currentStep:0, steps:[merge region 124272417 into region 124272421])"]
[2022/01/18 18:38:27.553 +08:00] [WARN] [operator_controller.go:224] ["remove operator because region disappeared"] [region-id=124272409] [operator="merge-region {merge: region 124272409 to 124272397} (kind:merge, region:124272409(46952,2428), createAt:2022-01-18 18:38:27.400681559 +0800 CST m=+525131.286520987, startAt:2022-01-18 18:38:27.400746111 +0800 CST m=+525131.286585560, currentStep:0, steps:[merge region 124272409 into region 124272397])"]
[2022/01/18 18:38:27.553 +08:00] [WARN] [operator_controller.go:224] ["remove operator because region disappeared"] [region-id=124272484] [operator="merge-region {merge: region 124272484 to 124272488} (kind:merge, region:124272484(47458,2459), createAt:2022-01-18 18:38:27.4233923 +0800 CST m=+525131.309231704, startAt:2022-01-18 18:38:27.423426617 +0800 CST m=+525131.309266020, currentStep:0, steps:[merge region 124272484 into region 124272488])"]
[2022/01/18 18:38:28.053 +08:00] [WARN] [operator_controller.go:224] ["remove operator because region disappeared"] [region-id=124272459] [operator="merge-region {merge: region 124272459 to 124272464} (kind:merge, region:124272459(52299,2156), createAt:2022-01-18 18:38:27.698440873 +0800 CST m=+525131.584280273, startAt:2022-01-18 18:38:27.698487875 +0800 CST m=+525131.584327275, currentStep:0, steps:[merge region 124272459 into region 124272464])"]
[2022/01/18 18:38:28.054 +08:00] [WARN] [operator_controller.go:224] ["remove operator because region disappeared"] [region-id=124272469] [operator="merge-region {merge: region 124272469 to 124272474} (kind:merge, region:124272469(52299,2156), createAt:2022-01-18 18:38:27.698824929 +0800 CST m=+525131.584664392, startAt:2022-01-18 18:38:27.698858337 +0800 CST m=+525131.584697762, currentStep:0, steps:[merge region 124272469 into region 124272474])"]
[2022/01/18 18:38:28.054 +08:00] [WARN] [operator_controller.go:224] ["remove operator because region disappeared"] [region-id=124272479] [operator="merge-region {merge: region 124272479 to 119915583} (kind:merge, region:124272479(52299,2156), createAt:2022-01-18 18:38:27.699180703 +0800 CST m=+525131.585020183, startAt:2022-01-18 18:38:27.699209517 +0800 CST m=+525131.585048940, currentStep:0, steps:[merge region 124272479 into region 119915583])"]
[2022/01/18 18:38:28.054 +08:00] [WARN] [operator_controller.go:224] ["remove operator because region disappeared"] [region-id=124272496] [operator="merge-region {merge: region 124272496 to 119771024} (kind:merge, region:124272496(50186,2004), createAt:2022-01-18 18:38:27.858211649 +0800 CST m=+525131.744051050, startAt:2022-01-18 18:38:27.858251607 +0800 CST m=+525131.744091006, currentStep:0, steps:[merge region 124272496 into region 119771024])"]
[2022/01/18 18:38:28.054 +08:00] [WARN] [operator_controller.go:224] ["remove operator because region disappeared"] [region-id=124272434] [operator="merge-region {merge: region 124272434 to 124272439} (kind:merge, region:124272434(3280,2664), createAt:2022-01-18 18:38:27.973181226 +0800 CST m=+525131.859020639, startAt:2022-01-18 18:38:27.973274547 +0800 CST m=+525131.859113960, currentStep:0, steps:[merge region 124272434 into region 124272439])"]
[2022/01/18 18:38:28.054 +08:00] [WARN] [operator_controller.go:224] ["remove operator because region disappeared"] [region-id=124272449] [operator="merge-region {merge: region 124272449 to 124209715} (kind:merge, region:124272449(3280,2664), createAt:2022-01-18 18:38:27.973546944 +0800 CST m=+525131.859386384, startAt:2022-01-18 18:38:27.973593446 +0800 CST m=+525131.859432875, currentStep:0, steps:[merge region 124272449 into region 124209715])"]
[2022/01/18 18:38:28.054 +08:00] [WARN] [operator_controller.go:224] ["remove operator because region disappeared"] [region-id=124272421] [operator="merge-region {merge: region 124272421 to 122640786} (kind:merge, region:124272421(42157,2279), createAt:2022-01-18 18:38:27.99956449 +0800 CST m=+525131.885403916, startAt:2022-01-18 18:38:27.999604741 +0800 CST m=+525131.885444150, currentStep:0, steps:[merge region 124272421 into region 122640786])"]
[2022/01/18 18:38:28.554 +08:00] [WARN] [operator_controller.go:224] ["remove operator because region disappeared"] [region-id=124272501] [operator="merge-region {merge: region 124272501 to 119828263} (kind:merge, region:124272501(7204,7467), createAt:2022-01-18 18:38:28.195949526 +0800 CST m=+525132.081788975, startAt:2022-01-18 18:38:28.196002733 +0800 CST m=+525132.081842130, currentStep:0, steps:[merge region 124272501 into region 119828263])"]
[2022/01/18 18:38:29.053 +08:00] [WARN] [operator_controller.go:224] ["remove operator because region disappeared"] [region-id=124272488] [operator="merge-region {merge: region 124272488 to 124272492} (kind:merge, region:124272488(47460,2459), createAt:2022-01-18 18:38:28.618988514 +0800 CST m=+525132.504827924, startAt:2022-01-18 18:38:28.619033518 +0800 CST m=+525132.504872921, currentStep:0, steps:[merge region 124272488 into region 124272492])"]
[2022/01/18 18:38:29.053 +08:00] [WARN] [operator_controller.go:224] ["remove operator because region disappeared"] [region-id=124272464] [operator="merge-region {merge: region 124272464 to 124272474} (kind:merge, region:124272464(52301,2156), createAt:2022-01-18 18:38:28.895264394 +0800 CST m=+525132.781103794, startAt:2022-01-18 18:38:28.895304561 +0800 CST m=+525132.781143964, currentStep:0, steps:[merge region 124272464 into region 124272474])"]
[2022/01/18 18:38:29.553 +08:00] [WARN] [operator_controller.go:224] ["remove operator because region disappeared"] [region-id=124272439] [operator="merge-region {merge: region 124272439 to 124209715} (kind:merge, region:124272439(3282,2664), createAt:2022-01-18 18:38:29.194415786 +0800 CST m=+525133.080255194, startAt:2022-01-18 18:38:29.194470628 +0800 CST m=+525133.080310030, currentStep:0, steps:[merge region 124272439 into region 124209715])"]
[2022/01/18 18:38:29.553 +08:00] [WARN] [operator_controller.go:224] ["remove operator because region disappeared"] [region-id=124272505] [operator="merge-region {merge: region 124272505 to 119863615} (kind:merge, region:124272505(7036,7469), createAt:2022-01-18 18:38:29.420118449 +0800 CST m=+525133.305957850, startAt:2022-01-18 18:38:29.420176789 +0800 CST m=+525133.306016191, currentStep:0, steps:[merge region 124272505 into region 119863615])"]
[2022/01/18 18:38:30.053 +08:00] [WARN] [operator_controller.go:224] ["remove operator because region disappeared"] [region-id=124272492] [operator="merge-region {merge: region 124272492 to 124181942} (kind:merge, region:124272492(47462,2459), createAt:2022-01-18 18:38:29.882673413 +0800 CST m=+525133.768512826, startAt:2022-01-18 18:38:29.882728928 +0800 CST m=+525133.768568345, currentStep:0, steps:[merge region 124272492 into region 124181942])"]
[2022/01/18 18:38:30.553 +08:00] [WARN] [operator_controller.go:224] ["remove operator because region disappeared"] [region-id=124272474] [operator="merge-region {merge: region 124272474 to 119915583} (kind:merge, region:124272474(52303,2156), createAt:2022-01-18 18:38:30.177577401 +0800 CST m=+525134.063416837, startAt:2022-01-18 18:38:30.177640415 +0800 CST m=+525134.063479812, currentStep:0, steps:[merge region 124272474 into region 119915583])"]
4 个赞
2

看来是region在做合并，旧的region没有了导致它的operator被删除，正常现象吧

1 个赞
3

我看了从2022/01/15报到现在，这种正常么？

1 个赞
4

你看一下15号到现在的region数量和空region数量的监控

1 个赞
5

image
image1447×347 44 KB

empty region看起来没啥变化

2 个赞
6

看着都是正常波动，这段时间内是有删除的操作吗

1 个赞
7

有一些删除操作，但都不大

8

关键是这个日志，每时每刻都在刷，从2022/01/15到现在，应该是tidb从4.0.1升级到5.3.0之后，才一直报这个日志

1 个赞
9

那估计新版本日志输出有问题，我找大佬帮你看一下

10

能把 pd 的监控导出一份来吗，想看看

11

什么时间段的？

12

pd-monitor.pdf (11.3 MB)

这个是最近三小时的pd监控

13

问一下 pd 监控下 的 balance 面板下的 store region count 数量在减少吗？（我看导出的监控少了这块内容）

14

image
image1819×894 169 KB

这个是12小时的数据，感觉没变

15

看 operator 大部分都是取消了，建议你拿 上面 pd 日志中的一个 region id，过滤一下 pd leader 的日志（找到对应的 store id，然后 去 对应的 store id 上 ，再根据 这个 region id或 peer id 过滤一下 对应时间的 日志）

16

主要问题应该是 在 tikv 侧，看看 这个 region 为啥 disappear 了（不过建议你也把 对应的 pd 日志和 tikv 日志也发出来一下，我也看一下）

17

我看了几个，这些region都不存在了

» region 124803388
null

» region 124803413
null

» region 124803405
null
18

嗯，我知道它们没了，我是想看看 他们怎么没的（PD 日志和 tikv 日志有记录的）

19

我看了四台tikv上面的日志，都是下面的日志：

[2022/01/19 11:55:48.293 +08:00] [WARN] [scanner.rs:88] ["resolved_ts scan get snapshot failed"] [err="Other(\"scan task cancelled\")"]
[2022/01/19 11:55:49.203 +08:00] [WARN] [scanner.rs:88] ["resolved_ts scan get snapshot failed"] [err="Other(\"scan task cancelled\")"]
[2022/01/19 11:55:52.614 +08:00] [WARN] [scanner.rs:88] ["resolved_ts scan get snapshot failed"] [err="Other(\"scan task cancelled\")"]
[2022/01/19 11:55:52.614 +08:00] [WARN] [scanner.rs:88] ["resolved_ts scan get snapshot failed"] [err="Other(\"scan task cancelled\")"]
[2022/01/19 11:55:52.614 +08:00] [WARN] [scanner.rs:88] ["resolved_ts scan get snapshot failed"] [err="Other(\"scan task cancelled\")"]
[2022/01/19 11:55:53.845 +08:00] [WARN] [scanner.rs:88] ["resolved_ts scan get snapshot failed"] [err="Other(\"scan task cancelled\")"]
[2022/01/19 11:55:53.846 +08:00] [WARN] [scanner.rs:88] ["resolved_ts scan get snapshot failed"] [err="Other(\"scan task cancelled\")"]
[2022/01/19 11:55:55.040 +08:00] [WARN] [scanner.rs:88] ["resolved_ts scan get snapshot failed"] [err="Other(\"scan task cancelled\")"]
[2022/01/19 11:55:56.262 +08:00] [WARN] [scanner.rs:88] ["resolved_ts scan get snapshot failed"] [err="Other(\"scan task cancelled\")"]
[2022/01/19 11:56:00.151 +08:00] [WARN] [scanner.rs:88] ["resolved_ts scan get snapshot failed"] [err="Other(\"scan task cancelled\")"]
[2022/01/19 11:56:12.695 +08:00] [WARN] [scanner.rs:88] ["resolved_ts scan get snapshot failed"] [err="Other(\"scan task cancelled\")"]
20

我找下之前的日志