TIKV实例内存不足问题

🪐 TiDB 技术问题
1

【TiDB 使用环境】生产环境
【TiDB 版本】6.1.5
【操作系统】CentOS79
【部署方式】机器部署，混合部署，每台服务器有4个TIKV实例节点，服务器内存有256G
【问题复现路径】一tikv实例发生OOM事件，通过后台日志分析发现，OS日志内一直提示无法分配内存，并且所有有TIKV实例的服务器系统内均有此类告警信息
【遇到的问题：问题现象及影响】
OS日志信息如下：
Feb 5 08:51:11 tidb-server01 kernel: rocksdb:low0: page allocation failure: order:1, mode:0x204020
Feb 5 08:51:11 tidb-server01 kernel: CPU: 56 PID: 1912 Comm: rocksdb:low0 Kdump: loaded Tainted: G W ------------ 3.10.0-957.el7.x86_64 #1
Feb 5 08:51:11 tidb-server01 kernel: Hardware name: Lenovo ThinkServer SR660 V2/SR660 V2, BIOS XWE106E-1.73 03/22/2023
Feb 5 08:51:11 tidb-server01 kernel: Call Trace:
Feb 5 08:51:11 tidb-server01 kernel: [] dump_stack+0x19/0x1b
Feb 5 08:51:11 tidb-server01 kernel: [] warn_alloc_failed+0x110/0x180
Feb 5 08:51:11 tidb-server01 kernel: [] ? zone_watermark_ok_safe+0x58/0xd0
Feb 5 08:51:11 tidb-server01 kernel: [] __alloc_pages_slowpath+0x6b6/0x724
Feb 5 08:51:11 tidb-server01 kernel: [] __alloc_pages_nodemask+0x405/0x420
Feb 5 08:51:11 tidb-server01 kernel: [] alloc_pages_current+0x98/0x110
Feb 5 08:51:11 tidb-server01 kernel: [] new_slab+0x383/0x390
Feb 5 08:51:11 tidb-server01 kernel: [] ___slab_alloc+0x3ac/0x4f0
Feb 5 08:51:11 tidb-server01 kernel: [] ? nvme_queue_rq+0xb4/0x8d0 [nvme]
Feb 5 08:51:11 tidb-server01 kernel: [] ? nvme_queue_rq+0xb4/0x8d0 [nvme]
Feb 5 08:51:11 tidb-server01 kernel: [] __slab_alloc+0x40/0x5c
Feb 5 08:51:11 tidb-server01 kernel: [] __kmalloc+0x1c0/0x230
Feb 5 08:51:11 tidb-server01 kernel: [] nvme_queue_rq+0xb4/0x8d0 [nvme]
Feb 5 08:51:11 tidb-server01 kernel: [] ? sbitmap_get+0x5d/0xb0
Feb 5 08:51:11 tidb-server01 kernel: [] ? __sbitmap_queue_get+0x2b/0xb0
Feb 5 08:51:11 tidb-server01 kernel: [] ? __blk_mq_get_tag+0x21/0x90
Feb 5 08:51:11 tidb-server01 kernel: [] ? blk_mq_get_tag+0xc3/0x2c0
Feb 5 08:51:11 tidb-server01 kernel: [] ? wake_up_atomic_t+0x30/0x30
Feb 5 08:51:11 tidb-server01 kernel: [] __blk_mq_try_issue_directly+0x135/0x1a0
Feb 5 08:51:11 tidb-server01 kernel: [] blk_mq_try_issue_directly+0x2d/0xb0
Feb 5 08:51:11 tidb-server01 kernel: [] blk_mq_make_request+0x385/0x630
Feb 5 08:51:11 tidb-server01 kernel: [] generic_make_request+0x147/0x380
Feb 5 08:51:11 tidb-server01 kernel: [] ? bvec_alloc+0x92/0x120
Feb 5 08:51:11 tidb-server01 kernel: [] submit_bio+0x70/0x150
Feb 5 08:51:11 tidb-server01 kernel: [] ? bio_alloc_bioset+0x213/0x310
Feb 5 08:51:11 tidb-server01 kernel: [] xfs_add_to_ioend+0x145/0x1d0 [xfs]
Feb 5 08:51:11 tidb-server01 kernel: [] xfs_do_writepage+0x1d7/0x510 [xfs]
Feb 5 08:51:11 tidb-server01 kernel: [] write_cache_pages+0x254/0x4e0
Feb 5 08:51:11 tidb-server01 kernel: [] ? xfs_aops_discard_page+0x150/0x150 [xfs]
Feb 5 08:51:11 tidb-server01 kernel: [] xfs_vm_writepages+0x6b/0xa0 [xfs]
Feb 5 08:51:11 tidb-server01 kernel: [] do_writepages+0x21/0x50
Feb 5 08:51:11 tidb-server01 kernel: [] __filemap_fdatawrite_range+0x65/0x80
Feb 5 08:51:11 tidb-server01 kernel: [] filemap_fdatawrite_range+0x13/0x20
Feb 5 08:51:11 tidb-server01 kernel: [] SyS_sync_file_range+0x172/0x190
Feb 5 08:51:11 tidb-server01 kernel: [] system_call_fastpath+0x22/0x27
Feb 5 08:51:11 tidb-server01 kernel: Mem-Info:
Feb 5 08:51:11 tidb-server01 kernel: active_anon:35643457 inactive_anon:315635 isolated_anon:0#012 active_file:5997484 inactive_file:18631799 isolated_fil
e:0#012 unevictable:0 dirty:3297 writeback:376 unstable:0#012 slab_reclaimable:1454461 slab_unreclaimable:97384#012 mapped:96825 shmem:1041065 pagetables:1
16052 bounce:0#012 free:2973991 free_pcp:8260 free_cma:0
Feb 5 08:51:11 tidb-server01 kernel: Node 0 DMA free:15904kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15904kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaim
able:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_sca
nned:0 all_unreclaimable? yes
Feb 5 08:51:11 tidb-server01 kernel: lowmem_reserve: 0 1373 128348 128348
Feb 5 08:51:11 tidb-server01 kernel: Node 0 DMA32 free:507488kB min:344kB low:428kB high:516kB active_anon:631476kB inactive_anon:2880kB active_file:11517
6kB inactive_file:116152kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1655124kB managed:1406736kB mlocked:0kB dirty:0kB writeback:0kB ma
pped:5240kB shmem:7056kB slab_reclaimable:8824kB slab_unreclaimable:2152kB kernel_stack:560kB pagetables:1512kB unstable:0kB bounce:0kB free_pcp:6332kB loc
al_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Feb 5 08:51:11 tidb-server01 kernel: lowmem_reserve: 0 0 126974 126974
Feb 5 08:51:11 tidb-server01 kernel: Node 0 Normal free:11283120kB min:32032kB low:40040kB high:48048kB active_anon:20385244kB inactive_anon:143676kB acti
ve_file:22504436kB inactive_file:73012396kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:132120576kB managed:130022124kB mlocked:0kB dirty
:48kB writeback:0kB mapped:292496kB shmem:444920kB slab_reclaimable:1095276kB slab_unreclaimable:125544kB kernel_stack:13984kB pagetables:51444kB unstable:
0kB bounce:0kB free_pcp:9756kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Feb 5 08:51:11 tidb-server01 kernel: lowmem_reserve: 0 0 0 0
Feb 5 08:51:11 tidb-server01 kernel: Node 1 Normal free:89452kB min:32544kB low:40680kB high:48816kB active_anon:121557108kB inactive_anon:1115984kB activ
e_file:1370324kB inactive_file:1398648kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:134217728kB managed:132099144kB mlocked:0kB dirty:13
140kB writeback:1504kB mapped:89564kB shmem:3712284kB slab_reclaimable:4713744kB slab_unreclaimable:261840kB kernel_stack:22384kB pagetables:411252kB unsta
ble:0kB bounce:0kB free_pcp:16952kB local_pcp:292kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Feb 5 08:51:11 tidb-server01 kernel: lowmem_reserve: 0 0 0 0
Feb 5 08:51:11 tidb-server01 kernel: Node 0 DMA: 04kB 08kB 016kB 132kB (U) 264kB (U) 1128kB (U) 1256kB (U) 0512kB 11024kB (U) 12048kB (M) 34096
kB (M) = 15904kB
Feb 5 08:51:11 tidb-server01 kernel: Node 0 DMA32: 1144kB (UEM) 7138kB (UEM) 38316kB (UEM) 18532kB (UEM) 7164kB (UEM) 27128kB (UEM) 10256kB (UEM) 1
1512kB (UM) 101024kB (UM) 62048kB (UE) 1104096kB (UM) = 507488kB
Feb 5 08:51:11 tidb-server01 kernel: Node 0 Normal: 1524164kB (UEM) 2940268kB (UEM) 12674116kB (UEM) 7863332kB (UEM) 2815564kB (UEM) 6947128kB (UEM)
2816256kB (UEM) 571512kB (UEM) 671024kB (UEM) 22048kB (EM) 04096kB = 11283072kB
Feb 5 08:51:11 tidb-server01 kernel: Node 1 Normal: 222514kB (UEM) 268kB (UM) 416kB (M) 032kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 04096k
B = 89276kB
Feb 5 08:51:11 tidb-server01 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Feb 5 08:51:11 tidb-server01 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Feb 5 08:51:11 tidb-server01 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Feb 5 08:51:11 tidb-server01 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Feb 5 08:51:11 tidb-server01 kernel: 25670606 total pagecache pages
Feb 5 08:51:11 tidb-server01 kernel: 0 pages in swap cache
Feb 5 08:51:11 tidb-server01 kernel: Swap cache stats: add 0, delete 0, find 0/0
Feb 5 08:51:11 tidb-server01 kernel: Free swap = 0kB
Feb 5 08:51:11 tidb-server01 kernel: Total swap = 0kB
Feb 5 08:51:11 tidb-server01 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x20)
Feb 5 08:51:11 tidb-server01 kernel: cache: kmalloc-8192, object size: 8192, buffer size: 8192, default order: 3, min order: 1
Feb 5 08:51:11 tidb-server01 kernel: node 0: slabs: 282, objs: 1128, free: 23
Feb 5 08:51:11 tidb-server01 kernel: node 1: slabs: 218, objs: 677, free: 0
Feb 5 08:51:11 tidb-server01 kernel: rocksdb:low0: page allocation failure: order:1, mode:0x204020
Feb 5 08:51:11 tidb-server01 kernel: CPU: 56 PID: 1912 Comm: rocksdb:low0 Kdump: loaded Tainted: G W ------------ 3.10.0-957.el7.x86_64 #1
Feb 5 08:51:11 tidb-server01 kernel: Hardware name: Lenovo ThinkServer SR660 V2/SR660 V2, BIOS XWE106E-1.73 03/22/2023
Feb 5 08:51:11 tidb-server01 kernel: Call Trace:
Feb 5 08:51:11 tidb-server01 kernel: [] dump_stack+0x19/0x1b
Feb 5 08:51:11 tidb-server01 kernel: [] warn_alloc_failed+0x110/0x180
Feb 5 08:51:11 tidb-server01 kernel: [] ? zone_watermark_ok_safe+0x58/0xd0
Feb 5 08:51:11 tidb-server01 kernel: [] __alloc_pages_slowpath+0x6b6/0x724
Feb 5 08:51:11 tidb-server01 kernel: [] __alloc_pages_nodemask+0x405/0x420
Feb 5 08:51:11 tidb-server01 kernel: [] alloc_pages_current+0x98/0x110
Feb 5 08:51:11 tidb-server01 kernel: [] new_slab+0x383/0x390
Feb 5 08:51:11 tidb-server01 kernel: [] ___slab_alloc+0x3ac/0x4f0
Feb 5 08:51:11 tidb-server01 kernel: [] ? nvme_queue_rq+0xb4/0x8d0 [nvme]
Feb 5 08:51:11 tidb-server01 kernel: [] ? nvme_queue_rq+0xb4/0x8d0 [nvme]
Feb 5 08:51:11 tidb-server01 kernel: [] __slab_alloc+0x40/0x5c
Feb 5 08:51:11 tidb-server01 kernel: [] __kmalloc+0x1c0/0x230
Feb 5 08:51:11 tidb-server01 kernel: [] nvme_queue_rq+0xb4/0x8d0 [nvme]
Feb 5 08:51:11 tidb-server01 kernel: [] ? sbitmap_get+0x5d/0xb0
Feb 5 08:51:11 tidb-server01 kernel: [] ? __sbitmap_queue_get+0x2b/0xb0
Feb 5 08:51:11 tidb-server01 kernel: [] ? __blk_mq_get_tag+0x21/0x90
Feb 5 08:51:11 tidb-server01 kernel: [] ? blk_mq_get_tag+0xc3/0x2c0
Feb 5 08:51:11 tidb-server01 kernel: [] ? wake_up_atomic_t+0x30/0x30
Feb 5 08:51:11 tidb-server01 kernel: [] __blk_mq_try_issue_directly+0x135/0x1a0
Feb 5 08:51:11 tidb-server01 kernel: [] blk_mq_try_issue_directly+0x2d/0xb0
Feb 5 08:51:11 tidb-server01 kernel: [] blk_mq_make_request+0x385/0x630
Feb 5 08:51:11 tidb-server01 kernel: [] generic_make_request+0x147/0x380
Feb 5 08:51:11 tidb-server01 kernel: [] ? bvec_alloc+0x92/0x120
Feb 5 08:51:11 tidb-server01 kernel: [] submit_bio+0x70/0x150
Feb 5 08:51:11 tidb-server01 kernel: [] ? bio_alloc_bioset+0x213/0x310
Feb 5 08:51:11 tidb-server01 kernel: [] xfs_add_to_ioend+0x145/0x1d0 [xfs]
Feb 5 08:51:11 tidb-server01 kernel: [] xfs_do_writepage+0x1d7/0x510 [xfs]
Feb 5 08:51:11 tidb-server01 kernel: [] write_cache_pages+0x254/0x4e0
Feb 5 08:51:11 tidb-server01 kernel: [] ? xfs_aops_discard_page+0x150/0x150 [xfs]
Feb 5 08:51:11 tidb-server01 kernel: [] xfs_vm_writepages+0x6b/0xa0 [xfs]
Feb 5 08:51:11 tidb-server01 kernel: [] do_writepages+0x21/0x50
Feb 5 08:51:11 tidb-server01 kernel: [] __filemap_fdatawrite_range+0x65/0x80
Feb 5 08:51:11 tidb-server01 kernel: [] filemap_fdatawrite_range+0x13/0x20
Feb 5 08:51:11 tidb-server01 kernel: [] SyS_sync_file_range+0x172/0x190
Feb 5 08:51:11 tidb-server01 kernel: [] system_call_fastpath+0x22/0x27
Feb 5 08:51:11 tidb-server01 kernel: Mem-Info:
Feb 5 08:51:11 tidb-server01 kernel: active_anon:35645505 inactive_anon:315635 isolated_anon:0#012 active_file:5997772 inactive_file:18630929 isolated_fil
e:0#012 unevictable:0 dirty:3344 writeback:376 unstable:0#012 slab_reclaimable:1454461 slab_unreclaimable:97384#012 mapped:96825 shmem:1041065 pagetables:1
16052 bounce:0#012 free:2973218 free_pcp:8003 free_cma:0
Feb 5 08:51:11 tidb-server01 kernel: Node 0 DMA free:15904kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15904kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaim
able:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_sca
nned:0 all_unreclaimable? yes
Feb 5 08:51:11 tidb-server01 kernel: lowmem_reserve: 0 1373 128348 128348
Feb 5 08:51:11 tidb-server01 kernel: Node 0 DMA32 free:507488kB min:344kB low:428kB high:516kB active_anon:631476kB inactive_anon:2880kB active_file:11517
6kB inactive_file:116152kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1655124kB managed:1406736kB mlocked:0kB dirty:0kB writeback:0kB ma
pped:5240kB shmem:7056kB slab_reclaimable:8824kB slab_unreclaimable:2152kB kernel_stack:560kB pagetables:1512kB unstable:0kB bounce:0kB free_pcp:6332kB loc
al_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Feb 5 08:51:11 tidb-server01 kernel: lowmem_reserve: 0 0 126974 126974
Feb 5 08:51:11 tidb-server01 kernel: Node 0 Normal free:11283120kB min:32032kB low:40040kB high:48048kB active_anon:20385244kB inactive_anon:143676kB acti
ve_file:22504436kB inactive_file:73012396kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:132120576kB managed:130022124kB mlocked:0kB dirty
:48kB writeback:0kB mapped:292496kB shmem:444920kB slab_reclaimable:1095276kB slab_unreclaimable:125544kB kernel_stack:13984kB pagetables:51444kB unstable:
0kB bounce:0kB free_pcp:9740kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Feb 5 08:51:11 tidb-server01 kernel: lowmem_reserve: 0 0 0 0
Feb 5 08:51:11 tidb-server01 kernel: Node 1 Normal free:86360kB min:32544kB low:40680kB high:48816kB active_anon:121565300kB inactive_anon:1115984kB activ
e_file:1371476kB inactive_file:1395168kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:134217728kB managed:132099144kB mlocked:0kB dirty:13
328kB writeback:1504kB mapped:89564kB shmem:3712284kB slab_reclaimable:4713744kB slab_unreclaimable:261840kB kernel_stack:22384kB pagetables:411252kB unsta
ble:0kB bounce:0kB free_pcp:15936kB local_pcp:232kB free_cma:0kB writeback_tmp:0kB pages_scanned:85 all_unreclaimable? no
Feb 5 08:51:11 tidb-server01 kernel: lowmem_reserve: 0 0 0 0
Feb 5 08:51:11 tidb-server01 kernel: Node 0 DMA: 04kB 08kB 016kB 132kB (U) 264kB (U) 1128kB (U) 1256kB (U) 0512kB 11024kB (U) 12048kB (M) 34096
kB (M) = 15904kB
Feb 5 08:51:11 tidb-server01 kernel: Node 0 DMA32: 1144kB (UEM) 7138kB (UEM) 38316kB (UEM) 18532kB (UEM) 7164kB (UEM) 27128kB (UEM) 10256kB (UEM) 1
1512kB (UM) 101024kB (UM) 62048kB (UE) 1104096kB (UM) = 507488kB
Feb 5 08:51:11 tidb-server01 kernel: Node 0 Normal: 1524164kB (UEM) 2940268kB (UEM) 12674016kB (UEM) 7863432kB (UEM) 2815564kB (UEM) 6947128kB (UEM)
2816256kB (UEM) 571512kB (UEM) 671024kB (UEM) 22048kB (EM) 04096kB = 11283088kB
Feb 5 08:51:11 tidb-server01 kernel: Node 1 Normal: 212964kB (UEM) 548kB (UM) 416kB (M) 032kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 04096k
B = 85680kB
Feb 5 08:51:11 tidb-server01 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Feb 5 08:51:11 tidb-server01 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Feb 5 08:51:11 tidb-server01 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Feb 5 08:51:11 tidb-server01 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Feb 5 08:51:11 tidb-server01 kernel: 25669995 total pagecache pages
Feb 5 08:51:11 tidb-server01 kernel: 0 pages in swap cache
Feb 5 08:51:11 tidb-server01 kernel: Swap cache stats: add 0, delete 0, find 0/0
Feb 5 08:51:11 tidb-server01 kernel: Free swap = 0kB
Feb 5 08:51:11 tidb-server01 kernel: Total swap = 0kB
Feb 5 08:51:11 tidb-server01 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x20)
Feb 5 08:51:11 tidb-server01 kernel: cache: kmalloc-8192, object size: 8192, buffer size: 8192, default order: 3, min order: 1
Feb 5 08:51:11 tidb-server01 kernel: node 0: slabs: 282, objs: 1128, free: 23
Feb 5 08:51:11 tidb-server01 kernel: node 1: slabs: 218, objs: 677, free: 0

目前TIKV实例的内存参数配置如下：
storage.block-cache.capacity: 16GB

以上问题我是不是可以通过在线调整storage.block-cache.capacity参数的大小来解决此隐患？

各位大神帮忙指导指导，非常感谢！

2

绑numa了吗？

3

16G*4 也才64G，没有超过256G吧，感觉只设置这个达不到效果。看看具体是哪个sql导致的？限制每个sql可用内存试试。

4

试试这个配置项呢？

image
image816×386 48.9 KB

5

绑了的

6

这个是部署的时候配置的，前期数据量没那么大！

7

由于混合部署，都在全局参数进行的设定，没有在每个TIKV实例上设置！

8

问题是服务器上还有PD和TIDB，我们最早部署的时候，数据量也没那么大，所以把TIKV内存的配置限制在16G

9

目前生产满产，也不可能在生产环境进行测试，测试环境也无法去验证，我的想法是在线调整参数storage.block-cache.capacity，由目前的16G调整未20G，或者25G，但不知道效果如何，各位大神有遇见过相同的问题么？

10

:thinking: 混部的话，还有一个可能是tidb节点内存OOM，可以看一下tidb server的日志。

12

image
image1708×653 37.7 KB

从内存监控看，当时的内存是由波动，但总体波动不大，总体利用率也才60%左右的，所以排除TIDB的大数据操作问题！

13

这是 Linux 系统内存碎片化 导致的 问题

14

那这种问题可有啥解决方法吗？

15

增加保留内存，调整为1GB，防止碎片化

echo 1048576 > /proc/sys/vm/min_free_kbytes

16

storage.block-cache.capacity 还可以调小一点比如12g或者8g

17

先按推荐值配置，观察 24-48 小时，若仍有内存告警，再微调block_cache_size、memory_usage_limit等参数，避免一次性大幅调整导致性能下降。

18

目前问题已经定位到，是由于在TIDB内的numa_node配置不均导致内存使用不均，目前生产满负荷生产，没法调整并轮询重启TIKV实例，可否将进行numactl --interleave=all设置，此种设置对于目前TIDB是否有效，对性能影响是否很大，有大神有经验的不！

19

应该可以吧, 这是解决 NUMA 内存分配不均导致 Swap 或 OOM 的标准手段

20

可以看下是怎么绑的？ 内存显示大页开了吗

21

内存你负载不匹配啊