ingest sst will add the sst to the bottommost level if there is no overlap with existing ssts, level 6 by default. We use ingest sst to replicate regions’ peers, so the LSM structure may like this:
L0 ssts
L1 ssts
L2 ssts
L3~L5 is empty
L6 ssts(ingested by replicating)
When user delete these datas which contained in ssts at level 6, these delete marks are hard to reach level 6. So the disk space occupied by these ssts is hard to reclaim.
When dynamic_level_bytes is on, for an empty DB, RocksDB makes last level the base level, which means merging L0 data into the last level, until it exceeds max_bytes_for_level_base. And then RocksDB makes the second last level to be base level, and so on.
For the above case, if dynamic_level_bytes is on, the LSM structure is like this:
L0 ssts
L5 ssts
L6 ssts
The delete marks will be compacted from L0 to L5, from L5 to L6. The relevant space occupied by these keys will be freed.
It is not recommended to change this value on db which is not empty. If you really want to change this value, please use pd-ctl to compact the whole db first.
$./bin/tikv-ctl --host={$tikv-ip}:{$tikv-port} compact -c write
$./bin/tikv-ctl --host={$tikv-ip}:{$tikv-port} compact -c default