Skip to content

feat(fusio-manifest): add run-v2 sparse index and block cache#264

Merged
ethe merged 2 commits intomainfrom
refactor/run-v2
Feb 7, 2026
Merged

feat(fusio-manifest): add run-v2 sparse index and block cache#264
ethe merged 2 commits intomainfrom
refactor/run-v2

Conversation

@ethe
Copy link
Member

@ethe ethe commented Feb 7, 2026

Why / Intent

fusio-manifest point-get on merge-tree runs could still degrade into reading large run payloads and scanning linearly.
This is especially costly for object storage (e.g. S3), where unnecessary range reads amplify latency and I/O.
This PR changes the read path to index-first + block-level reads:

  • filter and locate with run index metadata
  • read only the minimal candidate block
  • cache hot blocks to avoid repeated remote reads
    Backward compatibility is intentionally not preserved in this change.

Benefits

Measured on local-disk benchmark with cache disabled, bloom+sparse/block path significantly reduces wasted I/O in in-range misses:

  • L1-InRangeMiss
    • ckpt_payload_range: 980 -> 9
    • requested_bytes: 9,416,820 -> 86,481 (~-99.1%)
    • ops/s: 3204.47 -> 3702.02 (+15.5%)
    • p50: 312.17us -> 266.58us (-14.6%)
  • MultiLevel-InRangeMiss
    • ckpt_payload_range: 988 -> 6
    • requested_bytes: 9,746,620 -> 59,190 (~-99.4%)
    • ops/s: 2535.56 -> 2668.60 (+5.2%)
    • p50: 394.29us -> 373.12us (-5.4%)
      Net effect: much lower worst-path point-get cost through aggressive run skipping and minimal block reads.

Compatibility

This PR intentionally introduces a format break:

  • no backward compatibility layer for old run/checkpoint payloads
  • migration requires regenerating runs (re-compaction / rebuild)

Validation

  • cargo +nightly fmt --all -- --check
  • cargo clippy -p fusio-manifest -- -D warnings
  • cargo test -p fusio-manifest
  • cargo check -p fusio-manifest --benches --features cache-moka

@ethe ethe merged commit 22aa611 into main Feb 7, 2026
5 checks passed
@ethe ethe deleted the refactor/run-v2 branch February 7, 2026 13:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant