Skip to content

feat: support identity-safe reordered fragment rewrites - #8814

Draft
LuQQiu wants to merge 5 commits into
lance-format:mainfrom
LuQQiu:lu/reordered-fragment-rewrite
Draft

feat: support identity-safe reordered fragment rewrites#8814
LuQQiu wants to merge 5 commits into
lance-format:mainfrom
LuQQiu:lu/reordered-fragment-rewrite

Conversation

@LuQQiu

@LuQQiu LuQQiu commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Write caller-reordered streams into uncommitted fragments while preserving stable row IDs and per-row created/updated version metadata.
  • Preserve address-style row addresses in output order so existing indices can be remapped after a physical permutation.
  • Allow callers to reserve final output fragment IDs before committing a rewrite, then commit those IDs without a second reservation.
  • Add explicit fragment groups as hard compaction-planning boundaries.

Motivation

Some dataset maintenance operations physically permute existing rows instead of preserving input order. They still need to preserve logical row identity, keep existing indices correct, know final output fragment IDs when preparing associated work, and prevent later compaction from crossing caller-defined placement boundaries.

This PR adds generic write, remap, reservation, and planning primitives for those workflows. It does not define an ordering or partitioning policy.

Scope

This PR does not add fields to the transaction protobuf, change the Lance format, or combine a rewrite with a dataset-configuration update. Composite operations belong to Transaction V2.

Validation

  • cargo fmt --all
  • cargo check --quiet -p lance -p lance-table
  • cargo check --quiet in python/
  • cargo check --quiet in java/lance-jni/
  • cargo test --quiet -p lance reordered --lib: 8 passed
  • cargo test --quiet -p lance explicit_fragment_groups_are_hard_compaction_boundaries --lib: 1 passed

@github-actions github-actions Bot added the enhancement New feature or request label Aug 26, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Important

This PR touches the Lance format specification.

Substantive changes to the format specification — the .proto definitions
and the spec docs under docs/src/format/ — require a PMC vote before merge.
Minor edits such as typo fixes, wording, or formatting are excluded; use your
judgment.

If this is a meaningful format change:

  • Start a vote following the Lance community voting process.
    Format specification modifications need 3 binding +1 votes (excluding the
    proposer), held on GitHub Discussions, with a minimum voting period of 1 week.
  • Once the vote passes, link the completed vote in this PR. It should not be
    merged until the vote is linked.

@github-actions github-actions Bot added the A-format On-disk format: protos and format spec docs label Aug 26, 2026
@LuQQiu LuQQiu changed the title feat: support row identity remap for reordered rewrites feat: support metadata-aware reordered rewrites Aug 26, 2026
@github-actions github-actions Bot added A-python Python bindings A-java Java bindings + JNI labels Aug 26, 2026
@LuQQiu LuQQiu changed the title feat: support metadata-aware reordered rewrites feat: support identity-safe reordered fragment rewrites Aug 26, 2026
@LuQQiu LuQQiu removed A-python Python bindings A-format On-disk format: protos and format spec docs labels Aug 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-java Java bindings + JNI enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant