feat: CheckpointLeaf file Reader - #1501
Closed
OussamaSaoudi wants to merge 11 commits into
Closed
Conversation
This was referenced Nov 19, 2025
This was referenced Nov 19, 2025
scovich
reviewed
Nov 20, 2025
Comment on lines
+21
to
+25
| /// # Distributability | ||
| /// | ||
| /// This phase is designed to be distributable. To distribute: | ||
| /// 1. Partition `files` across N executors | ||
| /// 2. Create N `LeafCheckpointReader` instances, one per executor with its file partition |
Collaborator
There was a problem hiding this comment.
This seems like an odd place to bring up distributability? The log replay phase that consumes this reader is the thing getting distributed, and which needs the careful choreography? Seems like this reader is just a detail in that bigger picture.
Collaborator
Author
There was a problem hiding this comment.
good point. Removed and moving to higher level.
| let actions = engine | ||
| .parquet_handler() | ||
| .read_parquet_files(&files, schema, None)? | ||
| .map(|batch| batch.map(|b| ActionsBatch::new(b, false))); |
Collaborator
There was a problem hiding this comment.
Now that we're splitting up the phases instead of chaining everything into one big iterator, we should probably reconsider whether we still need these (batch, bool) pairs -- I suspect that whoever invokes the dedup visitor will now know, structurally, whether it's a log or checkpoint batch, and can just pass true or false accordingly?
Collaborator
Author
There was a problem hiding this comment.
Yep agreed. This will likely be a followup of the Deduplicator split. Then we'll just apply the correct deduplicator to the right stream.
Comment on lines
+104
to
+107
| if log_segment.checkpoint_parts.is_empty() { | ||
| println!("Test table has no checkpoint parts, skipping"); | ||
| return Ok(()); | ||
| } |
Collaborator
There was a problem hiding this comment.
This seems very strange to have in a test?? Either the test case has a checkpoint or it doesn't.
And in this case it should have a checkpoint, and panicking at L110 below is a perfectly reasonable test failure mode if the checkpoint somehow went missing.
Collaborator
Author
There was a problem hiding this comment.
ugh yeah LLMs are fickle 😓
| ManifestPhase::new(manifest_file, log_segment.log_root.clone(), engine.clone())?; | ||
|
|
||
| // Drain manifest phase and apply processor | ||
| for batch in manifest_phase.by_ref() { |
Collaborator
Author
There was a problem hiding this comment.
Yeah, it's handy for mutable iteration without giving ownership :D
OussamaSaoudi
force-pushed
the
stack/dlr_leaf
branch
5 times, most recently
from
November 25, 2025 19:58
f0900b6 to
6c60016
Compare
OussamaSaoudi
force-pushed
the
stack/dlr_leaf
branch
from
December 3, 2025 21:45
6c60016 to
511845a
Compare
This was referenced Dec 3, 2025
github-actions
Bot
added
the
breaking-change
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #1501 +/- ##
==========================================
+ Coverage 84.06% 84.10% +0.04%
==========================================
Files 119 121 +2
Lines 32619 33381 +762
Branches 32619 33381 +762
==========================================
+ Hits 27421 28076 +655
- Misses 3832 3919 +87
- Partials 1366 1386 +20 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
OussamaSaoudi
changed the title
OussamaSaoudi
force-pushed
the
stack/dlr_leaf
branch
from
December 8, 2025 17:56
146ec4c to
34850d1
Compare
OussamaSaoudi
force-pushed
the
stack/dlr_leaf
branch
3 times, most recently
from
December 9, 2025 18:44
e5166c8 to
5080bce
Compare
OussamaSaoudi
marked this pull request as ready for review
December 9, 2025 22:17
OussamaSaoudi
requested review from
emkornfield
and removed request for
nicklan
December 9, 2025 22:18
OussamaSaoudi
force-pushed
the
stack/dlr_leaf
branch
2 times, most recently
from
December 17, 2025 02:13
a620555 to
1f52bbc
Compare
OussamaSaoudi
added a commit
that referenced
this pull request
Dec 17, 2025
## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta-kernel-rs/pull/1500/files) to review incremental changes. - [**stack/dlr_manifest**](#1500) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1500/files)] - [stack/dlr_driver](#1502) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1502/files/2f0a14caf7cd1070625525868e18da9c300b8a0f..4285e9c8ae504fe697b83f4719c0e0995e0e3bd1)] - [stack/dlr_serialization_helpers](#1543) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1543/files/4285e9c8ae504fe697b83f4719c0e0995e0e3bd1..60f13ec6ee1f7cc2b57006a09d4b0e82b59733b9)] - [stack/dlr_serde](#1503) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1503/files/60f13ec6ee1f7cc2b57006a09d4b0e82b59733b9..377ae3f2596116d00cba596268090d2f69118b2e)] - [stack/dlr_add_rm_dedup](#1537) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1537/files/377ae3f2596116d00cba596268090d2f69118b2e..06fdf84431d7ee9cf693149784b11e4afb242de1)] - [stack/dlr_immutable_add_rm_dedup](#1538) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1538/files/06fdf84431d7ee9cf693149784b11e4afb242de1..991cdd69f18eb680dbafae3c213093b036cb7a85)] - [stack/dlr_leaf](#1501) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1501/files/991cdd69f18eb680dbafae3c213093b036cb7a85..1f52bbc6512d7cb0a5e2a68954eb434be7313ef0)] - [stack/dlr_executor](#1539) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1539/files/1f52bbc6512d7cb0a5e2a68954eb434be7313ef0..9a4f638d613121fe5819e3fef6c03b3e5102ab73)] - [stack/dlr_api](#1547) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1547/files/9a4f638d613121fe5819e3fef6c03b3e5102ab73..f385656ab1aa1f34caf77fa8518834d0ac0b998c)] - [stack/dlr_scan_state_serde](#1549) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1549/files/f385656ab1aa1f34caf77fa8518834d0ac0b998c..ac99eeec1f5d19fe76e8589f87cf53a7aa140e54)] --------- This PR adds a CheckpointManifestReader that is responsible for reading both manifest checkpoints and single-part checkpoints. For manifest checkpoints, it collects all the sidecar files that are present and returns them for further processing.
OussamaSaoudi
force-pushed
the
stack/dlr_leaf
branch
from
December 17, 2025 18:45
1f52bbc to
5f04e20
Compare
OussamaSaoudi
added a commit
that referenced
this pull request
Dec 17, 2025
## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta-kernel-rs/pull/1502/files) to review incremental changes. - [**stack/dlr_driver**](#1502) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1502/files)] - [stack/dlr_serialization_helpers](#1543) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1543/files/8f86b459613c0f018dd0e45473ba3e6e50d127af..0286f1d09006b099e73333c3d2d6a0873ede6cdd)] - [stack/dlr_serde](#1503) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1503/files/0286f1d09006b099e73333c3d2d6a0873ede6cdd..ee68c851e338a83584b004d919c40eca301e2ee3)] - [stack/dlr_add_rm_dedup](#1537) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1537/files/ee68c851e338a83584b004d919c40eca301e2ee3..436ce448e2dbf08ceb8f13350cb609ae3f554a7c)] - [stack/dlr_immutable_add_rm_dedup](#1538) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1538/files/436ce448e2dbf08ceb8f13350cb609ae3f554a7c..b2c2703bef78ef814c0579e82857cd67a2d1364b)] - [stack/dlr_leaf](#1501) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1501/files/b2c2703bef78ef814c0579e82857cd67a2d1364b..5f04e2059209c09a3ac8f5981d018347829bd8ab)] - [stack/dlr_executor](#1539) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1539/files/5f04e2059209c09a3ac8f5981d018347829bd8ab..7e2d26a6b063927e8b9a9fa62043ad30c705cc43)] - [stack/dlr_api](#1547) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1547/files/7e2d26a6b063927e8b9a9fa62043ad30c705cc43..b608475ed1402d8640b89e5948d066e0867585b3)] - [stack/dlr_scan_state_serde](#1549) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1549/files/b608475ed1402d8640b89e5948d066e0867585b3..c35994fbf457ed95556acc7688456feb81923abf)] --------- ## What changes are proposed in this pull request? This pull request introduces the SequentialPhase for distributed log replay. This phase reads the commit and checkpoint files for a logsegment and performs the log replay processor over the batches. Upon completion, the `finish` method produces `AfterSequential` which has two variants: * `Done`: Indicates that the metadata phase is complete and there is no more work to do. * `Distributed`: Indicates that a distributed phase is required (see the pr at stack/dlr_executor) ## How was this change tested? Ensure that the sequential phase extracts the expected actions for: * commits * single-part checkpoints Ensure that sequential phase produces `AfterSequential::Distributed` in the presence of: * sidecar files in a manifest checkpoint * multi-part checkpoints
OussamaSaoudi
force-pushed
the
stack/dlr_leaf
branch
2 times, most recently
from
December 19, 2025 22:13
1759e62 to
6aacca9
Compare
roeap
pushed a commit
to roeap/delta-kernel-rs
that referenced
this pull request
Jan 8, 2026
## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta-kernel-rs/pull/1499/files) to review incremental changes. - [**stack/dlr_commit**](delta-io#1499) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1499/files)] - [stack/dlr_manifest](delta-io#1500) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1500/files/69a8312ad609546d2fbfac74f9d9082108bca41e..41eba152cc46b0ab3c4daf5ba11d80d5b52e4975)] - [stack/dlr_driver](delta-io#1502) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1502/files/41eba152cc46b0ab3c4daf5ba11d80d5b52e4975..6227645580128ccd91946416e5fbb15ed46e3f64)] - [stack/dlr_serialization_helpers](delta-io#1543) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1543/files/6227645580128ccd91946416e5fbb15ed46e3f64..2e700abd0d6e12553819285b311c4ba1c49b4bf1)] - [stack/dlr_serde](delta-io#1503) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1503/files/2e700abd0d6e12553819285b311c4ba1c49b4bf1..c016ad815161e6d44e7280c194b3407f3322905f)] - [stack/dlr_add_rm_dedup](delta-io#1537) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1537/files/c016ad815161e6d44e7280c194b3407f3322905f..61f0908590d905f97ee8052aadaa5b64f74fc4de)] - [stack/dlr_immutable_add_rm_dedup](delta-io#1538) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1538/files/61f0908590d905f97ee8052aadaa5b64f74fc4de..6865ff9760fa5716bc75d92cef9d43422804118d)] - [stack/dlr_leaf](delta-io#1501) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1501/files/6865ff9760fa5716bc75d92cef9d43422804118d..8d59594b63db32fdf208026f8784b7622675e3d8)] - [stack/dlr_executor](delta-io#1539) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1539/files/8d59594b63db32fdf208026f8784b7622675e3d8..cb2a8c4f06d8aa412cc649348f6467b15bae2a9b)] - [stack/dlr_api](delta-io#1547) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1547/files/a3566ba6f3631224516b92238285ebe49626da01..4b6f568d2a6b57ac01af6264921a5ba6b2567abc)] - [stack/dlr_scan_state_serde](delta-io#1549) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1549/files/4b6f568d2a6b57ac01af6264921a5ba6b2567abc..be1183715647d39952529083bbb51c2266530f58)] --------- <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-incubator/delta-kernel-rs/blob/main/CONTRIBUTING.md 2. Run `cargo t --all-features --all-targets` to get started testing, and run `cargo fmt`. 3. Ensure you have added or run the appropriate tests for your PR. 4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 5. Be sure to keep the PR description updated to reflect all changes. --> <!-- PR title formatting: This project uses conventional commits: https://www.conventionalcommits.org/ Each PR corresponds to a commit on the `main` branch, with the title of the PR (typically) being used for the commit message on main. In order to ensure proper formatting in the CHANGELOG please ensure your PR title adheres to the conventional commit specification. Examples: - new feature PR: "feat: new API for snapshot.update()" - bugfix PR: "fix: correctly apply DV in read-table example" --> ## What changes are proposed in this pull request? <!-- Please clarify what changes you are proposing and why the changes are needed. The purpose of this section is to outline the changes, why they are needed, and how this PR fixes the issue. If the reason for the change is already explained clearly in an issue, then it does not need to be restated here. 1. If you propose a new API or feature, clarify the use case for a new API or feature. 2. If you fix a bug, you can clarify why it is a bug. --> This PR adds the commit logreader, a module that will hold all the logic for processing commit files when performing log replay. * CommitReader reads the delta log using the commit cover. * CommitReader implements `Iterator<Item = ActionBatch>`. * CommitReader can read using an arbitrary log schema. <!-- Uncomment this section if there are any changes affecting public APIs: ### This PR affects the following public APIs If there are breaking changes, please ensure the `breaking-changes` label gets added by CI, and describe why the changes are needed. Note that _new_ public APIs are not considered breaking. --> ## How was this change tested? <!-- Please make sure to add test cases that check the changes thoroughly including negative and positive cases if possible. If it was tested in a way different from regular unit tests, please clarify how you tested, ideally via a reproducible test documented in the PR description. --> This is tested with the `app-txn-no-checkpoint` which has multiple commits and multiple add actions in each commit. We ensure that the CommitReader reads the correct set of add actions.
roeap
pushed a commit
to roeap/delta-kernel-rs
that referenced
this pull request
Jan 8, 2026
…#1500) ## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta-kernel-rs/pull/1500/files) to review incremental changes. - [**stack/dlr_manifest**](delta-io#1500) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1500/files)] - [stack/dlr_driver](delta-io#1502) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1502/files/2f0a14caf7cd1070625525868e18da9c300b8a0f..4285e9c8ae504fe697b83f4719c0e0995e0e3bd1)] - [stack/dlr_serialization_helpers](delta-io#1543) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1543/files/4285e9c8ae504fe697b83f4719c0e0995e0e3bd1..60f13ec6ee1f7cc2b57006a09d4b0e82b59733b9)] - [stack/dlr_serde](delta-io#1503) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1503/files/60f13ec6ee1f7cc2b57006a09d4b0e82b59733b9..377ae3f2596116d00cba596268090d2f69118b2e)] - [stack/dlr_add_rm_dedup](delta-io#1537) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1537/files/377ae3f2596116d00cba596268090d2f69118b2e..06fdf84431d7ee9cf693149784b11e4afb242de1)] - [stack/dlr_immutable_add_rm_dedup](delta-io#1538) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1538/files/06fdf84431d7ee9cf693149784b11e4afb242de1..991cdd69f18eb680dbafae3c213093b036cb7a85)] - [stack/dlr_leaf](delta-io#1501) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1501/files/991cdd69f18eb680dbafae3c213093b036cb7a85..1f52bbc6512d7cb0a5e2a68954eb434be7313ef0)] - [stack/dlr_executor](delta-io#1539) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1539/files/1f52bbc6512d7cb0a5e2a68954eb434be7313ef0..9a4f638d613121fe5819e3fef6c03b3e5102ab73)] - [stack/dlr_api](delta-io#1547) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1547/files/9a4f638d613121fe5819e3fef6c03b3e5102ab73..f385656ab1aa1f34caf77fa8518834d0ac0b998c)] - [stack/dlr_scan_state_serde](delta-io#1549) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1549/files/f385656ab1aa1f34caf77fa8518834d0ac0b998c..ac99eeec1f5d19fe76e8589f87cf53a7aa140e54)] --------- This PR adds a CheckpointManifestReader that is responsible for reading both manifest checkpoints and single-part checkpoints. For manifest checkpoints, it collects all the sidecar files that are present and returns them for further processing.
roeap
pushed a commit
to roeap/delta-kernel-rs
that referenced
this pull request
Jan 8, 2026
## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta-kernel-rs/pull/1502/files) to review incremental changes. - [**stack/dlr_driver**](delta-io#1502) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1502/files)] - [stack/dlr_serialization_helpers](delta-io#1543) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1543/files/8f86b459613c0f018dd0e45473ba3e6e50d127af..0286f1d09006b099e73333c3d2d6a0873ede6cdd)] - [stack/dlr_serde](delta-io#1503) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1503/files/0286f1d09006b099e73333c3d2d6a0873ede6cdd..ee68c851e338a83584b004d919c40eca301e2ee3)] - [stack/dlr_add_rm_dedup](delta-io#1537) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1537/files/ee68c851e338a83584b004d919c40eca301e2ee3..436ce448e2dbf08ceb8f13350cb609ae3f554a7c)] - [stack/dlr_immutable_add_rm_dedup](delta-io#1538) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1538/files/436ce448e2dbf08ceb8f13350cb609ae3f554a7c..b2c2703bef78ef814c0579e82857cd67a2d1364b)] - [stack/dlr_leaf](delta-io#1501) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1501/files/b2c2703bef78ef814c0579e82857cd67a2d1364b..5f04e2059209c09a3ac8f5981d018347829bd8ab)] - [stack/dlr_executor](delta-io#1539) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1539/files/5f04e2059209c09a3ac8f5981d018347829bd8ab..7e2d26a6b063927e8b9a9fa62043ad30c705cc43)] - [stack/dlr_api](delta-io#1547) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1547/files/7e2d26a6b063927e8b9a9fa62043ad30c705cc43..b608475ed1402d8640b89e5948d066e0867585b3)] - [stack/dlr_scan_state_serde](delta-io#1549) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1549/files/b608475ed1402d8640b89e5948d066e0867585b3..c35994fbf457ed95556acc7688456feb81923abf)] --------- ## What changes are proposed in this pull request? This pull request introduces the SequentialPhase for distributed log replay. This phase reads the commit and checkpoint files for a logsegment and performs the log replay processor over the batches. Upon completion, the `finish` method produces `AfterSequential` which has two variants: * `Done`: Indicates that the metadata phase is complete and there is no more work to do. * `Distributed`: Indicates that a distributed phase is required (see the pr at stack/dlr_executor) ## How was this change tested? Ensure that the sequential phase extracts the expected actions for: * commits * single-part checkpoints Ensure that sequential phase produces `AfterSequential::Distributed` in the presence of: * sidecar files in a manifest checkpoint * multi-part checkpoints
emkornfield
pushed a commit
to emkornfield/delta-kernel-rs
that referenced
this pull request
Jan 8, 2026
…#1500) ## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta-kernel-rs/pull/1500/files) to review incremental changes. - [**stack/dlr_manifest**](delta-io#1500) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1500/files)] - [stack/dlr_driver](delta-io#1502) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1502/files/2f0a14caf7cd1070625525868e18da9c300b8a0f..4285e9c8ae504fe697b83f4719c0e0995e0e3bd1)] - [stack/dlr_serialization_helpers](delta-io#1543) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1543/files/4285e9c8ae504fe697b83f4719c0e0995e0e3bd1..60f13ec6ee1f7cc2b57006a09d4b0e82b59733b9)] - [stack/dlr_serde](delta-io#1503) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1503/files/60f13ec6ee1f7cc2b57006a09d4b0e82b59733b9..377ae3f2596116d00cba596268090d2f69118b2e)] - [stack/dlr_add_rm_dedup](delta-io#1537) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1537/files/377ae3f2596116d00cba596268090d2f69118b2e..06fdf84431d7ee9cf693149784b11e4afb242de1)] - [stack/dlr_immutable_add_rm_dedup](delta-io#1538) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1538/files/06fdf84431d7ee9cf693149784b11e4afb242de1..991cdd69f18eb680dbafae3c213093b036cb7a85)] - [stack/dlr_leaf](delta-io#1501) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1501/files/991cdd69f18eb680dbafae3c213093b036cb7a85..1f52bbc6512d7cb0a5e2a68954eb434be7313ef0)] - [stack/dlr_executor](delta-io#1539) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1539/files/1f52bbc6512d7cb0a5e2a68954eb434be7313ef0..9a4f638d613121fe5819e3fef6c03b3e5102ab73)] - [stack/dlr_api](delta-io#1547) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1547/files/9a4f638d613121fe5819e3fef6c03b3e5102ab73..f385656ab1aa1f34caf77fa8518834d0ac0b998c)] - [stack/dlr_scan_state_serde](delta-io#1549) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1549/files/f385656ab1aa1f34caf77fa8518834d0ac0b998c..ac99eeec1f5d19fe76e8589f87cf53a7aa140e54)] --------- This PR adds a CheckpointManifestReader that is responsible for reading both manifest checkpoints and single-part checkpoints. For manifest checkpoints, it collects all the sidecar files that are present and returns them for further processing.
emkornfield
pushed a commit
to emkornfield/delta-kernel-rs
that referenced
this pull request
Jan 8, 2026
## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta-kernel-rs/pull/1502/files) to review incremental changes. - [**stack/dlr_driver**](delta-io#1502) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1502/files)] - [stack/dlr_serialization_helpers](delta-io#1543) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1543/files/8f86b459613c0f018dd0e45473ba3e6e50d127af..0286f1d09006b099e73333c3d2d6a0873ede6cdd)] - [stack/dlr_serde](delta-io#1503) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1503/files/0286f1d09006b099e73333c3d2d6a0873ede6cdd..ee68c851e338a83584b004d919c40eca301e2ee3)] - [stack/dlr_add_rm_dedup](delta-io#1537) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1537/files/ee68c851e338a83584b004d919c40eca301e2ee3..436ce448e2dbf08ceb8f13350cb609ae3f554a7c)] - [stack/dlr_immutable_add_rm_dedup](delta-io#1538) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1538/files/436ce448e2dbf08ceb8f13350cb609ae3f554a7c..b2c2703bef78ef814c0579e82857cd67a2d1364b)] - [stack/dlr_leaf](delta-io#1501) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1501/files/b2c2703bef78ef814c0579e82857cd67a2d1364b..5f04e2059209c09a3ac8f5981d018347829bd8ab)] - [stack/dlr_executor](delta-io#1539) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1539/files/5f04e2059209c09a3ac8f5981d018347829bd8ab..7e2d26a6b063927e8b9a9fa62043ad30c705cc43)] - [stack/dlr_api](delta-io#1547) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1547/files/7e2d26a6b063927e8b9a9fa62043ad30c705cc43..b608475ed1402d8640b89e5948d066e0867585b3)] - [stack/dlr_scan_state_serde](delta-io#1549) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1549/files/b608475ed1402d8640b89e5948d066e0867585b3..c35994fbf457ed95556acc7688456feb81923abf)] --------- ## What changes are proposed in this pull request? This pull request introduces the SequentialPhase for distributed log replay. This phase reads the commit and checkpoint files for a logsegment and performs the log replay processor over the batches. Upon completion, the `finish` method produces `AfterSequential` which has two variants: * `Done`: Indicates that the metadata phase is complete and there is no more work to do. * `Distributed`: Indicates that a distributed phase is required (see the pr at stack/dlr_executor) ## How was this change tested? Ensure that the sequential phase extracts the expected actions for: * commits * single-part checkpoints Ensure that sequential phase produces `AfterSequential::Distributed` in the presence of: * sidecar files in a manifest checkpoint * multi-part checkpoints
Fokko
pushed a commit
to Fokko/delta-kernel-rs
that referenced
this pull request
Jan 9, 2026
* feat: Add CheckpointManifestReader to process sidecar files (delta-io#1500) ## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta-kernel-rs/pull/1500/files) to review incremental changes. - [**stack/dlr_manifest**](delta-io#1500) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1500/files)] - [stack/dlr_driver](delta-io#1502) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1502/files/2f0a14caf7cd1070625525868e18da9c300b8a0f..4285e9c8ae504fe697b83f4719c0e0995e0e3bd1)] - [stack/dlr_serialization_helpers](delta-io#1543) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1543/files/4285e9c8ae504fe697b83f4719c0e0995e0e3bd1..60f13ec6ee1f7cc2b57006a09d4b0e82b59733b9)] - [stack/dlr_serde](delta-io#1503) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1503/files/60f13ec6ee1f7cc2b57006a09d4b0e82b59733b9..377ae3f2596116d00cba596268090d2f69118b2e)] - [stack/dlr_add_rm_dedup](delta-io#1537) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1537/files/377ae3f2596116d00cba596268090d2f69118b2e..06fdf84431d7ee9cf693149784b11e4afb242de1)] - [stack/dlr_immutable_add_rm_dedup](delta-io#1538) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1538/files/06fdf84431d7ee9cf693149784b11e4afb242de1..991cdd69f18eb680dbafae3c213093b036cb7a85)] - [stack/dlr_leaf](delta-io#1501) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1501/files/991cdd69f18eb680dbafae3c213093b036cb7a85..1f52bbc6512d7cb0a5e2a68954eb434be7313ef0)] - [stack/dlr_executor](delta-io#1539) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1539/files/1f52bbc6512d7cb0a5e2a68954eb434be7313ef0..9a4f638d613121fe5819e3fef6c03b3e5102ab73)] - [stack/dlr_api](delta-io#1547) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1547/files/9a4f638d613121fe5819e3fef6c03b3e5102ab73..f385656ab1aa1f34caf77fa8518834d0ac0b998c)] - [stack/dlr_scan_state_serde](delta-io#1549) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1549/files/f385656ab1aa1f34caf77fa8518834d0ac0b998c..ac99eeec1f5d19fe76e8589f87cf53a7aa140e54)] --------- This PR adds a CheckpointManifestReader that is responsible for reading both manifest checkpoints and single-part checkpoints. For manifest checkpoints, it collects all the sidecar files that are present and returns them for further processing. * feat: Distributed Log Replay Sequential Phase (delta-io#1502) ## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta-kernel-rs/pull/1502/files) to review incremental changes. - [**stack/dlr_driver**](delta-io#1502) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1502/files)] - [stack/dlr_serialization_helpers](delta-io#1543) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1543/files/8f86b459613c0f018dd0e45473ba3e6e50d127af..0286f1d09006b099e73333c3d2d6a0873ede6cdd)] - [stack/dlr_serde](delta-io#1503) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1503/files/0286f1d09006b099e73333c3d2d6a0873ede6cdd..ee68c851e338a83584b004d919c40eca301e2ee3)] - [stack/dlr_add_rm_dedup](delta-io#1537) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1537/files/ee68c851e338a83584b004d919c40eca301e2ee3..436ce448e2dbf08ceb8f13350cb609ae3f554a7c)] - [stack/dlr_immutable_add_rm_dedup](delta-io#1538) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1538/files/436ce448e2dbf08ceb8f13350cb609ae3f554a7c..b2c2703bef78ef814c0579e82857cd67a2d1364b)] - [stack/dlr_leaf](delta-io#1501) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1501/files/b2c2703bef78ef814c0579e82857cd67a2d1364b..5f04e2059209c09a3ac8f5981d018347829bd8ab)] - [stack/dlr_executor](delta-io#1539) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1539/files/5f04e2059209c09a3ac8f5981d018347829bd8ab..7e2d26a6b063927e8b9a9fa62043ad30c705cc43)] - [stack/dlr_api](delta-io#1547) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1547/files/7e2d26a6b063927e8b9a9fa62043ad30c705cc43..b608475ed1402d8640b89e5948d066e0867585b3)] - [stack/dlr_scan_state_serde](delta-io#1549) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1549/files/b608475ed1402d8640b89e5948d066e0867585b3..c35994fbf457ed95556acc7688456feb81923abf)] --------- ## What changes are proposed in this pull request? This pull request introduces the SequentialPhase for distributed log replay. This phase reads the commit and checkpoint files for a logsegment and performs the log replay processor over the batches. Upon completion, the `finish` method produces `AfterSequential` which has two variants: * `Done`: Indicates that the metadata phase is complete and there is no more work to do. * `Distributed`: Indicates that a distributed phase is required (see the pr at stack/dlr_executor) ## How was this change tested? Ensure that the sequential phase extracts the expected actions for: * commits * single-part checkpoints Ensure that sequential phase produces `AfterSequential::Distributed` in the presence of: * sidecar files in a manifest checkpoint * multi-part checkpoints * feat/bugfix: Passing schema from C, plus example/tests in C (delta-io#1535) ## What changes are proposed in this pull request? * Don't use a macro to generate primitive visitors. `cbindgen` runs before macro expansion, so previously none of those visitors were actually being generated * Add code to `read_table` to allow specifying which columns to select, plus all the associated code to pass that back to kernel * Add c based tests to make sure it's working ## How was this change tested? New tests when running `make test` in `read_table` --------- Co-authored-by: OussamaSaoudi <45303303+OussamaSaoudi@users.noreply.github.com> * feat!: expose mod time in scan metadata callbacks (delta-io#1565) * feat: Support sidecar in inspect-table (delta-io#1566) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-incubator/delta-kernel-rs/blob/main/CONTRIBUTING.md 2. Run `cargo t --all-features --all-targets` to get started testing, and run `cargo fmt`. 3. Ensure you have added or run the appropriate tests for your PR. 4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 5. Be sure to keep the PR description updated to reflect all changes. --> <!-- PR title formatting: This project uses conventional commits: https://www.conventionalcommits.org/ Each PR corresponds to a commit on the `main` branch, with the title of the PR (typically) being used for the commit message on main. In order to ensure proper formatting in the CHANGELOG please ensure your PR title adheres to the conventional commit specification. Examples: - new feature PR: "feat: new API for snapshot.update()" - bugfix PR: "fix: correctly apply DV in read-table example" --> ## What changes are proposed in this pull request? <!-- Please clarify what changes you are proposing and why the changes are needed. The purpose of this section is to outline the changes, why they are needed, and how this PR fixes the issue. If the reason for the change is already explained clearly in an issue, then it does not need to be restated here. 1. If you propose a new API or feature, clarify the use case for a new API or feature. 2. If you fix a bug, you can clarify why it is a bug. --> Support sidecar in inspect-table. Solves the issue delta-io#695. <!-- Uncomment this section if there are any changes affecting public APIs: ### This PR affects the following public APIs If there are breaking changes, please ensure the `breaking-changes` label gets added by CI, and describe why the changes are needed. Note that _new_ public APIs are not considered breaking. --> ## How was this change tested? <!-- Please make sure to add test cases that check the changes thoroughly including negative and positive cases if possible. If it was tested in a way different from regular unit tests, please clarify how you tested, ideally via a reproducible test documented in the PR description. --> Test locally. Used `./kernel/tests/data/v2-checkpoints-parquet-with-sidecars.tar.zst`. The output successfully print sidecar actions: ``` cargo run -p inspect-table -- v2-checkpoints-parquet-with-sidecars actions | head -17 Action 1: Sidecar { path: "00000000000000000006.checkpoint.0000000001.0000000002.76931b15-ead3-480d-b86c-afe55a577fc3.parquet", size_in_bytes: 12461, modification_time: 1739329719752, tags: None, } Action 2: Sidecar { path: "00000000000000000006.checkpoint.0000000002.0000000002.4367b29c-0e87-447f-8e81-9814cc01ad1f.parquet", size_in_bytes: 12109, modification_time: 1739329719746, tags: None, } ``` * fix: force usage of ListedLogFiles::try_new() (delta-io#1562) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-incubator/delta-kernel-rs/blob/main/CONTRIBUTING.md 2. Run `cargo t --all-features --all-targets` to get started testing, and run `cargo fmt`. 3. Ensure you have added or run the appropriate tests for your PR. 4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 5. Be sure to keep the PR description updated to reflect all changes. --> <!-- PR title formatting: This project uses conventional commits: https://www.conventionalcommits.org/ Each PR corresponds to a commit on the `main` branch, with the title of the PR (typically) being used for the commit message on main. In order to ensure proper formatting in the CHANGELOG please ensure your PR title adheres to the conventional commit specification. Examples: - new feature PR: "feat: new API for snapshot.update()" - bugfix PR: "fix: correctly apply DV in read-table example" --> ## What changes are proposed in this pull request? <!-- Please clarify what changes you are proposing and why the changes are needed. The purpose of this section is to outline the changes, why they are needed, and how this PR fixes the issue. If the reason for the change is already explained clearly in an issue, then it does not need to be restated here. 1. If you propose a new API or feature, clarify the use case for a new API or feature. 2. If you fix a bug, you can clarify why it is a bug. --> Change field of `ListedLogFiles` to private to force the usage of `ListedLogFiles::try_new()` Solves the issue delta-io#1143 <!-- Uncomment this section if there are any changes affecting public APIs: ### This PR affects the following public APIs If there are breaking changes, please ensure the `breaking-changes` label gets added by CI, and describe why the changes are needed. Note that _new_ public APIs are not considered breaking. --> ## How was this change tested? Existing tests <!-- Please make sure to add test cases that check the changes thoroughly including negative and positive cases if possible. If it was tested in a way different from regular unit tests, please clarify how you tested, ideally via a reproducible test documented in the PR description. --> * fix compile * fix clippy --------- Co-authored-by: OussamaSaoudi <45303303+OussamaSaoudi@users.noreply.github.com> Co-authored-by: Nick Lanham <nicklan@users.noreply.github.com> Co-authored-by: dengsh12 <62536982+dengsh12@users.noreply.github.com>
Fokko
added a commit
to Fokko/delta-kernel-rs
that referenced
this pull request
Jan 9, 2026
* feat: Add CheckpointManifestReader to process sidecar files (delta-io#1500) ## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta-kernel-rs/pull/1500/files) to review incremental changes. - [**stack/dlr_manifest**](delta-io#1500) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1500/files)] - [stack/dlr_driver](delta-io#1502) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1502/files/2f0a14caf7cd1070625525868e18da9c300b8a0f..4285e9c8ae504fe697b83f4719c0e0995e0e3bd1)] - [stack/dlr_serialization_helpers](delta-io#1543) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1543/files/4285e9c8ae504fe697b83f4719c0e0995e0e3bd1..60f13ec6ee1f7cc2b57006a09d4b0e82b59733b9)] - [stack/dlr_serde](delta-io#1503) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1503/files/60f13ec6ee1f7cc2b57006a09d4b0e82b59733b9..377ae3f2596116d00cba596268090d2f69118b2e)] - [stack/dlr_add_rm_dedup](delta-io#1537) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1537/files/377ae3f2596116d00cba596268090d2f69118b2e..06fdf84431d7ee9cf693149784b11e4afb242de1)] - [stack/dlr_immutable_add_rm_dedup](delta-io#1538) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1538/files/06fdf84431d7ee9cf693149784b11e4afb242de1..991cdd69f18eb680dbafae3c213093b036cb7a85)] - [stack/dlr_leaf](delta-io#1501) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1501/files/991cdd69f18eb680dbafae3c213093b036cb7a85..1f52bbc6512d7cb0a5e2a68954eb434be7313ef0)] - [stack/dlr_executor](delta-io#1539) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1539/files/1f52bbc6512d7cb0a5e2a68954eb434be7313ef0..9a4f638d613121fe5819e3fef6c03b3e5102ab73)] - [stack/dlr_api](delta-io#1547) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1547/files/9a4f638d613121fe5819e3fef6c03b3e5102ab73..f385656ab1aa1f34caf77fa8518834d0ac0b998c)] - [stack/dlr_scan_state_serde](delta-io#1549) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1549/files/f385656ab1aa1f34caf77fa8518834d0ac0b998c..ac99eeec1f5d19fe76e8589f87cf53a7aa140e54)] --------- This PR adds a CheckpointManifestReader that is responsible for reading both manifest checkpoints and single-part checkpoints. For manifest checkpoints, it collects all the sidecar files that are present and returns them for further processing. * feat: Distributed Log Replay Sequential Phase (delta-io#1502) ## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta-kernel-rs/pull/1502/files) to review incremental changes. - [**stack/dlr_driver**](delta-io#1502) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1502/files)] - [stack/dlr_serialization_helpers](delta-io#1543) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1543/files/8f86b459613c0f018dd0e45473ba3e6e50d127af..0286f1d09006b099e73333c3d2d6a0873ede6cdd)] - [stack/dlr_serde](delta-io#1503) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1503/files/0286f1d09006b099e73333c3d2d6a0873ede6cdd..ee68c851e338a83584b004d919c40eca301e2ee3)] - [stack/dlr_add_rm_dedup](delta-io#1537) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1537/files/ee68c851e338a83584b004d919c40eca301e2ee3..436ce448e2dbf08ceb8f13350cb609ae3f554a7c)] - [stack/dlr_immutable_add_rm_dedup](delta-io#1538) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1538/files/436ce448e2dbf08ceb8f13350cb609ae3f554a7c..b2c2703bef78ef814c0579e82857cd67a2d1364b)] - [stack/dlr_leaf](delta-io#1501) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1501/files/b2c2703bef78ef814c0579e82857cd67a2d1364b..5f04e2059209c09a3ac8f5981d018347829bd8ab)] - [stack/dlr_executor](delta-io#1539) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1539/files/5f04e2059209c09a3ac8f5981d018347829bd8ab..7e2d26a6b063927e8b9a9fa62043ad30c705cc43)] - [stack/dlr_api](delta-io#1547) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1547/files/7e2d26a6b063927e8b9a9fa62043ad30c705cc43..b608475ed1402d8640b89e5948d066e0867585b3)] - [stack/dlr_scan_state_serde](delta-io#1549) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1549/files/b608475ed1402d8640b89e5948d066e0867585b3..c35994fbf457ed95556acc7688456feb81923abf)] --------- ## What changes are proposed in this pull request? This pull request introduces the SequentialPhase for distributed log replay. This phase reads the commit and checkpoint files for a logsegment and performs the log replay processor over the batches. Upon completion, the `finish` method produces `AfterSequential` which has two variants: * `Done`: Indicates that the metadata phase is complete and there is no more work to do. * `Distributed`: Indicates that a distributed phase is required (see the pr at stack/dlr_executor) ## How was this change tested? Ensure that the sequential phase extracts the expected actions for: * commits * single-part checkpoints Ensure that sequential phase produces `AfterSequential::Distributed` in the presence of: * sidecar files in a manifest checkpoint * multi-part checkpoints * feat/bugfix: Passing schema from C, plus example/tests in C (delta-io#1535) ## What changes are proposed in this pull request? * Don't use a macro to generate primitive visitors. `cbindgen` runs before macro expansion, so previously none of those visitors were actually being generated * Add code to `read_table` to allow specifying which columns to select, plus all the associated code to pass that back to kernel * Add c based tests to make sure it's working ## How was this change tested? New tests when running `make test` in `read_table` --------- Co-authored-by: OussamaSaoudi <45303303+OussamaSaoudi@users.noreply.github.com> * feat!: expose mod time in scan metadata callbacks (delta-io#1565) * feat: Support sidecar in inspect-table (delta-io#1566) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-incubator/delta-kernel-rs/blob/main/CONTRIBUTING.md 2. Run `cargo t --all-features --all-targets` to get started testing, and run `cargo fmt`. 3. Ensure you have added or run the appropriate tests for your PR. 4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 5. Be sure to keep the PR description updated to reflect all changes. --> <!-- PR title formatting: This project uses conventional commits: https://www.conventionalcommits.org/ Each PR corresponds to a commit on the `main` branch, with the title of the PR (typically) being used for the commit message on main. In order to ensure proper formatting in the CHANGELOG please ensure your PR title adheres to the conventional commit specification. Examples: - new feature PR: "feat: new API for snapshot.update()" - bugfix PR: "fix: correctly apply DV in read-table example" --> ## What changes are proposed in this pull request? <!-- Please clarify what changes you are proposing and why the changes are needed. The purpose of this section is to outline the changes, why they are needed, and how this PR fixes the issue. If the reason for the change is already explained clearly in an issue, then it does not need to be restated here. 1. If you propose a new API or feature, clarify the use case for a new API or feature. 2. If you fix a bug, you can clarify why it is a bug. --> Support sidecar in inspect-table. Solves the issue delta-io#695. <!-- Uncomment this section if there are any changes affecting public APIs: ### This PR affects the following public APIs If there are breaking changes, please ensure the `breaking-changes` label gets added by CI, and describe why the changes are needed. Note that _new_ public APIs are not considered breaking. --> ## How was this change tested? <!-- Please make sure to add test cases that check the changes thoroughly including negative and positive cases if possible. If it was tested in a way different from regular unit tests, please clarify how you tested, ideally via a reproducible test documented in the PR description. --> Test locally. Used `./kernel/tests/data/v2-checkpoints-parquet-with-sidecars.tar.zst`. The output successfully print sidecar actions: ``` cargo run -p inspect-table -- v2-checkpoints-parquet-with-sidecars actions | head -17 Action 1: Sidecar { path: "00000000000000000006.checkpoint.0000000001.0000000002.76931b15-ead3-480d-b86c-afe55a577fc3.parquet", size_in_bytes: 12461, modification_time: 1739329719752, tags: None, } Action 2: Sidecar { path: "00000000000000000006.checkpoint.0000000002.0000000002.4367b29c-0e87-447f-8e81-9814cc01ad1f.parquet", size_in_bytes: 12109, modification_time: 1739329719746, tags: None, } ``` * fix: force usage of ListedLogFiles::try_new() (delta-io#1562) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-incubator/delta-kernel-rs/blob/main/CONTRIBUTING.md 2. Run `cargo t --all-features --all-targets` to get started testing, and run `cargo fmt`. 3. Ensure you have added or run the appropriate tests for your PR. 4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 5. Be sure to keep the PR description updated to reflect all changes. --> <!-- PR title formatting: This project uses conventional commits: https://www.conventionalcommits.org/ Each PR corresponds to a commit on the `main` branch, with the title of the PR (typically) being used for the commit message on main. In order to ensure proper formatting in the CHANGELOG please ensure your PR title adheres to the conventional commit specification. Examples: - new feature PR: "feat: new API for snapshot.update()" - bugfix PR: "fix: correctly apply DV in read-table example" --> ## What changes are proposed in this pull request? <!-- Please clarify what changes you are proposing and why the changes are needed. The purpose of this section is to outline the changes, why they are needed, and how this PR fixes the issue. If the reason for the change is already explained clearly in an issue, then it does not need to be restated here. 1. If you propose a new API or feature, clarify the use case for a new API or feature. 2. If you fix a bug, you can clarify why it is a bug. --> Change field of `ListedLogFiles` to private to force the usage of `ListedLogFiles::try_new()` Solves the issue delta-io#1143 <!-- Uncomment this section if there are any changes affecting public APIs: ### This PR affects the following public APIs If there are breaking changes, please ensure the `breaking-changes` label gets added by CI, and describe why the changes are needed. Note that _new_ public APIs are not considered breaking. --> ## How was this change tested? Existing tests <!-- Please make sure to add test cases that check the changes thoroughly including negative and positive cases if possible. If it was tested in a way different from regular unit tests, please clarify how you tested, ideally via a reproducible test documented in the PR description. --> * fix: improve parse_json performance by removing line-by-line parsing (delta-io#1561) Co-authored-by: emkornfield <emkornfield@gmail.com> Co-authored-by: Nick Lanham <nick@databricks.com> * sync changelog from `0.18.2` (delta-io#1570) cherry-picks f105333 Co-authored-by: OussamaSaoudi <45303303+OussamaSaoudi@users.noreply.github.com> Co-authored-by: Drake Lin <drakelin18@gmail.com> * feature: short-circuit coalesce evaluation when array has no nulls (delta-io#1568) ## What changes are proposed in this pull request? <!-- Please clarify what changes you are proposing and why the changes are needed. The purpose of this section is to outline the changes, why they are needed, and how this PR fixes the issue. If the reason for the change is already explained clearly in an issue, then it does not need to be restated here. 1. If you propose a new API or feature, clarify the use case for a new API or feature. 2. If you fix a bug, you can clarify why it is a bug. --> Optimizes the coalesce expression evaluation in the default Arrow engine to short-circuit when an evaluated array has no null values. Previously, all expressions were eagerly evaluated before coalescing. - After evaluating each expression, check if `null_count() == 0` - If no nulls exist, validate the result type and return immediately without evaluating remaining expressions - `null_count()` on Arrow arrays is O(1) (cached value), so the check is essentially free. This optimization avoids evaluating potentially expensive expressions when earlier expressions already provide complete (non-null) results. Usecase: - For ParsedStats, we will see coalesces in both read/write - In read: we want to coalesce(parsed_stats, ParseJson(json_stats)) - In write: - ParsedStats: coalesce(parsed_stats, ParseJson(json_stats)) - JsonStats: coalesce(json_stats, ToJson(ParsedStats)) In each of these cases, all of one column will either be Null or non-Null so this optimization actually completely removes the need to evaluate the expensive parseJson/ToJson if unnecessary. Future improvements: - Track each row such that it short-circuits as soon as all rows are filled, even if no single array was 100% non-null - This can be implemented with FilteredEngineData, but that lacks support across the engine <!-- Uncomment this section if there are any changes affecting public APIs: ### This PR affects the following public APIs If there are breaking changes, please ensure the `breaking-changes` label gets added by CI, and describe why the changes are needed. Note that _new_ public APIs are not considered breaking. --> ## How was this change tested? <!-- Please make sure to add test cases that check the changes thoroughly including negative and positive cases if possible. If it was tested in a way different from regular unit tests, please clarify how you tested, ideally via a reproducible test documented in the PR description. --> new and existing unit tests * release 0.19.0 (delta-io#1571) release 0.19.0: [link to rendered changelog](https://github.com/zachschuermann/delta-kernel-rs/blob/a23efae27cf8bd7859a24ad66f57dcb0f6fac851/CHANGELOG.md) --------- Co-authored-by: OussamaSaoudi <45303303+OussamaSaoudi@users.noreply.github.com> Co-authored-by: Nick Lanham <nicklan@users.noreply.github.com> Co-authored-by: dengsh12 <62536982+dengsh12@users.noreply.github.com> Co-authored-by: R. Tyler Croy <rtyler@brokenco.de> Co-authored-by: emkornfield <emkornfield@gmail.com> Co-authored-by: Nick Lanham <nick@databricks.com> Co-authored-by: Zach Schuermann <zachary.zvs@gmail.com> Co-authored-by: Drake Lin <drakelin18@gmail.com>
OussamaSaoudi
force-pushed
the
stack/dlr_leaf
branch
from
January 12, 2026 17:48
6aacca9 to
0ed7aaf
Compare
This reverts commit 22c6b61.
OussamaSaoudi
force-pushed
the
stack/dlr_leaf
branch
from
January 12, 2026 17:51
0ed7aaf to
281f192
Compare
Collaborator
Author
|
Not needed. |
Fokko
added a commit
to Fokko/delta-kernel-rs
that referenced
this pull request
Jan 30, 2026
* feat: Add CheckpointManifestReader to process sidecar files (delta-io#1500) ## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta-kernel-rs/pull/1500/files) to review incremental changes. - [**stack/dlr_manifest**](delta-io#1500) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1500/files)] - [stack/dlr_driver](delta-io#1502) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1502/files/2f0a14caf7cd1070625525868e18da9c300b8a0f..4285e9c8ae504fe697b83f4719c0e0995e0e3bd1)] - [stack/dlr_serialization_helpers](delta-io#1543) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1543/files/4285e9c8ae504fe697b83f4719c0e0995e0e3bd1..60f13ec6ee1f7cc2b57006a09d4b0e82b59733b9)] - [stack/dlr_serde](delta-io#1503) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1503/files/60f13ec6ee1f7cc2b57006a09d4b0e82b59733b9..377ae3f2596116d00cba596268090d2f69118b2e)] - [stack/dlr_add_rm_dedup](delta-io#1537) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1537/files/377ae3f2596116d00cba596268090d2f69118b2e..06fdf84431d7ee9cf693149784b11e4afb242de1)] - [stack/dlr_immutable_add_rm_dedup](delta-io#1538) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1538/files/06fdf84431d7ee9cf693149784b11e4afb242de1..991cdd69f18eb680dbafae3c213093b036cb7a85)] - [stack/dlr_leaf](delta-io#1501) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1501/files/991cdd69f18eb680dbafae3c213093b036cb7a85..1f52bbc6512d7cb0a5e2a68954eb434be7313ef0)] - [stack/dlr_executor](delta-io#1539) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1539/files/1f52bbc6512d7cb0a5e2a68954eb434be7313ef0..9a4f638d613121fe5819e3fef6c03b3e5102ab73)] - [stack/dlr_api](delta-io#1547) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1547/files/9a4f638d613121fe5819e3fef6c03b3e5102ab73..f385656ab1aa1f34caf77fa8518834d0ac0b998c)] - [stack/dlr_scan_state_serde](delta-io#1549) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1549/files/f385656ab1aa1f34caf77fa8518834d0ac0b998c..ac99eeec1f5d19fe76e8589f87cf53a7aa140e54)] --------- This PR adds a CheckpointManifestReader that is responsible for reading both manifest checkpoints and single-part checkpoints. For manifest checkpoints, it collects all the sidecar files that are present and returns them for further processing. * feat: Distributed Log Replay Sequential Phase (delta-io#1502) ## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta-kernel-rs/pull/1502/files) to review incremental changes. - [**stack/dlr_driver**](delta-io#1502) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1502/files)] - [stack/dlr_serialization_helpers](delta-io#1543) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1543/files/8f86b459613c0f018dd0e45473ba3e6e50d127af..0286f1d09006b099e73333c3d2d6a0873ede6cdd)] - [stack/dlr_serde](delta-io#1503) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1503/files/0286f1d09006b099e73333c3d2d6a0873ede6cdd..ee68c851e338a83584b004d919c40eca301e2ee3)] - [stack/dlr_add_rm_dedup](delta-io#1537) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1537/files/ee68c851e338a83584b004d919c40eca301e2ee3..436ce448e2dbf08ceb8f13350cb609ae3f554a7c)] - [stack/dlr_immutable_add_rm_dedup](delta-io#1538) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1538/files/436ce448e2dbf08ceb8f13350cb609ae3f554a7c..b2c2703bef78ef814c0579e82857cd67a2d1364b)] - [stack/dlr_leaf](delta-io#1501) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1501/files/b2c2703bef78ef814c0579e82857cd67a2d1364b..5f04e2059209c09a3ac8f5981d018347829bd8ab)] - [stack/dlr_executor](delta-io#1539) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1539/files/5f04e2059209c09a3ac8f5981d018347829bd8ab..7e2d26a6b063927e8b9a9fa62043ad30c705cc43)] - [stack/dlr_api](delta-io#1547) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1547/files/7e2d26a6b063927e8b9a9fa62043ad30c705cc43..b608475ed1402d8640b89e5948d066e0867585b3)] - [stack/dlr_scan_state_serde](delta-io#1549) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1549/files/b608475ed1402d8640b89e5948d066e0867585b3..c35994fbf457ed95556acc7688456feb81923abf)] --------- ## What changes are proposed in this pull request? This pull request introduces the SequentialPhase for distributed log replay. This phase reads the commit and checkpoint files for a logsegment and performs the log replay processor over the batches. Upon completion, the `finish` method produces `AfterSequential` which has two variants: * `Done`: Indicates that the metadata phase is complete and there is no more work to do. * `Distributed`: Indicates that a distributed phase is required (see the pr at stack/dlr_executor) ## How was this change tested? Ensure that the sequential phase extracts the expected actions for: * commits * single-part checkpoints Ensure that sequential phase produces `AfterSequential::Distributed` in the presence of: * sidecar files in a manifest checkpoint * multi-part checkpoints * feat/bugfix: Passing schema from C, plus example/tests in C (delta-io#1535) ## What changes are proposed in this pull request? * Don't use a macro to generate primitive visitors. `cbindgen` runs before macro expansion, so previously none of those visitors were actually being generated * Add code to `read_table` to allow specifying which columns to select, plus all the associated code to pass that back to kernel * Add c based tests to make sure it's working ## How was this change tested? New tests when running `make test` in `read_table` --------- Co-authored-by: OussamaSaoudi <45303303+OussamaSaoudi@users.noreply.github.com> * feat!: expose mod time in scan metadata callbacks (delta-io#1565) * feat: Support sidecar in inspect-table (delta-io#1566) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-incubator/delta-kernel-rs/blob/main/CONTRIBUTING.md 2. Run `cargo t --all-features --all-targets` to get started testing, and run `cargo fmt`. 3. Ensure you have added or run the appropriate tests for your PR. 4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 5. Be sure to keep the PR description updated to reflect all changes. --> <!-- PR title formatting: This project uses conventional commits: https://www.conventionalcommits.org/ Each PR corresponds to a commit on the `main` branch, with the title of the PR (typically) being used for the commit message on main. In order to ensure proper formatting in the CHANGELOG please ensure your PR title adheres to the conventional commit specification. Examples: - new feature PR: "feat: new API for snapshot.update()" - bugfix PR: "fix: correctly apply DV in read-table example" --> ## What changes are proposed in this pull request? <!-- Please clarify what changes you are proposing and why the changes are needed. The purpose of this section is to outline the changes, why they are needed, and how this PR fixes the issue. If the reason for the change is already explained clearly in an issue, then it does not need to be restated here. 1. If you propose a new API or feature, clarify the use case for a new API or feature. 2. If you fix a bug, you can clarify why it is a bug. --> Support sidecar in inspect-table. Solves the issue delta-io#695. <!-- Uncomment this section if there are any changes affecting public APIs: ### This PR affects the following public APIs If there are breaking changes, please ensure the `breaking-changes` label gets added by CI, and describe why the changes are needed. Note that _new_ public APIs are not considered breaking. --> ## How was this change tested? <!-- Please make sure to add test cases that check the changes thoroughly including negative and positive cases if possible. If it was tested in a way different from regular unit tests, please clarify how you tested, ideally via a reproducible test documented in the PR description. --> Test locally. Used `./kernel/tests/data/v2-checkpoints-parquet-with-sidecars.tar.zst`. The output successfully print sidecar actions: ``` cargo run -p inspect-table -- v2-checkpoints-parquet-with-sidecars actions | head -17 Action 1: Sidecar { path: "00000000000000000006.checkpoint.0000000001.0000000002.76931b15-ead3-480d-b86c-afe55a577fc3.parquet", size_in_bytes: 12461, modification_time: 1739329719752, tags: None, } Action 2: Sidecar { path: "00000000000000000006.checkpoint.0000000002.0000000002.4367b29c-0e87-447f-8e81-9814cc01ad1f.parquet", size_in_bytes: 12109, modification_time: 1739329719746, tags: None, } ``` * fix: force usage of ListedLogFiles::try_new() (delta-io#1562) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-incubator/delta-kernel-rs/blob/main/CONTRIBUTING.md 2. Run `cargo t --all-features --all-targets` to get started testing, and run `cargo fmt`. 3. Ensure you have added or run the appropriate tests for your PR. 4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 5. Be sure to keep the PR description updated to reflect all changes. --> <!-- PR title formatting: This project uses conventional commits: https://www.conventionalcommits.org/ Each PR corresponds to a commit on the `main` branch, with the title of the PR (typically) being used for the commit message on main. In order to ensure proper formatting in the CHANGELOG please ensure your PR title adheres to the conventional commit specification. Examples: - new feature PR: "feat: new API for snapshot.update()" - bugfix PR: "fix: correctly apply DV in read-table example" --> ## What changes are proposed in this pull request? <!-- Please clarify what changes you are proposing and why the changes are needed. The purpose of this section is to outline the changes, why they are needed, and how this PR fixes the issue. If the reason for the change is already explained clearly in an issue, then it does not need to be restated here. 1. If you propose a new API or feature, clarify the use case for a new API or feature. 2. If you fix a bug, you can clarify why it is a bug. --> Change field of `ListedLogFiles` to private to force the usage of `ListedLogFiles::try_new()` Solves the issue delta-io#1143 <!-- Uncomment this section if there are any changes affecting public APIs: ### This PR affects the following public APIs If there are breaking changes, please ensure the `breaking-changes` label gets added by CI, and describe why the changes are needed. Note that _new_ public APIs are not considered breaking. --> ## How was this change tested? Existing tests <!-- Please make sure to add test cases that check the changes thoroughly including negative and positive cases if possible. If it was tested in a way different from regular unit tests, please clarify how you tested, ideally via a reproducible test documented in the PR description. --> * fix: improve parse_json performance by removing line-by-line parsing (delta-io#1561) Co-authored-by: emkornfield <emkornfield@gmail.com> Co-authored-by: Nick Lanham <nick@databricks.com> * sync changelog from `0.18.2` (delta-io#1570) cherry-picks f105333 Co-authored-by: OussamaSaoudi <45303303+OussamaSaoudi@users.noreply.github.com> Co-authored-by: Drake Lin <drakelin18@gmail.com> * feature: short-circuit coalesce evaluation when array has no nulls (delta-io#1568) ## What changes are proposed in this pull request? <!-- Please clarify what changes you are proposing and why the changes are needed. The purpose of this section is to outline the changes, why they are needed, and how this PR fixes the issue. If the reason for the change is already explained clearly in an issue, then it does not need to be restated here. 1. If you propose a new API or feature, clarify the use case for a new API or feature. 2. If you fix a bug, you can clarify why it is a bug. --> Optimizes the coalesce expression evaluation in the default Arrow engine to short-circuit when an evaluated array has no null values. Previously, all expressions were eagerly evaluated before coalescing. - After evaluating each expression, check if `null_count() == 0` - If no nulls exist, validate the result type and return immediately without evaluating remaining expressions - `null_count()` on Arrow arrays is O(1) (cached value), so the check is essentially free. This optimization avoids evaluating potentially expensive expressions when earlier expressions already provide complete (non-null) results. Usecase: - For ParsedStats, we will see coalesces in both read/write - In read: we want to coalesce(parsed_stats, ParseJson(json_stats)) - In write: - ParsedStats: coalesce(parsed_stats, ParseJson(json_stats)) - JsonStats: coalesce(json_stats, ToJson(ParsedStats)) In each of these cases, all of one column will either be Null or non-Null so this optimization actually completely removes the need to evaluate the expensive parseJson/ToJson if unnecessary. Future improvements: - Track each row such that it short-circuits as soon as all rows are filled, even if no single array was 100% non-null - This can be implemented with FilteredEngineData, but that lacks support across the engine <!-- Uncomment this section if there are any changes affecting public APIs: ### This PR affects the following public APIs If there are breaking changes, please ensure the `breaking-changes` label gets added by CI, and describe why the changes are needed. Note that _new_ public APIs are not considered breaking. --> ## How was this change tested? <!-- Please make sure to add test cases that check the changes thoroughly including negative and positive cases if possible. If it was tested in a way different from regular unit tests, please clarify how you tested, ideally via a reproducible test documented in the PR description. --> new and existing unit tests * release 0.19.0 (delta-io#1571) release 0.19.0: [link to rendered changelog](https://github.com/zachschuermann/delta-kernel-rs/blob/a23efae27cf8bd7859a24ad66f57dcb0f6fac851/CHANGELOG.md) * [Test Only] Minor refactor to log_segment tests (delta-io#1581) ## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta-kernel-rs/pull/1581/files) to review incremental changes. - [**stack/log_segment_tests_refactor**](delta-io#1581) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1581/files)] --------- This PR refactors the `create_segment_for` test helper method inside of `log_segment/tests.rs`. Instead of taking in N params, we take in a struct (that contains those N params). This will make adding new params in the future easier and cleaner (e.g. staged catalog commits) * Refactor `ListedLogFiles::try_new` to be more extensible and with default values by using builder pattern (delta-io#1585) ## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta-kernel-rs/pull/1585/files) to review incremental changes. - [**stack/listed_log_files_refactor**](delta-io#1585) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1585/files)] --------- This PR refactors `ListedLogFiles::try_new` to instead be `ListedLogFilesBuilder::build()`. This allows using default values in the `ListedLogFilesBuilder`, which will make creating `ListedLogFiles` easier in the future (e.g. when we add `max_known_published_commit_version`). * Fix: add type validation for `evaluate_expression` (delta-io#1575) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-incubator/delta-kernel-rs/blob/main/CONTRIBUTING.md 2. Run `cargo t --all-features --all-targets` to get started testing, and run `cargo fmt`. 3. Ensure you have added or run the appropriate tests for your PR. 4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 5. Be sure to keep the PR description updated to reflect all changes. --> <!-- PR title formatting: This project uses conventional commits: https://www.conventionalcommits.org/ Each PR corresponds to a commit on the `main` branch, with the title of the PR (typically) being used for the commit message on main. In order to ensure proper formatting in the CHANGELOG please ensure your PR title adheres to the conventional commit specification. Examples: - new feature PR: "feat: new API for snapshot.update()" - bugfix PR: "fix: correctly apply DV in read-table example" --> ## What changes are proposed in this pull request? <!-- Please clarify what changes you are proposing and why the changes are needed. The purpose of this section is to outline the changes, why they are needed, and how this PR fixes the issue. If the reason for the change is already explained clearly in an issue, then it does not need to be restated here. 1. If you propose a new API or feature, clarify the use case for a new API or feature. 2. If you fix a bug, you can clarify why it is a bug. --> Add missing `result_type` validation in `evaluate_expression` for `Literal`, `Column`, and `Binary` branches. This resolves delta-io#1572 <!-- Uncomment this section if there are any changes affecting public APIs: ### This PR affects the following public APIs If there are breaking changes, please ensure the `breaking-changes` label gets added by CI, and describe why the changes are needed. Note that _new_ public APIs are not considered breaking. --> ## How was this change tested? <!-- Please make sure to add test cases that check the changes thoroughly including negative and positive cases if possible. If it was tested in a way different from regular unit tests, please clarify how you tested, ideally via a reproducible test documented in the PR description. --> new and existing unit tests * refactor: move doctest into mods (delta-io#1574) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-incubator/delta-kernel-rs/blob/main/CONTRIBUTING.md 2. Run `cargo t --all-features --all-targets` to get started testing, and run `cargo fmt`. 3. Ensure you have added or run the appropriate tests for your PR. 4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 5. Be sure to keep the PR description updated to reflect all changes. --> <!-- PR title formatting: This project uses conventional commits: https://www.conventionalcommits.org/ Each PR corresponds to a commit on the `main` branch, with the title of the PR (typically) being used for the commit message on main. In order to ensure proper formatting in the CHANGELOG please ensure your PR title adheres to the conventional commit specification. Examples: - new feature PR: "feat: new API for snapshot.update()" - bugfix PR: "fix: correctly apply DV in read-table example" --> ## What changes are proposed in this pull request? <!-- Please clarify what changes you are proposing and why the changes are needed. The purpose of this section is to outline the changes, why they are needed, and how this PR fixes the issue. If the reason for the change is already explained clearly in an issue, then it does not need to be restated here. 1. If you propose a new API or feature, clarify the use case for a new API or feature. 2. If you fix a bug, you can clarify why it is a bug. --> Currently all doctests live inline in `kernel/src/lib.rs`. This PR moves them to `kernel/src/doctests/` with separate files per macro, making it easier to add new macro tests without enlarging `lib.rs`. Related: delta-io#991 <!-- Uncomment this section if there are any changes affecting public APIs: ### This PR affects the following public APIs If there are breaking changes, please ensure the `breaking-changes` label gets added by CI, and describe why the changes are needed. Note that _new_ public APIs are not considered breaking. --> ## How was this change tested? <!-- Please make sure to add test cases that check the changes thoroughly including negative and positive cases if possible. If it was tested in a way different from regular unit tests, please clarify how you tested, ideally via a reproducible test documented in the PR description. --> Ran existing tests * feat: add checkpoint schema discovery for stats_parsed detection (delta-io#1550) ## What changes are proposed in this pull request? <!-- Please clarify what changes you are proposing and why the changes are needed. The purpose of this section is to outline the changes, why they are needed, and how this PR fixes the issue. If the reason for the change is already explained clearly in an issue, then it does not need to be restated here. 1. If you propose a new API or feature, clarify the use case for a new API or feature. 2. If you fix a bug, you can clarify why it is a bug. --> This PR adds infrastructure for discovering checkpoint schemas and detecting whether `stats_parsed` is available for data skipping optimization. 1. Store checkpoint schema in LogSegment (`log_segment.rs`) 2. Add stats_parsed detection (`log_segment.rs`) - During log replay / sidecar visiting, we figure out the schema for the file action files (whether the sidecar parquet files for V2 checkpoints or the V1 checkpoint parquet. ## How was this change tested? <!-- Please make sure to add test cases that check the changes thoroughly including negative and positive cases if possible. If it was tested in a way different from regular unit tests, please clarify how you tested, ideally via a reproducible test documented in the PR description. --> - Existing log segment tests updated * refactor: deny panics in ffi crate (delta-io#1576) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-incubator/delta-kernel-rs/blob/main/CONTRIBUTING.md 2. Run `cargo t --all-features --all-targets` to get started testing, and run `cargo fmt`. 3. Ensure you have added or run the appropriate tests for your PR. 4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 5. Be sure to keep the PR description updated to reflect all changes. --> <!-- PR title formatting: This project uses conventional commits: https://www.conventionalcommits.org/ Each PR corresponds to a commit on the `main` branch, with the title of the PR (typically) being used for the commit message on main. In order to ensure proper formatting in the CHANGELOG please ensure your PR title adheres to the conventional commit specification. Examples: - new feature PR: "feat: new API for snapshot.update()" - bugfix PR: "fix: correctly apply DV in read-table example" --> ## What changes are proposed in this pull request? <!-- Please clarify what changes you are proposing and why the changes are needed. The purpose of this section is to outline the changes, why they are needed, and how this PR fixes the issue. If the reason for the change is already explained clearly in an issue, then it does not need to be restated here. 1. If you propose a new API or feature, clarify the use case for a new API or feature. 2. If you fix a bug, you can clarify why it is a bug. --> 1. Add clippy lints to deny panics in the FFI crate (non-test code) 2. Fix violations by returning errors instead of panicking 3. Allow `expect` in theoretically unreachable paths <!-- Uncomment this section if there are any changes affecting public APIs: ### This PR affects the following public APIs If there are breaking changes, please ensure the `breaking-changes` label gets added by CI, and describe why the changes are needed. Note that _new_ public APIs are not considered breaking. --> ## How was this change tested? <!-- Please make sure to add test cases that check the changes thoroughly including negative and positive cases if possible. If it was tested in a way different from regular unit tests, please clarify how you tested, ideally via a reproducible test documented in the PR description. --> Existing tests in `ffi/examples/read-table` * Remove comments and text from `pull_request_template.md` (delta-io#1589) ## What changes are proposed in this pull request? Move text from `pull_request_template.md` to `CONTRIBUTING.md`. Make PR template file name all-caps. ## How was this change tested? N/A. * feat!: add deletion vector APIs to transaction (delta-io#1430) Adds the ability to update deletion vectors on delta tables. Co-authored-by: Drake Lin <drakelin18@gmail.com> Co-authored-by: Nick Lanham <nicklan@users.noreply.github.com> * feat: Add doctests for `IntoEngineData` derive macro (delta-io#1580) <!-- PR title formatting: This project uses conventional commits: https://www.conventionalcommits.org/ Each PR corresponds to a commit on the `main` branch, with the title of the PR (typically) being used for the commit message on main. In order to ensure proper formatting in the CHANGELOG please ensure your PR title adheres to the conventional commit specification. Examples: - new feature PR: "feat: new API for snapshot.update()" - bugfix PR: "fix: correctly apply DV in read-table example" --> ## What changes are proposed in this pull request? <!-- Please clarify what changes you are proposing and why the changes are needed. The purpose of this section is to outline the changes, why they are needed, and how this PR fixes the issue. If the reason for the change is already explained clearly in an issue, then it does not need to be restated here. 1. If you propose a new API or feature, clarify the use case for a new API or feature. 2. If you fix a bug, you can clarify why it is a bug. --> This PR: - Changes the `IntoEngineData` macro to use `delta_kernel::` paths - Adds doctests for the derive macro Related: delta-io#991 <!-- Uncomment this section if there are any changes affecting public APIs: ### This PR affects the following public APIs If there are breaking changes, please ensure the `breaking-changes` label gets added by CI, and describe why the changes are needed. Note that _new_ public APIs are not considered breaking. --> ## How was this change tested? <!-- Please make sure to add test cases that check the changes thoroughly including negative and positive cases if possible. If it was tested in a way different from regular unit tests, please clarify how you tested, ideally via a reproducible test documented in the PR description. --> Added tests. * feat: Create `DefaultEngineBuilder` to build `DefaultEngine` (delta-io#1582) ## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta-kernel-rs/pull/1582/files) to review incremental changes. - [**stack/engine-builder**](delta-io#1582) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1582/files)] - [stack/remove-constructor](delta-io#1583) [[Files changed](https://github.com/delta-io/delta-kernel-rs/pull/1583/files/29e082a58b9382149285f30351d53468d6ed49dc..0026634ac5bf5f546d45a4d25dd0f46cb8d39d1b)] --------- ## What changes are proposed in this pull request? <!-- Please clarify what changes you are proposing and why the changes are needed. The purpose of this section is to outline the changes, why they are needed, and how this PR fixes the issue. If the reason for the change is already explained clearly in an issue, then it does not need to be restated here. 1. If you propose a new API or feature, clarify the use case for a new API or feature. 2. If you fix a bug, you can clarify why it is a bug. --> Add `DefaultEngineBuilder` for more flexible engine construction and align with the builder pattern used elsewhere in this repo. ## Detail changes - Add generic `DefaultEngineBuilder<E>` with builder pattern - Support `with_task_executor()` to use custom executors - Support `with_metrics_reporter()` for metrics collection - Add `DefaultEngine::builder()` convenience method <!-- Uncomment this section if there are any changes affecting public APIs: ### This PR affects the following public APIs If there are breaking changes, please ensure the `breaking-changes` label gets added by CI, and describe why the changes are needed. Note that _new_ public APIs are not considered breaking. --> ## How was this change tested? <!-- Please make sure to add test cases that check the changes thoroughly including negative and positive cases if possible. If it was tested in a way different from regular unit tests, please clarify how you tested, ideally via a reproducible test documented in the PR description. --> Added tests. * refactor: extract shared HTTP utilities to http.rs (delta-io#1590) ## What changes are proposed in this pull request? Extract `build_http_client`, `execute_with_retry`, and `handle_response` from `UCClient` into a shared `http.rs` module. This enables reuse by other client implementations (and is used in the child stacked PR delta-io#1590) ## How was this change tested? `cargo build -p uc-client -p uc-catalog` * Fix conflicts * Fix dangling conflicts * Fix the tests --------- Co-authored-by: OussamaSaoudi <45303303+OussamaSaoudi@users.noreply.github.com> Co-authored-by: Nick Lanham <nicklan@users.noreply.github.com> Co-authored-by: dengsh12 <62536982+dengsh12@users.noreply.github.com> Co-authored-by: R. Tyler Croy <rtyler@brokenco.de> Co-authored-by: emkornfield <emkornfield@gmail.com> Co-authored-by: Nick Lanham <nick@databricks.com> Co-authored-by: Zach Schuermann <zachary.zvs@gmail.com> Co-authored-by: Drake Lin <drakelin18@gmail.com> Co-authored-by: Scott Sandre <scott.sandre@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🥞 Stacked PR
Use this link to review incremental changes.