Fix filter pruning for string columns with numeric literals - #7523
Open
sivakumar-mahalingam wants to merge 1 commit into
Open
Fix filter pruning for string columns with numeric literals#7523sivakumar-mahalingam wants to merge 1 commit into
sivakumar-mahalingam wants to merge 1 commit into
Conversation
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which Delta project/connector is this regarding?
Description
This PR fixes filter pruning for predicates comparing string columns with numeric literals.
When a string column contains numeric-looking values, a predicate using a numeric literal can prevent Delta from effectively using file-level statistics for data skipping.
This change updates the filter/data-skipping logic to correctly handle these predicates and adds regression test coverage.
Resolves #7136
How was this patch tested?
Added regression test coverage in
DataSkippingDeltaTestsfor filtering string columns with numeric literals.The test verifies the expected filter pruning behavior.
Does this PR introduce any user-facing changes?
Yes. Queries using numeric literals against string columns can now benefit from Delta data skipping where applicable.