Exclude delete files from Iceberg $partitions - #30947
Open
ricardojr-centerbase wants to merge 2 commits into
Open
Exclude delete files from Iceberg $partitions#30947ricardojr-centerbase wants to merge 2 commits into
ricardojr-centerbase wants to merge 2 commits into
Conversation
The $partitions view aggregates over the $files table, which has one row per content file. Since the view had no predicate on the content column, position delete files and equality delete files were counted as data files, inflating record_count, file_count and total_size on merge-on-read tables. Bounds carried by equality delete files also polluted the min/max and null_count values of the data column, because those bounds are keyed by the field ids of the data schema. Restrict the view to data files, matching both the documented semantics of the columns and the behavior before the view replaced PartitionsTable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Report position and equality delete files in dedicated columns, matching the columns Iceberg exposes on its own partitions metadata table. Aggregates are now scoped with a FILTER clause per content type rather than a predicate on the whole view, so a partition that only holds delete files is listed again, with zero data files. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The
$partitionsmetadata table reports delete files as if they were data fileson merge-on-read tables.
PartitionsViewbuilds a view that aggregates over the$filestable, which hasone row per content file — data files, position delete files and equality delete
files alike. The generated SQL carries no predicate on the
contentcolumn, sorecord_count,file_countandtotal_sizeall count delete files:On a partitioned copy of
tpch.tiny.nationwith one position delete file and oneequality delete file,
SELECT sum(record_count), sum(file_count) FROM "…$partitions"returns
(27, 7)where the table holds 25 rows in 5 data files.The
datacolumn is affected as well: itsmin,maxandnull_countvalues comefrom
lower_bounds,upper_boundsandnull_value_countsof every row in$files,and equality delete files carry bounds keyed by the field ids of the data schema, so
they silently widen the reported ranges.
This is a regression from #28997, which replaced
PartitionsTablewith this view.The removed
PartitionsTablereadDataFile dataFile = fileScanTask.file()and onlyever saw data files.
Two commits:
record_count,file_count,total_sizeanddata. This commit stands alone and can be mergedon its own if you would rather not take the second one.
position_delete_record_count,position_delete_file_count,equality_delete_record_countandequality_delete_file_count, the same columns Iceberg exposes on its ownpartitions metadata table, so the information excluded by the first commit is
still reachable. Aggregates move from a
WHEREpredicate to a per-contentFILTERclause, which also means a partition holding only delete files is listedagain, with zero data files — again matching Iceberg.
Additional context and related issues
describe the corrected behavior; the docs are updated to say so explicitly.
$all_manifestsand$manifestsdisagree about the same manifest today for a relatedreason. Not addressed here.
total_sizeand beforedata, matchingIceberg's ordering and keeping the Trino-specific
datacolumn last.BaseIcebergSystemTables.testPartitionsTableWithDeleteFileswrites both aposition delete file and an equality delete file, then asserts
$partitionsagrees with$files WHERE content = 0. It fails on master with(27, 7)against the expected(25, 5).Release notes
( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(x) Release notes are required, with the following suggested text: