Skip to content

Exclude delete files from Iceberg $partitions - #30947

Open
ricardojr-centerbase wants to merge 2 commits into
trinodb:masterfrom
ricardojr-centerbase:iceberg-partitions-exclude-delete-files
Open

Exclude delete files from Iceberg $partitions#30947
ricardojr-centerbase wants to merge 2 commits into
trinodb:masterfrom
ricardojr-centerbase:iceberg-partitions-exclude-delete-files

Conversation

@ricardojr-centerbase

Copy link
Copy Markdown

Description

The $partitions metadata table reports delete files as if they were data files
on merge-on-read tables.

PartitionsView builds a view that aggregates over the $files table, which has
one row per content file — data files, position delete files and equality delete
files alike. The generated SQL carries no predicate on the content column, so
record_count, file_count and total_size all count delete files:

SELECT partition, SUM(record_count) AS record_count, COUNT(*) AS file_count, SUM(file_size_in_bytes) AS total_size, ...
FROM "catalog"."schema"."table$files"
GROUP BY 1

On a partitioned copy of tpch.tiny.nation with one position delete file and one
equality delete file, SELECT sum(record_count), sum(file_count) FROM "…$partitions"
returns (27, 7) where the table holds 25 rows in 5 data files.

The data column is affected as well: its min, max and null_count values come
from lower_bounds, upper_bounds and null_value_counts of every row in $files,
and equality delete files carry bounds keyed by the field ids of the data schema, so
they silently widen the reported ranges.

This is a regression from #28997, which replaced PartitionsTable with this view.
The removed PartitionsTable read DataFile dataFile = fileScanTask.file() and only
ever saw data files.

Two commits:

  1. Exclude delete files — restores the documented semantics of record_count,
    file_count, total_size and data. This commit stands alone and can be merged
    on its own if you would rather not take the second one.
  2. Add delete file metrics — adds position_delete_record_count,
    position_delete_file_count, equality_delete_record_count and
    equality_delete_file_count, the same columns Iceberg exposes on its own
    partitions metadata table, so the information excluded by the first commit is
    still reachable. Aggregates move from a WHERE predicate to a per-content
    FILTER clause, which also means a partition holding only delete files is listed
    again, with zero data files — again matching Iceberg.

Additional context and related issues

  • The documented column descriptions ("The number of records in the partition") already
    describe the corrected behavior; the docs are updated to say so explicitly.
  • $all_manifests and $manifests disagree about the same manifest today for a related
    reason. Not addressed here.
  • Column order places the four new columns after total_size and before data, matching
    Iceberg's ordering and keeping the Trino-specific data column last.
  • Testing: BaseIcebergSystemTables.testPartitionsTableWithDeleteFiles writes both a
    position delete file and an equality delete file, then asserts $partitions agrees with
    $files WHERE content = 0. It fails on master with (27, 7) against the expected (25, 5).

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(x) Release notes are required, with the following suggested text:

## Iceberg connector
* Fix `record_count`, `file_count`, `total_size` and `data` in the `$partitions` metadata
  table counting delete files as data files on merge-on-read tables. ({issue}`ISSUE`)
* Add `position_delete_record_count`, `position_delete_file_count`,
  `equality_delete_record_count` and `equality_delete_file_count` columns to the
  `$partitions` metadata table. ({issue}`ISSUE`)

ricardojr37 and others added 2 commits August 30, 2026 12:21
The $partitions view aggregates over the $files table, which has one row per
content file. Since the view had no predicate on the content column, position
delete files and equality delete files were counted as data files, inflating
record_count, file_count and total_size on merge-on-read tables. Bounds carried
by equality delete files also polluted the min/max and null_count values of the
data column, because those bounds are keyed by the field ids of the data schema.

Restrict the view to data files, matching both the documented semantics of the
columns and the behavior before the view replaced PartitionsTable.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Report position and equality delete files in dedicated columns, matching the
columns Iceberg exposes on its own partitions metadata table. Aggregates are
now scoped with a FILTER clause per content type rather than a predicate on the
whole view, so a partition that only holds delete files is listed again, with
zero data files.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added docs iceberg Iceberg connector labels Aug 30, 2026
@github-actions

Copy link
Copy Markdown

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

@findepi
findepi requested a review from ebyhr August 31, 2026 14:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs iceberg Iceberg connector

Development

Successfully merging this pull request may close these issues.

2 participants