Use table schema when reading Iceberg branches

[stefan-toubia]

Description

FOR VERSION AS OF '<ref>' resolves a named ref to its head snapshot and then reads with that snapshot's schema, which is the tag rule. Iceberg defines branch reads differently: a branch is a mutable reference, so it is read with the table's current schema, while a tag is read with the schema of the snapshot it points to (Schema selection with branches and tags). As a result, any schema evolution since the branch's head commit is not reflected when reading the branch (including FOR VERSION AS OF 'main'), and Trino disagrees with Spark and Flink on the same query. See #30970 for the full analysis and worked examples.

The schema is chosen twice, once when the table handle is built and again when Iceberg binds the scan, so the fix has two parts.

getTableHandle resolves every version to a snapshot ID as before, and when the version is a ref name that resolves to a branch, it also keeps the branch name. A branch reads with table.schema(); a tag, snapshot ID, or timestamp keeps schemaFor(table, snapshotId). The branch name is carried on the table handle for the scan.

Both scan sites build the scan through IcebergUtil.snapshotScan. Iceberg treats useSnapshot(id) as time travel and binds the scan, its filters, and its partition specs to that snapshot's schema, so a column added after the branch head cannot bind there. useRef(branch) binds table.schema() instead, so a non-main branch scans by ref.

A ref scan resolves the head from the current metadata rather than from the handle. snapshotScan therefore checks that the ref still points at the handle's snapshot and fails with TRANSACTION_CONFLICT if the branch moved, rather than silently reading a different snapshot or schema. Within a query the catalog serves analysis and split planning the same cached metadata, so the check should not fire in practice; it guards that invariant.

main and every other version keep useSnapshot. useRef("main") returns an unpinned scan, and at the current snapshot Iceberg skips the spec rebinding, so FOR VERSION AS OF 'main' behaves exactly like the unversioned read.

The split manager's projection is built from scan.schema() so that field IDs and column-stats names bind against the same schema the scan uses.

Tests: testReadingSnapshotReferenceAfterSchemaEvolution covers rename, name reuse, drop, and add, with branch and main reads seeing the table schema and tag and snapshot-ID reads keeping the snapshot schema. testReadingBranchReferenceWithDeleteFilesAfterSchemaEvolution filters on a column added after a branch head that carries delete files. testReadingSnapshotReferenceAfterPartitionEvolution documents that the branch/tag distinction is specific to schema selection, since readers resolve partition specs per manifest.

Additional context and related issues

• Fixes Iceberg FOR VERSION AS OF '<branch>' reads with the branch head snapshot's schema instead of the table's current schema #30970

Note for reviewers: this changes visible results for FOR VERSION AS OF '<branch>' when the table schema has evolved since the branch's head snapshot: the query returns the table's columns instead of the snapshot's. The old behavior contradicts Iceberg's documented semantics and other engines, so this is treated as a bug fix.

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(x) Release notes are required, with the following suggested text:

## Iceberg
* Fix `FOR VERSION AS OF` on a branch reading with the schema of the branch's head snapshot instead of the table's current schema. ({issue}`30970`)

[stefan-toubia]

Thanks @ebyhr, I've updated the tests

[stefan-toubia]

@ebyhr a heads up before I push the next fixup. While testing the branch management work that stacks on this PR, I hit a case this version does not handle: a filter on a column added after the branch head fails when the branch head carries delete files. Picking the table schema in getTableHandle is not enough on its own. The split manager still builds the scan with useSnapshot(id), which Iceberg treats as time travel and binds filters to the snapshot's schema, so the added column cannot bind there.

The fix I am finishing up carries the branch name on the table handle and scans a non-main branch with useRef(branch). It also builds the split manager's projection from the scan's schema and adds a test for the delete-files case. I will update the description to match when it lands, temporarily converting this to a draft until then. Should have it out soon.