[Spark] Stop metadata cleanup listing after retention checkpoint

[sivakumar-mahalingam]

Which Delta project/connector is this regarding?

• Spark
• Standalone
• Flink
• Kernel
• All JVM projects relying on the Azure/Hadoop LogStore implementation

Description

Related to #6531.

Metadata cleanup continued consuming the _delta_log listing after BufferingLogDeletionIterator processed the complete checkpoint beyond the maximum deletable version. At that point, all subsequent files are version-ineligible for deletion.

This change records that boundary and stops consuming the underlying iterator.

The stopping condition is based on ordered log versions. It does not assume that file modification timestamps are monotonic.

Correctness

The existing timestamp-adjustment and checkpoint-buffering behavior remains unchanged. In particular:

• Files must still satisfy both the timestamp and version retention limits.
• Buffered commits are released only after a complete checkpoint is found.
• A multipart checkpoint becomes a stopping boundary only after all its parts have been observed.
• An incomplete multipart checkpoint does not release buffered commits or establish an early-exit boundary.
• Non-monotonic modification timestamps continue to use the existing timestamp adjustment behavior.
• Single-part and V2 checkpoints continue through the existing checkpoint path.
• Checksum, coordinated commit, unbackfilled commit, and sidecar cleanup logic is unchanged.

An incomplete multipart checkpoint does not establish an early-exit boundary.

Performance

A deterministic regression test supplies more than 10,000 newer entries after the retention checkpoint.

For the nothing-expired and few-files-expired cases, the iterator consumes only the entries required to reach the complete checkpoint instead of consuming the entire synthetic listing.

Hadoop and Azure follow-up

This change reduces work when LogStore.listFrom provides a genuinely lazy or paginated iterator.

The Hadoop and Azure implementations currently call listStatus and materialize/sort the directory contents before returning their iterator. Therefore, this patch does not eliminate the provider-level listing cost for those implementations.

Making Hadoop/Azure listing lazy requires preserving the public LogStore.listFrom contract that results are lexicographically sorted. Generic Hadoop RemoteIterator APIs do not provide that ordering guarantee. This broader provider/API design is left.

For this reason, this PR is related to #6531 but does not close it.

Tests

The following focused tests passed:

• DeltaTimeTravelSuite filtered to BufferingLogDeletionIterator
  • 5 tests passed
  • 0 tests failed
• Existing incomplete multipart checkpoint cleanup test

A broader DeltaRetentionSuite run completed with 10 passing and 5 failing tests. The failures occurred during Windows table_changes_by_path validation after the relevant cleanup assertions:

Does this PR introduce any user-facing changes?

No. Metadata retention and deletion semantics are unchanged. The change only avoids consuming log-listing entries that cannot affect the cleanup result.