Skip to content

[4.3][Spark] Block unsupported maintenance on catalog-managed tables - #7527

Open
timothyw553 wants to merge 3 commits into
delta-io:branch-4.3from
timothyw553:codex/block-catalog-managed-metadata-cleanup-4.3
Open

[4.3][Spark] Block unsupported maintenance on catalog-managed tables#7527
timothyw553 wants to merge 3 commits into
delta-io:branch-4.3from
timothyw553:codex/block-catalog-managed-metadata-cleanup-4.3

Conversation

@timothyw553

@timothyw553 timothyw553 commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator

Backport of #7525 to branch-4.3.

Which Delta project/connector is this regarding?

  • Spark
  • Standalone
  • Flink
  • Kernel
  • Other (fill in here)

Description

Catalog-managed tables must not run maintenance operations without catalog permission. This patch adds checks at these entry points:

  • MetadataCleanup.doLogCleanup: Skip automatic expired-log cleanup after a checkpoint.
  • MetadataCleanup.cleanUpExpiredLogs: Reject a direct metadata-cleanup call.
  • AutoCompact.run: Skip automatic file compaction after a write.
  • WriteIntoDelta: Reject rearrange-only writes, which use dataChange=false.
  • RemoveColumnMappingCommand.run: Reject column-mapping removal because it rewrites data files.

Checkpoint creation and normal writes remain enabled. Tables that are not catalog-managed keep their current behavior.

How was this patch tested?

These focused tests passed:

build/sbt "spark/testOnly org.apache.spark.sql.delta.DeltaRetentionSuite -- -z catalog-managed"
build/sbt "spark/testOnly org.apache.spark.sql.delta.AutoCompactExecutionSuite -- -z catalog-managed"
build/sbt "spark/testOnly org.apache.spark.sql.delta.DeltaSuite org.apache.spark.sql.delta.columnmapping.RemoveColumnMappingSuite -- -z catalog-managed"

Does this PR introduce any user-facing changes?

Yes. Delta Spark no longer runs these unsupported maintenance operations on a catalog-managed table.

Signed-off-by: Timothy Wang <timothy.art@gmail.com>
(cherry picked from commit da24946)
Signed-off-by: Timothy Wang <timothy.art@gmail.com>
(cherry picked from commit 966cb2ed164c462c4d073c277fa73d9fe0960d10)
@timothyw553 timothyw553 changed the title [4.3][Spark] Block metadata cleanup on catalog-managed tables [4.3][Spark] Block unsupported automatic maintenance on catalog-managed tables Aug 25, 2026
Signed-off-by: Timothy Wang <timothy.art@gmail.com>
(cherry picked from commit 427b490)
@timothyw553 timothyw553 changed the title [4.3][Spark] Block unsupported automatic maintenance on catalog-managed tables [4.3][Spark] Block unsupported maintenance on catalog-managed tables Aug 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant