[Spark] Validate against only existing commit config in UniForm enforcement on v1 saveAsTable overwrite - #7538
Open
amogh-jahagirdar wants to merge 1 commit into
Conversation
…Table overwrite A v1 saveAsTable overwrite is data-only: it skips replaceMetadataIfNecessary, so the committed configuration is the snapshot's. UniForm dependency enforcement instead validated snapshot ++ deltaWriter.configuration, which lets session-default properties the writer picks up (e.g. delta.enableDeletionVectors=true) override the snapshot. On a UniForm table with deletion vectors disabled but the deletionVectors feature still in the protocol, the overwrite failed with DELETION_VECTORS_SHOULD_BE_DISABLED for a value that is never committed. Pass the snapshot configuration -- what the overwrite actually commits -- to enforcement.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which Delta project/connector is this regarding?
Description
A v1
saveAsTableoverwrite on a UniForm (IcebergCompat) table can fail withDELETION_VECTORS_SHOULD_BE_DISABLEDeven when deletion vectors are already disabled on the table.The UniForm dependency check in
CreateDeltaTableCommandvalidatesdeltaWriter.configuration.Currently, we validate against
snapshot.metadata.configuration ++ deltaWriter.configurationafter #7187 but this was to remain compatible with the previous check which just did deltaWriter.configuration. Including deltaWRiter.configuration in any case is incorrect though.For a v1
saveAsTableoverwrite that is the wrong config to check. This overwrite is data only: it skips
replaceMetadataIfNecessary, does not change table properties, and commits the snapshot's config.deltaWriter.configurationnever persists here, and it can carry session default properties thewrite never set, for example
spark.databricks.delta.properties.defaults.enableDeletionVectors=true.When the table still has the
deletionVectorsfeature in its protocol (the state after you disableDVs and enable UniForm), the check reads deletion vectors as enabled and fails the write, even though
that value is never committed.
The fix validates the snapshot config, which is what the overwrite commits. The snapshot config
already includes Delta-log-only properties like
delta.enableIcebergCompatV3, so the check stillsees the full committed config.
How was this patch tested?
New unit test in
UniversalFormatMiscSuiteBase. It creates a UniForm table that still supports thedeletionVectorsfeature with deletion vectors disabled, setsenableDeletionVectors=trueas asession default, and runs a v1
saveAsTableoverwrite. The write succeeds and leaves deletionvectors disabled. Without the change the same test fails with
DELETION_VECTORS_SHOULD_BE_DISABLED.Does this PR introduce any user-facing changes?
No