[Feature Request][Spark / Protocol] Add validation from `replaceWhere` to other write modes (append)

[FrankPortman]

Feature request

Which Delta project/connector is this regarding?

• Spark
• Standalone
• Flink
• Kernel
• Other (fill in here)

Overview

Feature request to add the same validation behavior from replaceWhere + "mode(overwrite)" to other write modes (append).

Motivation

The declarative "verify and atomically commit only if within expectation" semantics of replaceWhere are very useful, efficient (otherwise you'd have to roll your own staging concept, or risk computing a dataframe 2x just to validate), and would be useful for any type of write, including Append.

Willingness to contribute

The Delta Lake Community encourages new feature contributions. Would you or another member of your organization be willing to contribute an implementation of this feature?

• Yes. I can contribute this feature independently.
• Yes. I would be willing to contribute this feature with guidance from the Delta Lake community.
• No. I cannot contribute this feature at this time.

[AnudeepKonaboina]

Hello @FrankPortman ! I will work on this!

Tried to repro this using the below code

Code

import org.apache.spark.sql.functions._
val tbl = s"default.rw_5764_${System.currentTimeMillis()}"

// Create a partitioned Delta table
spark.sql(s"DROP TABLE IF EXISTS $tbl")
spark.sql(s"CREATE TABLE $tbl (id INT, ds STRING) USING delta PARTITIONED BY (ds)")

spark.range(0, 3).select(col("id").cast("int"), lit("2026-01-01").as("ds"))
  .write.format("delta").mode("append").saveAsTable(tbl)

// Incoming data contains a bad row outside the predicate
val incoming = Seq(
  (10, "2026-01-01"),
  (11, "2099-01-01")  // violates the replaceWhere predicate
).toDF("id","ds")

// Overwrite + replaceWhere will FAIL (validation)
try {
  incoming.write.format("delta")
    .mode("append")
    .option("replaceWhere", "ds = '2026-01-01'")
    .saveAsTable(tbl)
} catch {
  case e: Throwable =>
    println(e.getMessage)
}

Output:

With Overwrite

06:20:17.099 [task-result-getter-1] ERROR org.apache.spark.scheduler.TaskSetManager - Task 1 in stage 15.0 failed 4 times; aborting job
06:20:17.105 [main] ERROR org.apache.spark.sql.delta.files.DeltaFileFormatWriter - Aborting job e7871693-bbea-4576-bda1-b2ba04bb0e6f
org.apache.spark.sql.delta.schema.DeltaInvariantViolationException: [DELTA_VIOLATE_CONSTRAINT_WITH_VALUES] CHECK constraint EXPRESSION(('ds = 2026-01-01)) (ds = '2026-01-01') violated by row with values:
 - ds : 2099-01-01
	at org.apache.spark.sql.delta.schema.DeltaInvariantViolationException$.getConstraintViolationWithValuesException(InvariantViolationException.scala:78) ~[delta-spark-assembly-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
	at org.apache.spark.sql.delta.schema.DeltaInvariantViolationException$.apply(InvariantViolationException.scala:106) ~[delta-spark-assembly-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
	at org.apache.spark.sql.delta.schema.DeltaInvariantViolationException$.apply(InvariantViolationException.scala:117) ~[delta-spark-assembly-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
	at org.apache.spark.sql.delta.schema.DeltaInvariantViolationException.apply(InvariantViolationException.scala) ~[delta-spark-assembly-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source) ~[?:?]
	at org.apache.spark.sql.delta.constraints.DeltaInvariantCheckerExec.$anonfun$doExecute$2(DeltaInvariantCheckerExec.scala:97) ~[delta-spark-assembly-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
	at scala.collection.Iterator$$anon$9.next(Iterator.scala:584) ~[scala-library-2.13.16.jar:?]
	at org.apache.spark.sql.execution.UnsafeExternalRowSorter.sort(UnsafeExternalRowSorter.java:226) ~[spark-sql_2.13-4.0.0.jar:4.0.0]
	at org.apache.spark.sql.execution.SortExec.$anonfun$doExecute$1(SortExec.scala:119) ~[spark-sql_2.13-4.0.0.jar:4.0.0]
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:901) ~[spark-core_2.13-4.0.0.jar:4.0.0]
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:901) ~[spark-core_2.13-4.0.0.jar:4.0.0]
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) ~[spark-core_2.13-4.0.0.jar:4.0.0]
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:374) ~[spark-core_2.13-4.0.0.jar:4.0.0]
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:338) ~[spark-core_2.13-4.0.0.jar:4.0.0]
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93) ~[spark-core_2.13-4.0.0.jar:4.0.0]
	at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:171) ~[spark-core_2.13-4.0.0.jar:4.0.0]
	at org.apache.spark.scheduler.Task.run(Task.scala:147) ~[spark-core_2.13-4.0.0.jar:4.0.0]

With Append

+----------+---+                                                                
|ds        |cnt|
+----------+---+
|2026-01-01|1  |
|2099-01-01|1  |
+----------+---+

But i believe this would be a breaking change for anyone currently passing it accidentally and relying on it being ignored. Also there might be cases the incoming partition of data might be correct but we might error out when written in append mode .Can you give me a your real time use-case where you will need replaceWhere with append?

[FrankPortman]

Hi @AnudeepKonaboina - due to both backwards compatability concerns AND semantics, I would probably not call this option replaceWhere (especially for the append use case).

Can you give me a your real time use-case where you will need replaceWhere with append?

The motivation is pretty straightforward for me - I want to use the atomic write-time validation available in Delta, for any arbitrary write, and not just a selective overwrite. But again, I do think the name of the option should be changed for this new feature I am talking about.

[AnudeepKonaboina]

Hey @FrankPortman !

I have a proposal and will implement it if the reviewers approve it:

IMO , the best way would be is to Introduce a new writer option (optional), so that the current replaceWhere semantics is kept unchanged .

Example:

option("validateWhere", "<predicate>")

and implement:

If validateWhere is set:

• Always enforce it as a row-level check and abort the write if any row violates it.
• Works for append and will be compatible with overwrite as well.

Lmk if you have any other thoughts?

[FrankPortman]

Sounds great to me. With overwrite it has same behavior as replaceWhere, but otherwise it's a brand new option.