[PROTOCOL RFC] User-Defined Types (UDT) - #7560
Conversation
Address review: the two field combinations are Spark's usage, not a protocol-required set. Reframe the annotation as an open, engine-defined map; describe the stored-value <-> rich-object conversion as a contract (no Spark method names); keep class/pyClass/serializedClass and VectorUDT as an example; note other engines use their own members (or none).
|
updated the PR |
…s the engine's choice
|
|
||
| A UDT column is not eligible for `minValues` / `maxValues`, and readers must not perform min/max data skipping over one: `UserDefinedType` is not a skipping-eligible type, so no min/max is recorded, and its `sqlType` (even if it contains orderable leaves) is not descended into for skipping. A per-column `nullCount` is recorded for a UDT column, with the UDT treated as a single statistics leaf: the null count is for the column as a whole, and statistics do not descend into the `sqlType`. Per-file `numRecords` is unaffected. | ||
|
|
||
| ## Partitioning and Clustering |
There was a problem hiding this comment.
what about generated columns, column defaults, and all other features?
There was a problem hiding this comment.
Handled with a general rule now: "a UDT column is subject to the same protocol rules as a column of its sqlType, except as stated." The stated exceptions (validated against Delta): partition, clustering, and identity columns and IcebergCompat reject a UDT, and type widening does not apply. Generated columns and column defaults are intentionally left to the general rule and not called out: Delta-Spark does not restrict them for a UDT today, and a UDT-typed generated column or default is not expressible anyway.
Let me know if you think otherwise
| ## IcebergCompat | ||
|
|
||
| Apache Iceberg has no UDT concept. When any of the `icebergCompatV1`, `icebergCompatV2`, or `icebergCompatV3` features is enabled, a writer must reject a schema containing a UDT column. |
There was a problem hiding this comment.
we need to update the corresponding sections when merging this into the protocol.
There was a problem hiding this comment.
Agreed. Added a reviewer note: on adoption, update the corresponding PROTOCOL.md sections (statistics, type widening, column mapping, IcebergCompat) to reference UDT, rather than relying only on this standalone section.
(keeping this open to not lose track of this since the sections in the RFC have been collapsed)
…ons; forbid nested udt; fix stats wording
…m reader physical-ops list
2 participants
Which Delta project/connector is this regarding?
Description
Adds the protocol RFC for User-Defined Type (
udt) support (see #7559) to the RFC folder. Documents the disposition of Spark'sudtcolumns: read the column as its physicalsqlType, and preserve the annotation as an opaque string-to-string map, without gating it behind a table feature (grandfathered, likevoidand interval types). A prototype kernel implementation is in delta-io/delta-kernel-rs#2815.How was this patch tested?
N/A (documentation-only RFC).
Does this PR introduce any user-facing changes?
No.