Iceberg: support variant column type for object, array, and multi-type fields #4466

Let's add a materialize-iceberg task-level option to use the Iceberg variant column type for object, array, and multi-type fields.

Current Landscape

Parquet Variant

The Parquet variant binary encoding originated in Apache Spark's VARIANT type and was standardized in the Apache Parquet format spec (V1) in 2024. It provides the on-disk representation that Iceberg v3 adopts directly.

Iceberg v3

Iceberg v3 adds the variant semi-structured type, using the Parquet Variant V1 binary encoding (metadata + value fields). Variant supports richer primitives than JSON (date, timestamp, timestamptz, binary, decimal), nested arrays and objects, and disallows bucket/identity partitioning.

Flow

Flow documents are JSON, schematized via JSON Schema. Today, materialize-iceberg maps any field that isn't pinned to a single primitive (open type: object, oneOf/anyOf, multi-type, unconstrained) to a JSON-as-text column. Variant would preserve the semi-structured shape natively and avoid bespoke string parsing downstream.

Notes

Reader compatibility

Variant requires Iceberg v3 readers. Validate writes with a DuckDB-based unit test covering nested objects, arrays, mixed-type fields, and nulls.

Schema evolution

No type migration is supported.