what's the point of parquet.avro.write-old-list-structure=false for hudi->iceberg

[parisni]

Hey,
AFAIK hudi default for parquet.avro.write-old-list-structure is true. In the hudi->iceberg limitation it's mentionned:

Be sure to enable parquet.avro.write-old-list-structure=false for proper compatibility with lists when syncing from Hudi to Iceberg.

I did some tests, and i could't find any issues with array<struct> hudi parquet table, read with both spark and athena.

Can someone be more specific on this issue and the risks ?

Thanks

[the-other-tim-brown]

@parisni the Iceberg spec notes that lists must use the 3-level representation. By default, Hudi was producing 2-level lists in the parquet files at the time that the project was created. We saw that this meant that some engines were unable to read the data at the time.

[parisni]

We saw that this meant that some engines were unable to read the data at the time.

Thanks . both spark and athena are fine with 2 lvl. Didn't tryied other engine.

[the-other-tim-brown]

Did you confirm that the file is 2-level? If I remember correctly, different paths in Hudi may produce different list structure depending on the underlying writer used (avro vs spark native).