[yaml] Iceberg add files yaml transform and test by derrickaw · Pull Request #37938 · apache/beam

derrickaw · 2026-03-24T19:18:32Z

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

gemini-code-assist · 2026-03-24T19:18:45Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances Apache Beam's capabilities by integrating a new YAML-based transform for managing Iceberg tables. It provides a streamlined way to add data files to an existing Iceberg table, which is crucial for data ingestion and maintenance workflows. The changes include the definition of the new transform and a comprehensive test to ensure its proper operation.

Highlights

New Iceberg Transform: Introduced the IcebergAddFiles YAML transform, enabling users to add files to an Iceberg table directly through Beam YAML pipelines.
Test Coverage: Added a dedicated YAML test file to validate the functionality of the new IcebergAddFiles transform, ensuring it correctly processes and adds files to an Iceberg table.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

derrickaw · 2026-03-25T16:58:38Z

Run Python PreCommit 3.13

derrickaw · 2026-03-25T17:11:09Z

Run Portable_Python PreCommit 3.10

derrickaw · 2026-03-25T17:11:34Z

Run PythonDocker PreCommit 3.13

tarun-google · 2026-03-25T17:44:16Z

sdks/python/apache_beam/yaml/standard_io.yaml

+    'IcebergAddFiles': 'IcebergAddFiles'
+  config:
+    mappings:
+      'IcebergAddFiles':


I think we have missed partition_fields and table_properties supported by AddFilesSchemaTransform

ahmedabu98 · 2026-03-25T17:44:12Z

sdks/python/apache_beam/yaml/standard_io.yaml

+      'IcebergAddFiles':
+        table: 'table'
+        catalog_properties: 'catalog_properties'
+        config_properties: 'config_properties'
+        triggering_frequency_seconds: 'triggering_frequency_seconds'
+        append_batch_size: 'append_batch_size'
+        location_prefix: 'location_prefix'


We added a few more properties to the provider:

partition_fields

table_properties

error_handling

ahmedabu98 · 2026-03-25T17:44:37Z

sdks/python/apache_beam/yaml/tests/iceberg_add_files.yaml

+  # Pipeline 0: Initialize Iceberg table
+  - pipeline:
+      type: chain
+      transforms:
+        - type: Create
+          config:
+            elements:
+              - {label: "dummy", rank: -1, bool: true}
+        - type: WriteToIceberg
+          config:
+            table: "default.table"
+            catalog_properties:
+              type: "hadoop"
+              warehouse: "{TEMP_DIR}/dir"


Also we now support creating the table from scratch, so no need to have this initial table-creation step

github-actions · 2026-03-25T17:51:23Z

Assigning reviewers:

R: @jrmccluskey for label python.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

chamikaramj · 2026-03-25T17:51:52Z

sdks/python/apache_beam/yaml/tests/iceberg_add_files.yaml

+            elements:
+              # By default, Beam writes a sharded file like <prefix>-00000-of-00001
+              - {file: "{TEMP_DIR}/data/data-00000-of-00001.parquet"}
+        - type: IcebergAddFiles


Let's also add a test that configures a DLQ (to GCS) since AddFiles transform may push some of the unprocessed files to a DLQ.

ahmedabu98 · 2026-03-25T18:23:52Z

sdks/python/apache_beam/yaml/tests/iceberg_add_files.yaml

+        - type: Create
+          config:
+            elements:
+              - {file: "gs://dummy-bucket/does-not-exist.txt"}


Thanks, this test made me realize I forgot to handle FileNotFound errors!

Your test will still pass though, because .txt is an unsupported file format

github-actions bot added python yaml labels Mar 24, 2026

derrickaw force-pushed the 20260324_icebergAddFilesYamlTransform branch from 6239eb8 to 2f8112d Compare March 25, 2026 15:34

derrickaw changed the title ~~[wip] Iceberg add files yaml transform and test~~ Iceberg add files yaml transform and test Mar 25, 2026

derrickaw changed the title ~~Iceberg add files yaml transform and test~~ [yaml] Iceberg add files yaml transform and test Mar 25, 2026

derrickaw marked this pull request as ready for review March 25, 2026 17:37

tarun-google reviewed Mar 25, 2026

View reviewed changes

ahmedabu98 reviewed Mar 25, 2026

View reviewed changes

github-actions bot added the Next Action: Reviewers label Mar 25, 2026

chamikaramj reviewed Mar 25, 2026

View reviewed changes

derrickaw added 5 commits March 25, 2026 18:22

something

c181e02

add iceberg transform

7bddca5

update pipelines with verification

34c3458

fix comments

d4c704b

address dlq error

017e081

ahmedabu98 reviewed Mar 25, 2026

View reviewed changes

derrickaw force-pushed the 20260324_icebergAddFilesYamlTransform branch from ab2d80f to 017e081 Compare March 25, 2026 18:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[yaml] Iceberg add files yaml transform and test#37938

[yaml] Iceberg add files yaml transform and test#37938
derrickaw wants to merge 5 commits intoapache:masterfrom
derrickaw:20260324_icebergAddFilesYamlTransform

derrickaw commented Mar 24, 2026

Uh oh!

gemini-code-assist bot commented Mar 24, 2026

Uh oh!

derrickaw commented Mar 25, 2026

Uh oh!

derrickaw commented Mar 25, 2026

Uh oh!

derrickaw commented Mar 25, 2026

Uh oh!

tarun-google Mar 25, 2026

Uh oh!

ahmedabu98 Mar 25, 2026 •

edited

Loading

Uh oh!

ahmedabu98 Mar 25, 2026

Uh oh!

github-actions bot commented Mar 25, 2026

Uh oh!

chamikaramj Mar 25, 2026

Uh oh!

ahmedabu98 Mar 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

derrickaw commented Mar 24, 2026

GitHub Actions Tests Status (on master branch)

Uh oh!

gemini-code-assist bot commented Mar 24, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

derrickaw commented Mar 25, 2026

Uh oh!

derrickaw commented Mar 25, 2026

Uh oh!

derrickaw commented Mar 25, 2026

Uh oh!

tarun-google Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

ahmedabu98 Mar 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ahmedabu98 Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 25, 2026

Uh oh!

chamikaramj Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

ahmedabu98 Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ahmedabu98 Mar 25, 2026 •

edited

Loading