Adding time partitioning support for python storage write api#35737
Adding time partitioning support for python storage write api#35737TanuSharma2511 wants to merge 11 commits intoapache:masterfrom
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary of Changes
Hello @TanuSharma2511, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request significantly enhances the BigQuery Storage Write API support in Apache Beam by introducing comprehensive functionality for BigQuery time partitioning. It enables users to define and apply time-based partitioning to their BigQuery tables directly through the Beam Python SDK, improving data organization, query performance, and cost efficiency for time-series data. The changes span across the Java and Python SDKs, including configuration, API integration, and thorough testing.
Highlights
- BigQuery Time Partitioning Configuration (Java): I've introduced a new
TimePartitioningConfigclass withinBigQueryWriteConfiguration.javato encapsulate BigQuery time partitioning properties (type, field, expirationMs, requirePartitionFilter). This configuration can now be set via theBigQueryWriteConfigurationbuilder. - BigQuery Time Partitioning Application (Java): The
PortableBigQueryDestinations.javafile has been updated to consume the newTimePartitioningConfig. ThegetTablemethod now constructs and applies the appropriate BigQueryTimePartitioningobject when creatingTableDestinations, ensuring that the specified partitioning is applied to the target BigQuery table. - Python Storage Write API Integration: The Python
WriteToBigQuerytransform inbigquery.pynow supports passingtimePartitioningparameters via theadditional_bq_parametersargument. These parameters are then forwarded to the underlying Java SDK for cross-language execution, enabling Python users to configure time-partitioned tables. - Integration Tests: New integration tests have been added in both Java (
BigQueryManagedIT.java) and Python (xlang_bigqueryio_it_test.py) to validate the correct application of time partitioning when writing data to BigQuery using the managed IO and Storage Write API, respectively. The Python test includes anevent_timefield in the test schema and data. - Documentation Update: The documentation for BigQuery managed IO (
managed-io.md) has been updated to include details about the newTime Partitioningconfiguration option, providing guidance for users on how to leverage this feature.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments or fill out our survey to provide feedback.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
There was a problem hiding this comment.
Code Review
This pull request adds time partitioning support for the Python Storage Write API and BigQuery Managed IO. The changes look good overall, but I've found a few issues that need to be addressed:
- A critical issue in a Python integration test where a project name is hardcoded.
- A high-severity bug in the Python BigQuery IO implementation related to config initialization.
- Some minor issues like commented-out code in a test and a typo in the documentation.
Please address these points to ensure the code is robust and maintainable.
sdks/python/apache_beam/io/external/xlang_bigqueryio_it_test.py
Outdated
Show resolved
Hide resolved
...-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/providers/BigQueryManagedIT.java
Outdated
Show resolved
Hide resolved
|
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment |
|
Assigning reviewers: R: @shunping for label python. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
|
Java Unit Tests |
|
Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment R: @jrmccluskey for label python. Available commands:
|
ahmedabu98
left a comment
ahmedabu98
left a comment
There was a problem hiding this comment.
Thanks for this PR! Left a few comments but generally looks good
...rc/main/java/org/apache/beam/sdk/io/gcp/bigquery/providers/PortableBigQueryDestinations.java
Outdated
Show resolved
Hide resolved
.../src/main/java/org/apache/beam/sdk/io/gcp/bigquery/providers/BigQueryWriteConfiguration.java
Show resolved
Hide resolved
|
Reminder, please take a look at this pr: @jrmccluskey @Abacn |
|
Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment R: @tvalentyn for label python. Available commands:
|
|
Run Yaml_Xlang_Direct PreCommit |
|
Run Python PreCommit 3.11 |
|
R: @ahmedabu98 since you reviewed before |
|
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment |
|
Run Python_ML PreCommit 3.9 |
|
Run Python_ML PreCommit 3.12 |
|
Run Python_ML PreCommit 3.11 |
|
Run Python_ML PreCommit 3.12 |
|
@ahmedabu98 can you take a look at latest changes? |
| Boolean requirePartitionFilter = timePartitioningConfig.getRequirePartitionFilter(); | ||
|
|
||
| if (type == null) { | ||
| Set<String> allowedTypes = new HashSet<>(Arrays.asList("DAY", "HOUR", "MONTH", "YEAR")); |
There was a problem hiding this comment.
nit: this should be a final static class variable
ahmedabu98
left a comment
ahmedabu98
left a comment
There was a problem hiding this comment.
left a small nit, but over LGTM! Thanks for writing this @TanuSharma2511
|
@TanuSharma2511 please respond here once it's ready for the next review iteration/merge. thanks! |
|
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions. |
|
This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. |
|
is there any chance we can resurrect this? this features remains very useful when loading data into bigquery via |
6 participants
It resolves a part of #35329
1.Dynamic Time Partitioning is not supported yet. It makes sense to support dynamic Time Partitioning after we support dynamic schema for Python Bigquery Storage Write API.
2.Time Partitioning Support for Bigquery Managed IO.
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.