[Iceberg] Support async scan planning in Iceberg REST planning client - #7562
Open
Bartekszost wants to merge 7 commits into
Open
[Iceberg] Support async scan planning in Iceberg REST planning client#7562Bartekszost wants to merge 7 commits into
Bartekszost wants to merge 7 commits into
Conversation
Follow the Iceberg REST scan planning spec: when POST /plan returns a
"submitted" status with a plan-id, poll GET /plan/{plan-id}
(fetchPlanningResult) until the plan reaches a terminal status, matching
the behavior of Iceberg's RESTTableScan.
- Switch on planStatus from POST /plan: completed converts inline file
scan tasks as before; submitted requires a plan-id and polls; failed
and cancelled surface the error payload.
- Poll with Iceberg's Tasks exponential backoff, retrying only while the
plan is still submitted; poll timeout, retries, min/max wait, and
scale factor are read from the catalog /v1/config properties, using
the same rest-scan-planning.poll-* keys and defaults as Iceberg.
- Require the fetchPlanningResult endpoint to be advertised before
polling, URL-encode the plan-id, and close the POST response before
polling to avoid holding a pooled connection.
- Reject plan-tasks: completed responses must contain inline file scan
tasks; plan-task pagination (fetchScanTasks) is not supported yet.
- Extend the test REST server and adapter to inject submitted responses
and serve GET /plan/{plan-id}, and add tests covering poll-to-completed,
retry exhaustion, missing endpoint, and terminal failed/cancelled.
Signed-off-by: Bartosz Szostakiewicz <b.szostakiewicz2003@gmail.com>
There was a problem hiding this comment.
Pull request overview
Adds support for Iceberg REST asynchronous scan planning in Delta’s Iceberg REST planning client: when POST .../plan returns SUBMITTED with a plan-id, the client polls GET .../plan/{plan-id} using Iceberg Tasks exponential backoff and poll settings derived from /v1/config.
Changes:
- Implement polling of
fetchPlanningResultforSUBMITTEDscan plans, including terminalFAILED/CANCELLEDhandling and explicit rejection ofplan-taskspagination. - Expand
/v1/configparsing to include endpoints and poll tuning properties (timeout/retries/min/max wait/scale factor). - Extend the shaded test REST server + servlet to simulate async planning and advertise (or omit) the fetch-planning-result endpoint; add corresponding suite coverage.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| iceberg/src/main/scala/org/apache/spark/sql/delta/serverSidePlanning/IcebergRESTCatalogPlanningClient.scala | Adds async planning polling logic, poll configuration parsing, and endpoint capability checks. |
| iceberg/src/test/scala/org/apache/spark/sql/delta/serverSidePlanning/IcebergRESTCatalogPlanningClientSuite.scala | Adds tests for async planning completion, retry exhaustion, missing endpoint, and terminal statuses. |
| iceberg/src/test/java/shadedForDelta/org/apache/iceberg/rest/IcebergRESTServletWithPlanSupport.java | Enhances /v1/config to return defaults + endpoints and supports GET plan-result requests. |
| iceberg/src/test/java/shadedForDelta/org/apache/iceberg/rest/IcebergRESTServer.java | Adds test hooks for config defaults, endpoint advertising, poll counting, and async status injection. |
| iceberg/src/test/java/shadedForDelta/org/apache/iceberg/rest/IcebergRESTCatalogAdapterWithPlanSupport.java | Implements async planning simulation (submitted polls, terminal result responses) and poll request counting. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
60
to
+63
| private case class CatalogConfigResponse( | ||
| defaults: Map[String, String], | ||
| overrides: Map[String, String]) | ||
| overrides: Map[String, String], | ||
| endpoints: Option[Seq[String]] = None) |
Author
There was a problem hiding this comment.
Fixed. defaults and overrides now default to empty maps, and the merge treats a null field as empty so a partial /v1/config body cannot NPE.
Default /v1/config defaults and overrides to empty maps so a partial JSON body cannot NPE when merging catalog properties for poll settings. Signed-off-by: Bartosz Szostakiewicz <b.szostakiewicz2003@gmail.com>
murali-db
reviewed
Aug 31, 2026
…d unsafe retries. Reject delete files, cancel timed-out submitted plans, require an advertised fetch endpoint, and stop retrying POSTs that may already have allocated a plan-id. Signed-off-by: Bartosz Szostakiewicz <b.szostakiewicz2003@gmail.com>
murali-db
reviewed
Sep 1, 2026
Best-effort cancel submitted plans on any non-terminal poll exit, request vended credentials, close the planning client on failure, and stop retrying POST /plan on 5xx. Signed-off-by: Bartosz Szostakiewicz <b.szostakiewicz2003@gmail.com>
Validate poll settings before POST /plan, cancel if fetchPlanningResult fails during setup, and memoize failed scanPlan so a later access does not reuse a closed client. Signed-off-by: Bartosz Szostakiewicz <b.szostakiewicz2003@gmail.com>
murali-db
approved these changes
Sep 1, 2026
foss-contributor
added
delta-spark
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
POST /planreturnssubmittedwith aplan-id, pollGET /plan/{plan-id}(fetchPlanningResult) until the plan is terminal, matching Iceberg'sRESTTableScan.Tasksexponential backoff. Timeout, retries, min/max wait, and scale factor come from catalog/v1/configusing the samerest-scan-planning.poll-*keys and defaults as Iceberg.file-scan-tasks.plan-tasks/fetchScanTaskspagination is out of scope.Test plan
IcebergRESTCatalogPlanningClientSuite(Spark 4.1): poll until completed, retry exhaustion, missing fetch endpoint, terminal failed/cancelled, existing 503/404 and sync plan tests