[Iceberg] Configure REST scan polling from catalog properties - #7584
Draft
Bartekszost wants to merge 8 commits into
Draft
[Iceberg] Configure REST scan polling from catalog properties#7584Bartekszost wants to merge 8 commits into
Bartekszost wants to merge 8 commits into
Conversation
Follow the Iceberg REST scan planning spec: when POST /plan returns a
"submitted" status with a plan-id, poll GET /plan/{plan-id}
(fetchPlanningResult) until the plan reaches a terminal status, matching
the behavior of Iceberg's RESTTableScan.
- Switch on planStatus from POST /plan: completed converts inline file
scan tasks as before; submitted requires a plan-id and polls; failed
and cancelled surface the error payload.
- Poll with Iceberg's Tasks exponential backoff, retrying only while the
plan is still submitted; poll timeout, retries, min/max wait, and
scale factor are read from the catalog /v1/config properties, using
the same rest-scan-planning.poll-* keys and defaults as Iceberg.
- Require the fetchPlanningResult endpoint to be advertised before
polling, URL-encode the plan-id, and close the POST response before
polling to avoid holding a pooled connection.
- Reject plan-tasks: completed responses must contain inline file scan
tasks; plan-task pagination (fetchScanTasks) is not supported yet.
- Extend the test REST server and adapter to inject submitted responses
and serve GET /plan/{plan-id}, and add tests covering poll-to-completed,
retry exhaustion, missing endpoint, and terminal failed/cancelled.
Signed-off-by: Bartosz Szostakiewicz <b.szostakiewicz2003@gmail.com>
Default /v1/config defaults and overrides to empty maps so a partial JSON body cannot NPE when merging catalog properties for poll settings. Signed-off-by: Bartosz Szostakiewicz <b.szostakiewicz2003@gmail.com>
…d unsafe retries. Reject delete files, cancel timed-out submitted plans, require an advertised fetch endpoint, and stop retrying POSTs that may already have allocated a plan-id. Signed-off-by: Bartosz Szostakiewicz <b.szostakiewicz2003@gmail.com>
Best-effort cancel submitted plans on any non-terminal poll exit, request vended credentials, close the planning client on failure, and stop retrying POST /plan on 5xx. Signed-off-by: Bartosz Szostakiewicz <b.szostakiewicz2003@gmail.com>
Validate poll settings before POST /plan, cancel if fetchPlanningResult fails during setup, and memoize failed scanPlan so a later access does not reuse a closed client. Signed-off-by: Bartosz Szostakiewicz <b.szostakiewicz2003@gmail.com>
Pass Spark catalog poll settings to the REST planning client and merge them using server overrides, client configuration, server defaults, then client defaults. Signed-off-by: Bartosz Szostakiewicz <b.szostakiewicz2003@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #7562. Please review commit
8736b8273; the diff will narrow automatically after #7562 merges.spark.sql.catalog.<catalog>.rest-scan-planning.poll-*properties to the Iceberg REST planning clientTest plan
build/sbt -DsparkVersion=4.1 \"iceberg / Test / testOnly org.apache.spark.sql.delta.serverSidePlanning.IcebergRESTCatalogPlanningClientSuite\"build/sbt -DsparkVersion=4.1 \"spark / Test / testOnly org.apache.spark.sql.delta.serverSidePlanning.ServerSidePlannedTableSuite\"