CI: retry transient HTTP timeouts in Studio smoke probes#7052
Conversation
The post() helper in the Studio inference smoke workflows does a single urlopen with a 240s timeout against the local Studio server. On shared runners this sporadically hits TimeoutError while the server is stalled, failing the whole job for a transport hiccup; the same flake has recurred across unrelated PRs on Linux and Windows (JSON/images and tool-calling jobs) and passes on rerun. Retry the probe up to 3 times on transport-level failures only (TimeoutError, ConnectionError, non-HTTP URLError), 15s apart. HTTP status errors still surface immediately, so genuine server failures are unaffected. post_sse() is left unchanged: it has a 600s budget and has not flaked.
|
Note Gemini is unable to generate a review for this pull request due to the file types involved not being currently supported. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fc76ce3391
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| return resp.status, json.loads(resp.read().decode()) | ||
| # Shared CI runners stall sporadically, so retry transport-level | ||
| # failures only; HTTP status errors surface immediately. | ||
| for attempt in range(3): |
There was a problem hiding this comment.
This unconditional three-attempt loop also wraps the JSON/images helper, where the Linux job calls post(..., timeout = 600) while the whole job is limited to 30 minutes. If that probe times out more than once, this step alone can spend about 30.5 minutes before re-raising, after install/model-load time has already elapsed, so the workflow is likely to be killed by the job timeout instead of failing with the Python error and collecting the later logs; the Windows JSON copy has the same risk across its 600-second probes. Please bound the total retry deadline or reduce/disable retries for the long-timeout calls.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in b4c02db: retries now apply only when timeout <= 300s (the observed flaky 180-240s probes), so the 600s json-images calls keep their single attempt and worst-case step time stays well inside timeout-minutes.
Some json-images calls pass timeout=600; three attempts there could spend 30 minutes in one step and hit the job's timeout-minutes instead of failing with the Python error. Retry (3 attempts) only when timeout <= 300s, which covers the observed flaky 180-240s probes; longer probes keep the pre-PR single attempt.
|
@codex review |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b4c02db66f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # failures only; HTTP status errors surface immediately. Long | ||
| # probes (timeout > 300s) get a single attempt so the worst | ||
| # case stays inside the job's timeout-minutes budget. | ||
| attempts = 3 if timeout <= 300 else 1 |
There was a problem hiding this comment.
In this JSON/images block the only raw urllib probe below calls post(..., timeout = 600) (line 1017), so this ternary sets attempts to 1 and the new retry path is never used for the long JSON-mode request. The same pattern exists in the Windows JSON/images job for its raw posts, so a transient TimeoutError in those probes will still fail on the first occurrence despite this change.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in e739dc1: long probes now get one retry with the attempt timeout capped at 300s, so the 600s JSON-mode probes are covered while worst-case wall time still fits timeout-minutes (one long probe 15.25 min; the Windows JSON job's two long probes 30.5 min vs its 35 minute budget).
Round two of bounding the retries: timeout>300s probes previously got a single attempt, so a transient stall in the 600s JSON-mode probes still failed on first occurrence. Give them one retry with the attempt timeout capped at 300s. Worst cases stay inside timeout-minutes: 240s probes 12.5 min, one 600s probe 15.25 min, the Windows JSON job's two long probes 30.5 min against its 35 minute budget.
|
@codex review |
|
Codex Review: Didn't find any major issues. Chef's kiss. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
1 participant
Summary
The Studio inference smoke workflows (
studio-inference-smoke.yml,studio-mac-inference-smoke.yml,studio-windows-inference-smoke.yml) probe the local Studio server with apost()helper that does a singleurllib.request.urlopenwith a 240s timeout. On shared runners this sporadically raisesTimeoutError: timed outwhile the server is stalled, failing the whole 25 minute job for one transport hiccup.This is a recurring flake, not a code signal: the same timeout hit the "JSON, images" job on #7042 and #7043 and the "Tool calling Tests" step on a Windows staging run, on unrelated diffs, and every occurrence passed on plain re-run.
Change
Retry the probe in
post()up to 3 times, 15s apart, on transport-level failures only:TimeoutError,ConnectionError, and non-HTTPURLError.HTTPErrorre-raises immediately, so any response the server actually produced (4xx/5xx) still fails on the first attempt and genuine regressions surface unchanged.post_sse()is deliberately untouched: it has a 600s budget and has not flaked.post()definitions (2 per OS workflow), plus thetime/urllib.errorimports.These probes hit a seeded local test server, so re-sending a request is safe.
Validation
ast.parseof every inline python block in the three workflows.