Hotfix Soniox fallback keys for legacy cloud#3219
Conversation
📋 PR Review Helper📱 Mobile App Build⏳ Waiting for build... 🕶️ ASG Client Build⏳ Waiting for build... 🔀 Test Locallygh pr checkout 3219 |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8e8ff20339
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| lower.includes("unauthorized") | ||
| ) { | ||
| return { kind: "auth", cooldownMs: Number.POSITIVE_INFINITY, disabled: true }; | ||
| } |
There was a problem hiding this comment.
403 errors not treated as auth
Low Severity
classifySonioxCredentialFailure disables credentials for HTTP 401 but not 403, so a Soniox error 403 is classified as transient with a short cooldown. That conflicts with transcription retry logic, which treats 403 as a non-retryable auth failure, and the key stays in rotation instead of being disabled for the process.
Reviewed by Cursor Bugbot for commit 08f06fd. Configure here.
There was a problem hiding this comment.
Leaving this open intentionally for follow-up. For the ASAP prod outage fix we are not broadening permanent disable behavior to 403; the production incident is the 402/budget quota path, which is fixed in e06781b.
|
Related Soniox fallback PRs:
Current prod/main hotfix head: e06781b. |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
There are 2 total unresolved issues (including 1 from previous review).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit e06781b. Configure here.
| this.config, | ||
| throw ( | ||
| lastError ?? | ||
| new Error(`No available Soniox credentials: ${JSON.stringify(this.keyPool.describeAvailability())}`) |
There was a problem hiding this comment.
Wrong error when keys cooling
Medium Severity
When stream creation stops because every remaining credential is on cooldown, the code still throws the last failure from an earlier key if one was tried. TranscriptionManager treats that as non-retryable for auth-style Soniox errors, so creation can give up even though fallback keys were never tried and would become available after their cooldown.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit e06781b. Configure here.
There was a problem hiding this comment.
Leaving this open intentionally for follow-up. It is related to the broader last-error masking/retry semantics, but the ASAP prod hotfix is intentionally narrowed to the confirmed 402/budget quota outage path and shared fallback cooldown handling.
1 participant
Summary
Adds Soniox fallback API keys for legacy Cloud v1 transcription and translation streams.
Why
The primary Soniox org/key hit its monthly spend ceiling, which made every new Soniox stream fail through the same exhausted credential. The old code only had provider-level retry/fallback; it did not rotate between multiple Soniox credentials.
What changed
SONIOX_FALLBACK_API_KEYSas a comma-separated fallback list.SONIOX_API_KEYas the preferred primary key while healthy.cloud/issues/108-soniox-fallback-api-keys/.Validation
bun test cloud/packages/cloud/src/services/session/soniox/__tests__/SonioxKeyPool.test.tsgit diff --checkcd cloud && bun run buildcd cloud/packages/cloud && bun run build8e8ff2033941a236963a7aa12e577b8aa6881d66to legacycloud-debugon Azure Central US viaporter-debug.yml.https://debugapi.mentra.glass/livezand/healthreturn healthy.Note
Medium Risk
Touches live STT/translation credential selection and retry behavior on a production hotfix path; misconfiguration or aggressive cooldowns could still exhaust all keys, though changes are scoped and tested at the pool layer.
Overview
Adds
SONIOX_FALLBACK_API_KEYS(comma-separated) so legacy Cloud v1 can rotate across multiple Soniox credentials when the primary org/key is quota- or rate-limited.Introduces a shared
SonioxKeyPool: primary-first selection, round-robin fallbacks, per-key cooldowns (auth disables for the process; quota/rate/concurrency/transient use different durations), and SHA-256 fingerprints in logs instead of raw keys. Cooldown state is shared in-process between transcription and translation viagetSharedSonioxKeyPool.Transcription and translation Soniox providers now loop credentials on stream creation, record runtime failures on streams, and managers treat capacity errors and “no credentials available” as retryable so a new stream can pick another key.
Also bumps the default Soniox realtime model from
stt-rt-v4tostt-rt-v5in config and stream setup. Spike/spec docs live undercloud/issues/108-soniox-fallback-api-keys/.Reviewed by Cursor Bugbot for commit 9e0a49f. Bugbot is set up for automated code reviews on this repo. Configure here.
Summary by cubic
Adds fallback Soniox API keys and a shared in‑process key pool so legacy Cloud v1 transcription and translation can rotate keys with smart cooldowns. Also defaults the Soniox model to
stt-rt-v5and tightens quota cooldown handling.Bug Fixes
SONIOX_FALLBACK_API_KEYSand a shared key pool used by transcription and translation; cooldowns propagate across providers within the process.stt-rt-v5(SDK and WebSocket paths).Migration
SONIOX_FALLBACK_API_KEYSas a comma-separated list of Soniox keys in Cloud v1 environments. No other changes needed.Written for commit e06781b. Summary will update on new commits.