fix(extractor): cap downloader retry budget to keep CI test runtime sane#318
Merged
SimplicityGuy merged 1 commit intomainfrom Apr 29, 2026
Merged
fix(extractor): cap downloader retry budget to keep CI test runtime sane#318SimplicityGuy merged 1 commit intomainfrom
SimplicityGuy merged 1 commit intomainfrom
Conversation
PR #317 raised both downloaders' transport-error retry budget from 3 attempts at 2s base to 5 attempts at 30s base. The intent was rate-limit politeness, but rate-limit handling actually lives in `polite_http` and never reaches this loop — only post-connect transport errors do (partial reads, flush/sync errors, mid-stream drops). Side effect: integration tests in `tests/http_integration_test.rs` see `cfg(not(test))` (the `#[cfg(test)]` override only applies when the *library* itself is being unit-tested, not when external test binaries link against it). Failing-download tests therefore paid the full production backoff: 30+60+120+240 = 450s sleeping per test. That blew up `http_integration_test` from ~50s on main to 460s locally, and made `test-extractor` time out under `cargo-llvm-cov` in CI on PR #317. Restoring the original 3-retry / 2s-base — those are the right numbers for transport-error retry. The polite client owns the rate-limit backoff path with its own server-driven Retry-After handling and per- provider caps (2h Discogs, 30m MusicBrainz). Wall time: `cargo test --test http_integration_test` 460s → 75s sequentially, 20s parallel. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Contributor
E2E Coverage (webkit)
|
Contributor
E2E Coverage (chromium)
|
Contributor
E2E Coverage (firefox)
|
Contributor
E2E Coverage (webkit - iPhone 15)
|
Contributor
E2E Coverage (webkit - iPad Pro 11)
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PR #317 raised both downloaders' transport-error retry budget from 3 attempts at 2s base to 5 attempts at 30s base. The intent was rate-limit politeness, but rate-limit handling actually lives in `polite_http` and never reaches this loop — only post-connect transport errors do (partial reads, flush/sync errors, mid-stream drops).
Side effect: integration tests in `tests/http_integration_test.rs` see `cfg(not(test))` (the `#[cfg(test)]` override only applies when the library itself is being unit-tested, not when external test binaries link against it). Failing-download tests therefore paid the full production backoff:
```
attempt 2: 30s
attempt 3: 60s
attempt 4: 120s
attempt 5: 240s
total: 450s sleeping per test
```
That blew up `http_integration_test` from ~50s on main to 460s locally, and made `test-extractor` time out under `cargo-llvm-cov` in CI on PR #317 (this is the hung CI run that sat pending for ~10+ minutes before timing out).
Fix
Restore the original 3-retry / 2s-base — those are the right numbers for transport-error retry. The polite client owns the rate-limit backoff path with its own server-driven `Retry-After` handling and per-provider caps (2h Discogs, 30m MusicBrainz). No need to also retry slowly here.
Wall time
Test plan
🤖 Generated with Claude Code