Pipeline file cache publication for streamed wheels - #21333
Draft
charliermarsh wants to merge 2 commits into
Draft
Pipeline file cache publication for streamed wheels#21333charliermarsh wants to merge 2 commits into
charliermarsh wants to merge 2 commits into
Conversation
charliermarsh
had a problem deploying
to
automations
August 28, 2026 01:02 — with 
GitHub Actions
Failure
GitHub Actions
Failure
charliermarsh
force-pushed
the
charlie/pipeline-wheel-file-cache
branch
from
August 28, 2026 01:34
d5fb3b8 to
acf995e
Compare
charliermarsh
changed the base branch from
charlie/dirhash-all-files
to
charlie/wheel-mode-manifests
August 28, 2026 01:34
charliermarsh
had a problem deploying
to
automations
August 28, 2026 01:35 — with
GitHub Actions
Error
GitHub Actions
Error
charliermarsh
force-pushed
the
charlie/pipeline-wheel-file-cache
branch
from
August 28, 2026 01:36
acf995e to
92ab703
Compare
charliermarsh
had a problem deploying
to
automations
August 28, 2026 01:37 — with
GitHub Actions
Error
GitHub Actions
Error
This was referenced Aug 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
We currently finish streaming extraction before publishing the extracted files to
files-v0. This PR publishes each completed file after checking its local header and data descriptor, so cache linking overlaps downloading and extracting the remaining payloads. A bounded queue feeds batches to blocking workers, retaining the existing grouping by cache shard.We still keep
RECORDprivate and wait for extraction and file publication to finish before persisting the executable manifest and exposing the wheel inarchive-v0. A rejected or interrupted ZIP can leave completed file objects behind; cache pruning removes those unreferenced objects. The temporary archive stays alive while any blocking publication work is running, including when extraction is cancelled.This uses the existing streaming download without additional requests. Executable status still comes from the central directory and is stored in the manifest; the published file objects remain non-executable.