Fix wheel metadata reads after redirects

[dimbleby]

When a wheel URL redirected, the HEAD request followed the redirect but the range requests still used the original URL. Their HTTP client did not follow redirects, so metadata reads failed and uv downloaded whole wheels.

Use the final URL from the HEAD response for range requests. Drop the original Authorization header when that URL is in a different auth realm.

---

A healthy assist from the AI on this: but I am a human in the loop and it looks good to me.

Tested live against a feed that does not support PEP658 but does support range requests; which is what prompted the investigation.

[zanieb]

👋 thanks for the pull, I'll take a look.

In what context did you encounter this?

[dimbleby]

In what context did you encounter this?

the day job has moved us to using an Azure Artifacts feed, at which point resolution started downloading whole wheels

[zanieb]

This sounded familiar so I pulled some history with Codex, I think there's a bunch of nuance we'll need to consider here:

• Retain authentication when making range requests #1902 established that authorization from the initial HEAD must carry into range requests.
• #2025, #3255, and #2843 found that some indexes redirect HEAD and GET to different presigned S3 URLs. A URL signed for HEAD returns 403 when reused for GET.
• Consequently, in fix: always use requested URL, rather than response URL prefix-dev/async_http_range_reader#16 we explicitly changed the reader to “always use requested URL, rather than response URL (see also Upgrade async_http_range_reader to v0.8.0 #3460).
• Support netrc and same-origin credential propagation on index redirects #14126 later introduced uv’s manual redirect wrapper so redirect requests rerun authentication middleware. The underlying reqwest client has redirects disabled, and converting the wrapper into the ClientWithMiddleware accepted by AsyncHttpRangeReader drops that wrapper. This appears to be the actual regression encountered here.
• Avoid batch prefetching for un-optimized registries #7226 amplifies the effect: once a range failure is classified as unsupported, uv records that capability for the index and stops range-prefetching there.
• uv's HEAD-based range detection for wheel metadata is inconsistent with RFC 9110 and misclassifies GAR wheel URLs #18998 reports essentially the same GAR shape
• Since Stream zip archive when fetching non-range-request metadata #1792, the fallback streams until it finds METADATA; it does not necessarily download the entire wheel to disk, though it may consume most of one.

[astral-sh-bot]

uv test inventory changes

This PR changes the tests when compared with the main base revision.

• Added tests: 1
• Removed tests: 0
• Changed suites: 1

uv-client::it: +1 / -0

Added:

• uv-client::it::remote_metadata::remote_metadata_uses_head_redirect_target_for_range_requests

Removed: none

[dimbleby]

What fun...

I had found #18998 and decided it wasn't the same: our HEAD does advertise "Accepts-Ranges: bytes", whereas the other seems to be saying that: even if a HEAD does not advertise support, that might not mean much.

But I didn't spot the others, and it seems this is going to be more complicated than I probably want for a drive-by fix.

Shall I raise an issue report instead?

[zanieb]

If you want to open an issue with details about what happened and some logs that'd be great.

I will probably add some more test coverage and weigh a couple different approaches here. I haven't really decided if this change is correct or if something else is more appropriate, I have to refresh my memory on all that context.

[dimbleby]

AFAIK the testcase here is an accurate representation of the observed real world behaviour (and fails without the fix) so I will just reference that unless you want something specific.

[thomasaarholt]

Hi @zanieb, we're a few folks at Microsoft discussing this right now.

We can test against a new PyPI proxy feed which is open to the public. It doesn't support PEP 658, but should support range requests. https://packagefeedproxy.microsoft.io/pypi/simple

We can create an issue with some more context.

[zanieb]

Thanks a public registry is super helpful here.

What level of urgency is this for ya'll?

[thomasaarholt]

Thanks for asking. It would be great to get in a fix in the next week or so, but no-one is fully blocked here.

[thomasaarholt]

I had my own Opus instance have a go at this, and it came up with a very similar solution, branch here. Sol had a go too, but I think it went a bit overboard trying to make things secure.

Very cool to see the automations bot at work here! Thanks for taking the time.