Skip to content

fix(source_metadata): deduplicate metadata requests across environments#5647

Open
mohitdebian wants to merge 5 commits intoprefix-dev:mainfrom
mohitdebian:fix-metadata-deduplication
Open

fix(source_metadata): deduplicate metadata requests across environments#5647
mohitdebian wants to merge 5 commits intoprefix-dev:mainfrom
mohitdebian:fix-metadata-deduplication

Conversation

@mohitdebian
Copy link
Contributor

@mohitdebian mohitdebian commented Mar 10, 2026

Description

When the same source package is used across multiple pixi environments (for example default and test), each environment may request SourceMetadata concurrently.

Previously, CommandDispatcherProcessor deduplicated requests using the full SourceMetadataSpec:

HashMap<SourceMetadataSpec, SourceMetadataId>

However, SourceMetadataSpec embeds BuildBackendMetadataSpec, which contains environment-specific fields such as:

  • channels
  • channel_config
  • build_environment
  • variant_configuration
  • variant_files

These fields often differ between environments even when they refer to the exact same source package. Because of this, the dispatcher treated the requests as different tasks and spawned multiple metadata computations.

When those tasks attempted to write to the disk cache concurrently, they could race on try_write, resulting in:

WriteResult::Conflict

Changes in this PR

This PR introduces a normalized in-memory deduplication key:

SourceMetadataKey

SourceMetadataKey flattens the fields from SourceMetadataSpec that determine the identity of the metadata request. This allows the dispatcher to deduplicate requests without relying on the nested structure of SourceMetadataSpec.

The key includes:

  • package
  • manifest_source
  • preferred_build_source
  • enabled_protocols
  • build_environment
  • channels
  • channel_config
  • variant_configuration
  • variant_files

Additional changes:

  • Added SourceMetadataKey struct in source_metadata/mod.rs
  • Added SourceMetadataSpec::dedup_key() to construct the normalized key
  • Updated dispatcher map:
HashMap<SourceMetadataSpec, SourceMetadataId>
→
HashMap<SourceMetadataKey, SourceMetadataId>
  • Updated on_source_metadata to compute the key once and reuse it
  • Updated gen_source_metadata_id to accept the precomputed key
  • Improved handling of WriteResult::Conflict so that the metadata already written to the cache is reused

Effect

Before:

env A -> compute metadata
env B -> compute metadata
↓
both write to cache
↓
WriteResult::Conflict

After:

env A -> compute metadata
env B -> attach to same pending task
↓
single computation
↓
result broadcast to all callers

This ensures metadata for the same source package is computed only once per dispatcher execution and prevents redundant cache writes.

Fixes #5626


How Has This Been Tested?

The changes were verified locally with:

cargo check
cargo test
cargo clippy --all-targets --all-features
cargo fmt

Additionally verified that multiple environments requesting the same source metadata attach to the same dispatcher task instead of spawning duplicate computations.


AI Disclosure

  • This PR contains AI-generated content.
    • I have tested any AI-generated content in my PR.
    • I take responsibility for any AI-generated content in my PR.

Tools: Claude


Checklist:

  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have added sufficient tests to cover my changes.
  • I have verified that changes that would impact the JSON schema have been made in schema/model.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cache was updated by another process warning when using pixi build

2 participants