ref.toml doesn't survive image/python upgrades: bake packaged-data paths and unknown providers

[lewisjared]

Summary

A persistent ref.toml (on a shared volume, shared between workers, CLI, API) should be durable across image upgrades. Today two separate things can take every process down after an otherwise-routine upgrade:

1. Fields that reference packaged data files (currently: dimensions_cv) get serialized as absolute venv paths that break when the Python version or venv layout changes.
2. Fields that reference named modules (currently: [[diagnostic_providers]]) hard-crash the whole process if any entry no longer resolves.

Both were hit during an upgrade from climate-ref-frontend:v0.2.x (py3.11) to v0.3.0 (py3.13). The full trail is in Climate-REF/ref-app#29.

1. Packaged data should behave like ignore_datasets_file, not a venv path

ignore_datasets_file already solves this cleanly (packages/climate-ref/src/climate_ref/config.py::_get_default_ignore_datasets_file): the default materializes a cached copy under REF_CONFIGURATION/cache/climate_ref/, i.e. on the persistent volume, not in the venv. The path written to ref.toml stays valid across image changes, and the loader re-materializes if the cached file is missing.

dimensions_cv uses a different pattern (DimensionsCV._dimensions_cv_factory): it resolves importlib.resources.files(\"climate_ref_core.pycmec\") / \"cv_cmip7_aft.yaml\", gets the resolved absolute path (e.g. /app/.venv/lib/python3.11/site-packages/...), and that path gets persisted into ref.toml. When the image moves to py3.13 the file is still present at the new venv path, but the stale py3.11 path in ref.toml wins and the service fails at startup with FileNotFoundError.

Generalize the ignore_datasets_file approach to any packaged data file referenced from ref.toml. Options:

• Introduce a helper (e.g. _get_default_packaged_file(pkg, filename) -> Path) that materializes the package resource into REF_CONFIGURATION/cache/<pkg>/<filename> on first read, the way _get_default_ignore_datasets_file does for the grey list. dimensions_cv (and any future sibling) uses this instead of importlib.resources directly.
• Or don't serialize packaged-data fields to ref.toml at all when the value equals the packaged default; let the loader re-resolve from importlib.resources every start.
• Or store a sentinel / packaged:<pkg>/<file> URI rather than an absolute filesystem path; loader expands it at load time.

Any of these fixes the symptom. The first matches the existing grey-list pattern and keeps a stable on-disk artefact under REF_CONFIGURATION, which is the shape the deployment already expects.

2. Unknown providers should degrade, not crash

climate_ref_core.providers.import_provider raises InvalidProviderException on ModuleNotFoundError, and ProviderRegistry.build_from_config propagates it, so one stale entry in [[diagnostic_providers]] kills every process (API, workers, CLI) at boot.

Trigger: ref.toml still carried provider = \"climate_ref_example:provider\" after the image stopped shipping that package. Nothing worked until the entry was manually sed'd out.

Fix options (pick at least one):

• In build_from_config (or the config loader), try/except ModuleNotFoundError per provider: log a warning, skip the entry, continue. A missing provider should not take down the whole stack.
• Add a ref config providers prune command (and run it implicitly from ref providers setup) that removes entries whose module doesn't resolve, so ref.toml self-heals after upgrades that drop a provider.
• Discover providers by entry-point / distribution name rather than dotted module path, so ref.toml doesn't have to redeclare what's installed.

Acceptance

• Path durability: a ref.toml written by one Python / image loads cleanly under a different Python / image without operator intervention for any field that currently references package data. Covered by a round-trip test that simulates the py3.11 → py3.13 move for dimensions_cv.
• Provider resilience: a ref.toml with one unknown provider and two valid ones starts successfully; the two valid providers register, the unknown one is logged and skipped. Covered by a unit test against build_from_config.
• Existing behavior unchanged when everything listed is installed.

[lewisjared]

#622 (in review) establishes exactly the pattern I was describing for (1): move the file into $REF_CONFIGURATION/, decouple fetching from Config construction, and tolerate a missing file gracefully. Once that merges, the same shape applies 1:1 to dimensions_cv:

• Default location: $REF_CONFIGURATION/cv_cmip7_aft.yaml (beside ref.toml / grey_list.yaml / the DB) instead of an absolute venv path that bakes Python version into the config.
• Loader materializes it from importlib.resources.files("climate_ref_core.pycmec") on first read (and optionally on upgrade), same as refresh_grey_list_file().
• Field only gets serialized to ref.toml when the user has explicitly overridden it.

So (1) probably reduces to reusing the helper #622 lands. (2) is still independent — unknown [[diagnostic_providers]] entries should degrade rather than crash, regardless of where package data lives.