feat: multi-and-per-flow rates schema (v2.2.0)#65
Merged
Conversation
added 7 commits
May 9, 2026 18:54
Both originated from sims work in skyprospector.com/skypro-service (NCESF + HMCE Axle integration). HMCE 202604 is the validation case for both — its rebuild today landed with known biases that these two changes close. - per-flow-imbalance-source-override: lets BSCP550 wire BESS flows to Axle-stacked imbalance and site flows to plain imbalance within a single rates block. Closes a ~£500-700/mo over-attribution bias on HMCE Apr-26 BSCP550 scenarios where Axle currently leaks onto gridToLoad and solarToGrid. - multi-final-rates: lets one simulation run produce N parallel ratesFinal column-sets against the same dispatch. Avoids re-running the optimiser to compare settlement structures (e.g. Axle on vs off, Trio vs flat, OSAM on/off). Eliminates the standalone Impr0 scenario pattern. Both are ~3-4 hour PRs. Independent — can ship in either order.
…ation Scenarios marked `enabled: false` in simulate.yaml are now dropped before marshmallow validation, mirroring the rebuilder's existing skip logic. This lets externally-managed scenarios (e.g. Monte-Carlo runs whose outputs are produced outside skypro) coexist with live scenarios in the same simulate.yaml — no need for the rebuilder to write a temp staged yaml that scrubs them out. - parse_config.py: pre-load filter walks simulations dict and removes entries with `enabled is False` before Config.Schema().load - report-side parser unchanged: report.yaml has no enabled annotations in any current tenant config Existing 34 unit + integration tests pass unchanged.
Closes the structural Axle-premium leak on two-MPAN BSCP550 sites where the
BESS and site MPANs settle against different imbalance signals. Previously
every flow in a rates block read from the block-level `imbalanceDataSource`,
so site-MPAN flows (gridToLoad, solarToGrid) absorbed any premium intended
only for the BESS-MPAN flows (gridToBatt, battToGrid).
Schema (backward-compatible — legacy list shape still parses):
- New `FlowFiles` dataclass: `{rates: [paths], imbalanceDataSourceOverride: ...}`
- New `FlowFilesField` marshmallow field accepts either
`gridToBatt: [a.json, b.json]` # legacy
or
`gridToBatt: {rates: [a.json], imbalanceDataSourceOverride: {price: ..., volume: ...}}`
- `RatesFiles` flows now typed `FlowFilesType` (always FlowFiles after parsing)
Plumbing:
- `parse_vol_rates_files_for_all_energy_flows` accepts
`flow_imbalance_pricings: Optional[Dict[str, pd.Series]]` keyed by flow name;
cache key now `(files_str, id(pricing))` so flows with same files but
different overrides don't share rate instances.
- `_get_rates_from_config` collects unique override sources, fetches +
normalises each (final-style for final-block overrides, live-style for
live-block), passes per-flow pricing dict down. Block-level df continues
to drive the canonical `imbalance_volume_*` output columns.
- Per-flow override + ratesDB combination rejected with a clear error.
- Override sources contribute to the flowsMarketData engine-needed gate.
Multiplier and OSAM safety:
- MultiplierVolRate.set_all_rates_in_set already operates per-flow, so
`Statkraft × imbalance` resolves to whatever imbalance ShapedVolRate is
in its flow's set — verified, no change needed.
- OSAM NCSP is dispatch-driven and computed once after the algo runs — per-flow
overrides don't touch it.
Report-side parser unchanged: `commands/report/rates.py` still passes a single
`imbalance_pricing` and per-flow override dict defaults to `{}`.
Tests: 4 new unit tests cover legacy list shape, dict shape with/without
override, and sibling-flow defaulting. Existing 34 tests pass unchanged.
…lation
Lets a single simulation declare multiple `finals: {<name>: Rates, ...}` and
get N output CSVs back, each settling the same dispatch under a different
final-rates structure. Removes the cut-and-paste duplication for
fullfcl-vs-trio-imbflex-style scenario pairs.
Schema (mutually exclusive with the legacy `final: Rates` field, mirrors
the `peak`/`peaks` precedent in PriceCurveAlgo):
rates:
live: *ratesLive_basecase
finals:
fullfcl: *ratesFinal_fullfcl
trio_imbflex: *ratesFinal_trio_imbflex
Implementation: parse-time fan-out in `parse_config`. Each multi-final sim is
replaced with N entries `<orig_name>.<variant_name>`, each carrying a deep-copied
SimulationCase with `rates.final` resolved to that variant. The simulator main
loop never sees a multi-final scenario — zero changes to `_run_one_simulation`,
`_get_rates_from_config`, `_process_final_rates`, or `generate_output_df`.
The optimiser runs N times; accepted trade-off for YAML ergonomics over compute
saving (deferred — see proposals/multi-final-rates.md for the column-multiplex
alternative).
CSV path de-clashing:
- Paths containing `$_SIM_NAME` are left alone — the substitution loop runs
after fan-out and naturally produces unique paths via `<orig>.<variant>`.
- Hardcoded paths get `.<variant>` inserted before the extension to avoid
two variants overwriting the same file.
Composes orthogonally with commit 2 — each variant is a complete `Rates` block
and can carry its own per-flow `imbalanceDataSourceOverride`.
Tests: 10 new unit tests cover variant suffix logic, fan-out expansion, deep-
copy isolation, declaration-order preservation, and the legacy single-final
no-op path. All 48 tests (34 existing + 4 FlowFiles + 10 fan-out) pass.
- pyproject.toml: 2.1.1 → 2.2.0 (minor bump for the three additive schema features on feature/multi-and-per-flow-rates) - CLAUDE.md merge log: summary entry for the v2.2.0 changes Three commits in this minor release: - feat: accept enabled:false on simulations (ea2d492) - feat: per-flow imbalanceDataSource override on RatesFiles (ca77995) - feat: multi-final fan-out — declare N settlement variants in one simulation (2188e35) All 48 unit + integration tests pass. Backward-compatible — legacy list-shape flows and single-`final` blocks unchanged.
- CHANGELOG.md: Added v2.2.0 entry with the three schema additions (enabled:false, per-flow imbalanceDataSourceOverride, multi-finals), YAML examples, and compatibility notes. - CLAUDE.md: Added a "simulate.yaml schema reference (post-v2.2.0)" cheatsheet under Key Concepts, summarising all three features with example YAML and the symmetry rules (apply override to live AND final; finals is mutually exclusive with final). - docs/proposals/per-flow-imbalance-source-override.md: Marked as ✅ IMPLEMENTED in v2.2.0, commit ca77995. - docs/proposals/multi-final-rates.md: Marked as ✅ IMPLEMENTED in v2.2.0, commit 2188e35 — with note that the shipped fan-out approach differs from the brief's column-multiplex design (deferred — user prioritised YAML ergonomics over compute saving).
Leftover from an earlier draft that used copy.deepcopy. The current test file uses SimpleNamespace mocks throughout. Caught by ruff in CI.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three additive schema features for
simulate.yaml. All backward-compatible — existing configs unchanged. Closes the structural Axle-premium leak on two-MPAN BSCP550 sites and removes cut-and-paste duplication for multi-settlement workflows.enabled: falseon a simulation drops it pre-schema, so externally-managed scenarios (e.g. Monte-Carlo runs whose outputs are produced outside skypro) can sit alongside live ones in the same yaml. Removes the rebuilder's temp-yaml staging workaround.imbalanceDataSourceOverrideinsideRatesFilesflows. Each flow can override the rates-block-levelimbalanceDataSourcevia a new dict shape; legacy list shape stays valid. Closes the HMCE Apr-26 BSCP550 ~£500–700/mo over-attribution where site-MPAN flows absorbed Axle premium intended only for BESS-MPAN flows.finalsdeclares N settlement variants per simulation. Mutually exclusive with the legacyfinal:field (peak/peaksprecedent). Fanned out at parse time into one expanded sim per variant; CSV paths get an automatic variant suffix when not using$_SIM_NAME.Commits (preserve as merge commit, not squash, so CHANGELOG SHAs resolve)
7b7caa6docs: design briefs for the two non-trivial changesea2d492feat: acceptenabled: falseon simulationsca77995feat: per-flowimbalanceDataSourceoverride onRatesFiles2188e35feat: multi-finalfan-outc54c7ccchore: bump version to 2.2.03502780docs: CHANGELOG entry + CLAUDE.md schema reference + mark proposals IMPLEMENTEDCompatibility
finalconfigs unchanged. Verified byintegrationTestPriceCurve,integrationTestPriceCurveMultiPeak,integrationTestPerfectHindsightLP— bit-identical LP output within tolerance0.01.ratesDBsource rejects per-flow overrides with a clear error (override only supported with the YAMLfilessource).YAML reference
Test plan
PYTHONPATH=src python -m unittest discover --start-directory src)FlowFilespolymorphic shape (legacy list, dict no-override, dict with override, sibling defaults)pip install -e .exercised on a real HMCE scenario; output matched the pre-2.2.0 baseline once a stale market-data refresh was identified as the apparent diff source.CLAUDE.mdagainst the actual YAML form by parsing test fixtures.Out of scope (follow-ups, not in this PR)
_stage_simulate_yaml_for_skypro()workaround inskypro-service— needs to wait for 2.2.0 on PyPI, thenskypro-servicebumps the dependency.data_hash(separate skypro-service ergonomic gap surfaced during release-day investigation; market-data refresh silently invalidates LP outputs without staleness detection).finals(deferred — see updateddocs/proposals/multi-final-rates.md).Release plan
After merge:
git checkout main && git pullgit tag -a v2.2.0 -m "v2.2.0 — multi-and-per-flow rates schema"git push origin v2.2.0rm -rf dist/ && python -m build) and publish (uv publish dist/* --token "$(cat ~/.simt/pypi.token)")pip install --upgrade skypro && skypro --version