Eliminate redundant dataset deep-copies in multi-year extension (11x faster)

[anth-volk]

Fixes #9001

Summary

Microsimulation() setup profiling showed ~half of init time was spent deep-copying the full dataset ~23 times during multi-year extension: extend_single_year_dataset deep-copies all six entity DataFrames once per projected year (2024–2035), then _apply_uprating deep-copies the entire multi-year set again. Uprated columns are overwritten immediately after being copied; carried-forward columns never change at all.

Under pandas copy-on-write (always on in pandas ≥ 3), year frames are now shallow copies of the base year: carried-forward columns share base-year buffers, and uprating's whole-column assignments materialize only the columns they replace. _apply_uprating's redundant defensive copy is skipped on the extension path (external callers keep the copying default).

Measured results (full Populace 2024 dataset, 12 years)

Metric	Before	After
Median extension wall-time (3 runs)	20.95s	1.90s (11×)
Per-column checksums (3,876 across all years/tables)	—	bit-identical
End-to-end Microsimulation() init	~30s	~8s

Equivalence was verified with pd.util.hash_pandas_object checksums of every column of every entity table for every extended year, identical before/after.

Impact on state-level runs

Each Microsimulation() build sheds ~19s of redundant copying (init ~27s → ~8s). A reform analysis constructing baseline + reform simulations saves ~40s — roughly 13–33% of a 2–5 minute state-level run.

Compatibility

• pandas pin: pandas>=3.0 on Python ≥ 3.11; Python 3.9/3.10 (retained for Census, Support Python 3.9 and 3.10 #8035) keep pandas>=2.0 via environment markers — mirroring policyengine-uk's tables marker pattern. uv lock resolution is unchanged (2.3.3 / 3.0.0).
• On pandas 2.x the extension falls back to deep copies (today's exact behavior); shallow copies without CoW alias mutably — verified empirically that in-place ops (*=, .loc) write through on 2.x, which is why the fast path is version-gated.
• Core-side aliasing is safe: holder._to_array casts float64→float32 at set_input, materializing fresh arrays, and pandas 3 hands out read-only .values views.

Tests

• New copy-semantics tests for copy(deep=...) on both dataset classes.
• Regression tests pin: exact equivalence with a deep-copy reference path, caller-dataset isolation, and the buffer sharing itself (so deep copies can't silently return), plus _apply_uprating's default defensive copy.

Commits

1. Marker-based pandas pin (no resolution change)
2. deep parameter on dataset copy methods (pure refactor, default unchanged)
3. Shallow extension + regression tests

🤖 Generated with Claude Code

[PavelMakarchuk]

Review (two focused reviewers: CoW correctness/aliasing, packaging/test quality)

Verdict: Approve — the shallow-copy mechanism is verified safe and the tests genuinely pin both the equivalence and the perf win.

✅ Verified

• No silent write-through paths on pandas ≥ 3 (checked empirically against pandas 3.0.0): .values on CoW views comes back read-only, so a raw in-place write raises ValueError loudly rather than corrupting sibling years; .loc/whole-column assignment are CoW-isolated. Traced the full consumer chain (extend_single_year_dataset → build_from_dataset → holder.set_input/InMemoryStorage.put) plus all in-repo mutators — set_input_divide_by_period copies first, the FLAT_FILE *= block operates on a copy, labor-supply blocks don't mutate frames.
• Version gate is correct and fails safe: pandas 2 + copy_on_write=True still deep-copies (only forfeits speed); the monkeypatched _PANDAS_COW global is read at call time so the fallback test is effective.
• _apply_uprating(copy=False) has no leaking call sites — and the base year now being copied too (datasets = [dataset.copy(...)]) fixes what on main was only safe by accident.
• Tests would catch a regression: test_extension_equals_deep_copy_reference rebuilds the old deep-copy path and compares every table/year with check_exact=True; the buffer-sharing test guards against deep copies silently creeping back. All fixture-based, no dataset download.
• Markers/lock: PEP 508 markers are valid (no lexicographic version trap), 3.9/3.10 branch is reachable (requires-python >=3.9), uv.lock consistent, no conflict with policyengine-core (pandas>=1). Downstream floors are permissive: policyengine.py pandas>=2.0.0, policyengine-us-data pandas>=2.3.1 — no <3 upper bounds, so the new floor resolves cleanly.

🟡 Worth stating explicitly / small follow-ups

1. The pandas>=3.0 floor is stricter than the code requires — the _PANDAS_COW runtime gate already makes pandas 2.x fully correct, so the floor is a deliberate trade-off (guaranteed perf + CI on pandas 3, at the cost of force-upgrading downstream envs onto pandas 3's own breaking changes). Fine either way, but worth a sentence in the PR/changelog since a patch-level .changed bump understates a dependency tightening.
2. Behavior change to document: holder input arrays become read-only under pandas 3 (true on the deep path too, since .values is a CoW view). Downstream code doing arr = holder.get_array(p); arr[mask] = x will now raise where it previously worked — a one-liner in the changelog would save someone a confusing afternoon.
3. Cheap CI win: the py3.9/3.10 compat legs are install+smoke-import only, so real pandas-2 aliasing semantics (the thing the gate defends against) never run in CI. The new tests are fixture-based and fast — adding pytest policyengine_us/tests/microsimulation/data/ to those legs would cover it for seconds of CI time.
4. copy(deep=False) is a public footgun on pandas 2: the safety gate lives in economic_assumptions, but the dataset classes are exported API — a py3.9/3.10 caller gets mutable aliasing with only a docstring warning. Consider raising or silently deepening inside copy() when CoW is unavailable (which would also fix the mild layering smell of tests importing the private _PANDAS_COW).

Nice work on the equivalence methodology — the checksum verification plus the reconstructed-reference regression test is exactly how this kind of change should be validated.