ABM Task 3

# Separate Data Interface from Model Logic

**Type**: Refactor / Architecture  
**Priority**: High  
**Labels**: `architecture`, `refactor`, `good-first-project`, `intern`

---

## Background

The ABSiCE ABM is a well-functioning simulation, but as a result of organic growth over time, 
the model constructor (`ABM_CE_PV.__init__`) does too many things at once:

- It accepts **70+ raw parameters** as function arguments
- It **loads dozens of data files** (CSV, Excel, YAML) inline
- It **constructs file paths** by hand with string concatenation
- It **transforms data** (unit conversions, timestep scaling) mixed in with business logic
- **All of this lives in one 600-line `__init__` method**

This makes the simulation very hard to:
- Configure for a new scenario without reading the source code
- Test in isolation (you can't run the model without all the data files present)
- Maintain (changing a file path means hunting through `__init__`)
- Hand off to collaborators or students

The goal of this issue is to **cleanly separate three concerns**:

1. **Configuration** – what parameters control the simulation (YAML files)
2. **Data** – what external files feed the simulation (a typed data layer)
3. **Model logic** – the actual ABM mechanics (unchanged)

> ⚠️ **Scope note**: A CLI runner is a planned follow-up task. This issue is only about the
> internal architecture — do not add a CLI here.

---

## Goals

- [ ] All simulation parameters live in human-readable, version-controlled YAML files
- [ ] Users can run a new scenario by editing a YAML file, not Python code
- [ ] All data file loading is isolated in a single, testable `DataLoader` class
- [ ] The model `__init__` receives **typed configuration objects**, not 70 raw arguments
- [ ] Existing simulation results do not change (this is a pure refactor)

---

## Proposed Architecture

```
ABSiCE/
│
├── config/                        ← NEW: user-facing YAML files (edit these to change a scenario)
│   ├── default_simulation.yaml    ← simulation parameters
│   └── data_paths.yaml            ← file paths for all data sources
│
├── absice/                        ← NEW: importable Python package
│   ├── __init__.py
│   ├── schemas/                   ← Pydantic classes that parse & validate the config/ YAML files
│   │   ├── __init__.py
│   │   ├── simulation_config.py
│   │   └── data_paths_config.py
│   ├── data/                      ← all file I/O lives here
│   │   ├── __init__.py
│   │   └── data_loader.py
│   └── model/
│       ├── __init__.py
│       └── ABM_CE_PV_Model.py     ← model receives schema objects, not raw params
│
├── ABM_CE_PV_ConsumerAgents.py    ← unchanged for now
├── ABM_CE_PV_RecyclerAgents.py    ← unchanged for now
│   ... (other agent files unchanged)
└── ABM_CE_PV_MultipleRun.py       ← updated to load YAML and pass schema objects
```

**The distinction in one sentence**: `config/` contains the YAML files *users edit*; `absice/schemas/` contains the Python classes that *read and validate* those files.

> **Note**: Moving files into `absice/` is optional for this iteration. If it feels like too
> much churn, you can keep everything at the top level and just add the new `config/` and
> `data/` modules. Discuss with the senior dev before deciding.

---

## Phase 1 — Extract Configuration into Pydantic Models and YAML

### 1a. Understand what a Pydantic model is

[Pydantic](https://docs.pydantic.dev/latest/) lets you define data classes that **validate
their fields automatically**. If you pass a string where a float is expected, you get a clear
error immediately, not a confusing crash 200 lines later.

```python
# Example: instead of passing w_sn_eol=0.23 as a raw float, you define:
from pydantic import BaseModel, Field

class TpbWeights(BaseModel):
    w_sn_eol: float = Field(default=0.23, ge=0.0, le=1.0,
                            description="Weight of subjective norm for EoL decisions")
    w_pbc_eol: float = Field(default=0.44, ge=0.0, le=1.0)
    w_a_eol:   float = Field(default=0.59, ge=0.0, le=1.0)
    w_sn_reuse:  float = Field(default=0.497, ge=0.0, le=1.0)
    w_pbc_reuse: float = Field(default=0.382, ge=0.0, le=1.0)
    w_a_reuse:   float = Field(default=0.464, ge=0.0, le=1.0)

# Now if someone writes w_sn_eol=2.5 (a weight > 1, which is nonsensical),
# Pydantic raises a clear ValidationError before the model even starts.
```

### 1b. Create `absice/schemas/simulation_config.py`

Split the 70+ `__init__` parameters into logical Pydantic sub-models, then compose them
into one top-level `SimulationConfig`. Each sub-model should cover one conceptual area.

**Suggested groupings** (derive from the existing `__init__` signature):

```python
# absice/schemas/simulation_config.py

from pydantic import BaseModel, Field, field_validator
# Import the enums that already exist in utils — do NOT redefine them here.
from utils import TIMESTEP, ConsumerAgentResolution


# ── helpers ───────────────────────────────────────────────────────────────────

def _parse_timestep(v: object) -> TIMESTEP:
    """
    Coerce a YAML string like "annual" to the TIMESTEP enum.
    TIMESTEP uses integer values (ANNUAL=1, MONTHLY=12, QUARTERLY=4), so
    Pydantic cannot coerce the name automatically — we do it here.
    """
    if isinstance(v, TIMESTEP):
        return v
    if isinstance(v, str):
        try:
            return TIMESTEP[v.upper()]
        except KeyError:
            raise ValueError(f"Invalid timestep '{v}'. Choose from: "
                             f"{[m.name.lower() for m in TIMESTEP]}")
    raise TypeError(f"Expected str or TIMESTEP, got {type(v)}")

def _parse_resolution(v: object) -> ConsumerAgentResolution:
    """Coerce a YAML string like "site" to ConsumerAgentResolution."""
    if isinstance(v, ConsumerAgentResolution):
        return v
    if isinstance(v, str):
        try:
            return ConsumerAgentResolution[v.upper()]
        except KeyError:
            raise ValueError(f"Invalid resolution '{v}'. Choose from: "
                             f"{[m.name.lower() for m in ConsumerAgentResolution]}")
    raise TypeError(f"Expected str or ConsumerAgentResolution, got {type(v)}")


# ── sub-models (one per domain) ───────────────────────────────────────────────

class RunConfig(BaseModel):
    """Core simulation run settings."""
    seed:       int | None = None
    last_step:  int        = Field(default=31, gt=0)
    timestep:   TIMESTEP   = TIMESTEP.ANNUAL

    @field_validator("timestep", mode="before")
    @classmethod
    def _coerce_timestep(cls, v: object) -> TIMESTEP:
        return _parse_timestep(v)

class NetworkConfig(BaseModel):
    """Agent network topology."""
    num_consumers:            int   = Field(default=1000, gt=0)
    consumers_node_degree:    int   = Field(default=10,   gt=0)
    consumers_network_type:   str   = "small-world"
    rewiring_prob:            float = Field(default=0.1, ge=0.0, le=1.0)
    num_producers:            int   = Field(default=60,  gt=0)
    num_recyclers:            int   = Field(default=16,  gt=0)
    num_refurbishers:         int   = Field(default=15,  gt=0)
    prod_n_recyc_node_degree: int   = Field(default=5,   gt=0)
    prod_n_recyc_network_type: str  = "small-world"

class ConsumerConfig(BaseModel):
    """Consumer agent settings."""
    resolution:             ConsumerAgentResolution = ConsumerAgentResolution.SITE
    model_states:           list[str] | None = None
    consumers_distribution: dict[str, float]   = Field(
        default={"residential": 1.0, "commercial": 0.0, "utility": 0.0})
    product_distribution:   dict[str, float]   = Field(
        default={"residential": 1.0, "commercial": 0.0, "utility": 0.0})

    @field_validator("resolution", mode="before")
    @classmethod
    def _coerce_resolution(cls, v: object) -> ConsumerAgentResolution:
        return _parse_resolution(v)

class ProductConfig(BaseModel):
    """Product lifecycle parameters."""
    product_lifetime:          int   = Field(default=30, gt=0)
    product_growth:            list[float] = Field(default=[0.166, 0.045])
    growth_threshold:          int   = Field(default=10, gt=0)
    failure_rate_alpha:        list[float] = Field(default=[2.4928, 5.3759, 3.93495])
    mass_to_function_reg_coeff: float = 0.03
    max_storage:               list[float] = Field(default=[1, 8, 4])

class EolConfig(BaseModel):
    """End-of-life pathway settings."""
    init_eol_rate: dict[str, float] = Field(
        default={"repair": 0.005, "sell": 0.01, "recycle": 0.1,
                 "landfill": 0.885, "hoard": 0.0})
    all_eol_pathways: dict[str, bool] = Field(
        default={"repair": False, "sell": False, "recycle": True,
                 "landfill": True, "hoard": False})
    init_purchase_choice: dict[str, float] = Field(
        default={"new": 0.9995, "used": 0.0005, "certified": 0.0})
    purchase_choices: dict[str, bool] = Field(
        default={"new": True, "used": True, "certified": False})

class TpbConfig(BaseModel):
    """Theory of Planned Behavior weights."""
    theory_of_planned_behavior: dict[str, bool] = Field(
        default={"residential": True, "commercial": True, "utility": True})
    w_sn_eol:   float = Field(default=0.23, ge=0.0, le=1.0)
    w_pbc_eol:  float = Field(default=0.44, ge=0.0, le=1.0)
    w_a_eol:    float = Field(default=0.59, ge=0.0, le=1.0)
    w_sn_reuse: float = Field(default=0.497, ge=0.0, le=1.0)
    w_pbc_reuse: float = Field(default=0.382, ge=0.0, le=1.0)
    w_a_reuse:  float = Field(default=0.464, ge=0.0, le=1.0)
    att_distrib_param_eol:   list[float] = Field(default=[0.515, 0.1])
    att_distrib_param_reuse: list[float] = Field(default=[0.01, 0.185])
    extended_tpb: dict = Field(default={
        "Extended tpb": False, "w_convenience": 0.28,
        "w_knowledge": -0.51, "knowledge_distrib": [0.5, 0.49]})

class CostConfig(BaseModel):
    """All cost-related parameters."""
    landfill_cost:    list[float]       # 48 state-specific values $/ton
    hoarding_cost:    list[float]       = Field(default=[0, 130, 65])
    original_recycling_cost: list[float] = Field(default=[400-1e-6, 400+1e-6, 400])
    original_repairing_cost: list[float] = Field(default=[12987, 58442, 29870])
    hazardous_waste_management_cost: dict[str, float] = Field(
        default={"repair": 0.0, "sell": 0.0, "recycle": 0.0,
                 "landfill": 300.0, "hoard": 0.0})
    transportation_cost:           float = 0.095
    hazardous_transportation_cost: float = 0.095
    fsthand_mkt_pric:              float = 58442.0
    fsthand_mkt_pric_reg_param:    list[float] = Field(default=[1.0, 0.04])
    scndhand_mkt_pric_rate:        list[float] = Field(default=[0.4, 0.2])
    refurbisher_margin:            list[float] = Field(default=[0.4, 0.6, 0.5])
    repairability:                 float = Field(default=0.55, ge=0.0, le=1.0)
    used_product_substitution_rate: list[float] = Field(default=[0.6, 1, 0.8])
    imperfect_substitution:        float = 0.0
    recycling_learning_shape_factor:  float = -0.01
    repairing_learning_shape_factor:  float = -0.31
    sa_landfill_costs: tuple[bool, float] = (False, 481.0)

class MaterialConfig(BaseModel):
    """Material composition and market parameters."""
    product_mass_fractions: dict[str, float] = Field(default={
        "Product": 1.0, "Aluminum": 0.08, "Glass": 0.76, "Copper": 0.01,
        "Insulated cable": 0.012, "Silicon": 0.036, "Silver": 0.00032})
    established_scd_mkt: dict[str, bool] = Field(default={
        "Product": True, "Aluminum": True, "Glass": True, "Copper": True,
        "Insulated cable": True, "Silicon": False, "Silver": False})
    scd_mat_prices: dict[str, list] = ...   # triangular dist params [min, max, mode]
    virgin_mat_prices: dict[str, list] = ...
    material_waste_ratio: dict[str, float] = ...
    recovery_fractions: dict[str, float] = ...

class ScenarioConfig(BaseModel):
    """Optional/advanced scenario toggles."""
    epr_business_model: bool = False
    recycling_process: dict[str, bool] = Field(
        default={"frelp": False, "asu": False, "hybrid": False})
    dynamic_lifetime_model: dict = Field(default={
        "Dynamic lifetime": False, "d_lifetime_intercept": 15.9,
        "d_lifetime_reg_coeff": 0.87, "Seed": False,
        "Year": 5, "avg_lifetime": 50})
    seeding: dict = Field(default={
        "Seeding": False, "Year": 10, "number_seed": 50})
    seeding_recyc: dict = Field(default={
        "Seeding": False, "Year": 10, "number_seed": 50, "discount": 0.35})
    recycling_states: list[str] = Field(default=[
        "Texas", "Arizona", "Oregon", "Oklahoma",
        "Wisconsin", "Ohio", "Kentucky", "South Carolina"])
    hazardous_waste_regulation_enabled: bool = False
    landfill_solar_waste_acceptance_ratio: float = Field(
        default=0.4, ge=0.0, le=1.0)

class TclpConfig(BaseModel):
    """TCLP leach-test parameters."""
    bsf_mean:     float = 3.35
    bsf_std:      float = 1.17
    non_bsf_mean: float = 1.85
    non_bsf_std:  float = 0.97
    hazard_cutoff: dict[str, float] = Field(default_factory=lambda: {
        "federal": 5.0,
        **{s: 5.0 for s in ["AL","AZ","AR","CA","CO","CT","DE","FL","GA",
                             "ID","IL","IN","IA","KS","KY","LA","ME","MD",
                             "MA","MI","MN","MS","MO","MT","NE","NV","NH",
                             "NJ","NM","NY","NC","ND","OH","OK","OR","PA",
                             "RI","SC","SD","TN","TX","UT","VT","VA","WA",
                             "WV","WI","WY"]}})
    min_std: float = 0.05

class DataSourceConfig(BaseModel):
    """Feature flags that control which external data source is active."""
    rtn:         bool = False
    solar_cycle: bool = False
    use_pv_ice:  bool = False   # renamed from 'pv_ice' to avoid shadowing module


# ── top-level composed config ─────────────────────────────────────────────────

class SimulationConfig(BaseModel):
    """
    Top-level configuration for a full ABSiCE simulation run.
    Compose all sub-configs here.
    """
    run:          RunConfig         = Field(default_factory=RunConfig)
    network:      NetworkConfig     = Field(default_factory=NetworkConfig)
    consumer:     ConsumerConfig    = Field(default_factory=ConsumerConfig)
    product:      ProductConfig     = Field(default_factory=ProductConfig)
    eol:          EolConfig         = Field(default_factory=EolConfig)
    tpb:          TpbConfig         = Field(default_factory=TpbConfig)
    cost:         CostConfig        = ...   # required; no default (cost list is long)
    material:     MaterialConfig    = ...   # required
    scenario:     ScenarioConfig    = Field(default_factory=ScenarioConfig)
    tclp:         TclpConfig        = Field(default_factory=TclpConfig)
    data_source:  DataSourceConfig  = Field(default_factory=DataSourceConfig)

    @classmethod
    def from_yaml(cls, path: str) -> "SimulationConfig":
        """Load configuration from a YAML file."""
        import yaml
        with open(path) as f:
            raw = yaml.safe_load(f)
        return cls(**raw)

    def to_yaml(self, path: str) -> None:
        """Serialize configuration to a YAML file."""
        import yaml
        with open(path, "w") as f:
            yaml.dump(self.model_dump(), f, default_flow_style=False, sort_keys=False)
```

### 1c. Create `config/default_simulation.yaml`

This file is what users edit to change parameters. It should be fully commented.

```yaml
# config/default_simulation.yaml
# ─────────────────────────────────────────────────────────────────────────────
# ABSiCE Simulation Configuration
# Edit this file to change simulation parameters.
# All monetary values are in $/metric ton unless noted otherwise.
# ─────────────────────────────────────────────────────────────────────────────

run:
  seed: null          # set an integer for reproducible results, e.g. seed: 42
  last_step: 31       # number of time steps to simulate
  timestep: annual    # options: annual | quarterly | monthly

network:
  num_consumers: 1000
  consumers_node_degree: 10
  consumers_network_type: small-world   # small-world | random | cycle | scale-free
  rewiring_prob: 0.1
  num_producers: 60
  num_recyclers: 16
  num_refurbishers: 15
  prod_n_recyc_node_degree: 5
  prod_n_recyc_network_type: small-world

consumer:
  resolution: site      # site | pca
  model_states: null    # null = all US states; or list, e.g. [California, Texas]
  consumers_distribution:
    residential: 1.0
    commercial: 0.0
    utility: 0.0
  product_distribution:
    residential: 1.0
    commercial: 0.0
    utility: 0.0

# ... (truncated for brevity — the full YAML mirrors all Pydantic sub-models)
```

> **Tip**: Use `SimulationConfig(...).to_yaml("config/default_simulation.yaml")` to
> generate the first version of this file programmatically from the Pydantic defaults.
> That way you can't accidentally introduce a typo.

---

## Phase 2 — Separate Data Paths Config

Data file paths should **not** be hard-coded strings inside `__init__`. They should live in
their own config model and YAML file so that switching from one dataset to another (e.g., RTN
vs. default landfill data) is a one-line YAML change.

### 2a. Create `absice/schemas/data_paths_config.py`

```python
# absice/schemas/data_paths_config.py

from pathlib import Path
from pydantic import BaseModel, field_validator

class DataPathsConfig(BaseModel):
    """
    Resolved absolute paths to all external data files used by the simulation.
    
    Design rule: every field is a Path, not a str. Pydantic will coerce str → Path.
    Existence of the file is NOT validated here (that is the DataLoader's job),
    so that unit tests can construct a config without needing real files on disk.
    """
    # ── PV ICE / ReEDS ────────────────────────────────────────────────────────
    reeds_solar_futures:  Path
    reeds_std_scen24:     Path
    gis_centroids:        Path
    baseline_module_mass: Path
    baseline_module_energy: Path
    pvice_pca_merged_dir: Path    # directory, not a file

    # ── Distance matrices ─────────────────────────────────────────────────────
    recycler_distances:         Path
    landfill_distances:         Path
    hazardous_landfill_distances: Path
    uw_landfill_distances:      Path
    uw_recycler_distances:      Path

    # ── Facility data ─────────────────────────────────────────────────────────
    recycler_data:          Path
    landfill_data:          Path
    hazardous_landfill_data: Path
    uw_landfill_data:       Path
    uw_recycler_data:       Path

    # ── Market / auxiliary ────────────────────────────────────────────────────
    correct_mat_factor:     Path
    states_adjacency_matrix: Path
    uspvdb:                 Path   # only used when resolution == SITE

    # ── Policy ────────────────────────────────────────────────────────────────
    policy_by_state:        Path
    policy_schedule:        Path
    tclp_market_share:      Path
    generator_threshold:    Path
    uw_generator_threshold: Path

    @classmethod
    def from_yaml(cls, path: str) -> "DataPathsConfig":
        import yaml
        with open(path) as f:
            raw = yaml.safe_load(f)
        return cls(**raw)

    @classmethod
    def make_default(cls, project_root: Path) -> "DataPathsConfig":
        """
        Construct default paths relative to a known project root.
        This replaces the ad-hoc os.path.join(...) calls in Model.__init__.
        """
        temp = project_root / "TEMP"
        pvice = project_root / "PV_ICE"
        sup = pvice / "baselines" / "SupportingMaterial"
        pol = project_root / "policy_regulation"
        return cls(
            reeds_solar_futures = sup / "December Core Scenarios ReEDS Outputs Solar Futures v3a.xlsx",
            reeds_std_scen24    = project_root / "ReEDS" / "StdScen24_annual_balancingAreas_Mid_Case_CO2e_95by2035.xlsx",
            gis_centroids       = sup / "gis_centroid_n.csv",
            baseline_module_mass   = pvice / "baselines" / "baseline_modules_mass_US.csv",
            baseline_module_energy = pvice / "baselines" / "baseline_modules_energy.csv",
            pvice_pca_merged_dir   = pvice / "TEMP" / "PCA_merged",
            recycler_distances     = temp / "site_recycler_distances.csv",
            landfill_distances     = temp / "site_landfill_distances.csv",
            hazardous_landfill_distances = temp / "hazardous_site_landfill_distances.csv",
            uw_landfill_distances  = temp / "universal_waste_site_landfill_distances.csv",
            uw_recycler_distances  = temp / "universal_waste_site_recycler_distances.csv",
            recycler_data          = temp / "recycler_data.csv",
            landfill_data          = temp / "Landfills_data_2023.csv",
            hazardous_landfill_data = temp / "Landfills_data_SA.csv",
            uw_landfill_data       = temp / "Universal_Waste_Landfills_data.csv",
            uw_recycler_data       = temp / "Universal_Waste_Recyclers_data.csv",
            correct_mat_factor     = temp / "correct_mat_factor.csv",
            states_adjacency_matrix = project_root / "StatesAdjacencyMatrix.csv",
            uspvdb                 = project_root / "USPVDB" / "uspvdb_v3_0_20250430_with_pca.xlsx",
            policy_by_state        = pol / "policy_by_state.csv",
            policy_schedule        = pol / "policy_schedule.yaml",
            tclp_market_share      = pol / "tclp_market_share_interpolated.csv",
            generator_threshold    = pol / "generator_threshold.csv",
            uw_generator_threshold = pol / "universal_waste_generator_threshold.csv",
        )
```

### 2b. Create `config/data_paths.yaml`

Users who clone the repo and put data files in a custom location can override just the paths
they need to change, without touching any Python.

```yaml
# config/data_paths.yaml
# All paths are relative to the project root unless they start with /

reeds_solar_futures: PV_ICE/baselines/SupportingMaterial/December Core Scenarios ReEDS Outputs Solar Futures v3a.xlsx
reeds_std_scen24:    ReEDS/StdScen24_annual_balancingAreas_Mid_Case_CO2e_95by2035.xlsx
gis_centroids:       PV_ICE/baselines/SupportingMaterial/gis_centroid_n.csv

recycler_data:       TEMP/recycler_data.csv
landfill_data:       TEMP/Landfills_data_2023.csv
# ... etc.
```

---

## Phase 3 — Create a `DataLoader` Class

All file I/O should live in one place. The `DataLoader`:
- Receives a `DataPathsConfig` so it knows where to find files
- Exposes one method per logical dataset (easy to mock in tests)
- Returns `pandas.DataFrame` or typed objects — never raw file handles

```python
# absice/data/data_loader.py

from pathlib import Path
import pandas as pd
from pydantic import BaseModel, ConfigDict
from absice.schemas.data_paths_config import DataPathsConfig


class LoadedData(BaseModel):
    """
    Typed container for all DataFrames the model needs at startup.
    Passed as a single argument to ABM_CE_PV.__init__.

    model_config sets arbitrary_types_allowed=True because pd.DataFrame
    is not a type Pydantic knows how to validate by default.

    frozen=True prevents *field reassignment* on this object
    (e.g. ``data.recycler_data = other_df`` will raise an error).
    It does NOT prevent mutating the DataFrames themselves — agents that
    modify a DataFrame in-place (e.g. ``df.iloc[0] = value``) are
    unaffected.  In practice the model does
    ``self.recycler_data = data.recycler_data`` in __init__ and then
    works exclusively through ``self.*``, so frozen=True is safe.
    """
    model_config = ConfigDict(arbitrary_types_allowed=True, frozen=True)

    reeds_raw:                     pd.DataFrame
    gis_centroids:                 pd.DataFrame
    recycler_data:                 pd.DataFrame
    landfill_cost_df:              pd.DataFrame
    hazardous_landfill_cost_df:    pd.DataFrame
    uw_landfill_data:              pd.DataFrame
    uw_recycler_data:              pd.DataFrame
    recycler_distance_df:          pd.DataFrame
    landfill_distance_df:          pd.DataFrame
    hazardous_landfill_distance_df: pd.DataFrame
    uw_landfill_distance_df:       pd.DataFrame
    uw_recycler_distance_df:       pd.DataFrame
    correct_mat_factor:            pd.DataFrame
    states_adjacency_matrix:       pd.DataFrame
    pvice_waste_eol_df:            pd.DataFrame
    tclp_market_share_df:          pd.DataFrame
    policy_schedule_by_state:      dict
    uspvdb:                        pd.DataFrame | None   # None when resolution == PCA
    reeds_data:                    pd.DataFrame | None   # None when resolution == PCA


class DataLoader:
    """
    Loads all external data files needed by the simulation.
    
    By keeping all I/O here, the Model class itself becomes pure logic:
    it never calls pd.read_csv() directly.
    
    Parameters
    ----------
    paths : DataPathsConfig
        Resolved paths to every data file.
    """

    def __init__(self, paths: DataPathsConfig) -> None:
        self._paths = paths

    def load_all(self, resolution: str, model_states: list[str] | None = None) -> LoadedData:
        """
        Load every dataset and return a single LoadedData bundle.
        
        Parameters
        ----------
        resolution : str
            "site" or "pca" — determines which distance matrices and USPVDB files to load.
        model_states : list[str] | None
            If provided, filter spatial data to these US states only.
        """
        return LoadedData(
            reeds_raw              = self._load_reeds(),
            gis_centroids          = self._load_gis(),
            recycler_data          = self._load_recycler_data(),
            landfill_cost_df       = self._load_landfill_data(),
            hazardous_landfill_cost_df = self._load_hazardous_landfill_data(),
            uw_landfill_data       = self._load_csv(self._paths.uw_landfill_data),
            uw_recycler_data       = self._load_csv(self._paths.uw_recycler_data),
            recycler_distance_df   = self._load_csv(self._paths.recycler_distances),
            landfill_distance_df   = self._load_csv(self._paths.landfill_distances),
            hazardous_landfill_distance_df = self._load_csv(
                                        self._paths.hazardous_landfill_distances),
            uw_landfill_distance_df = self._load_csv(self._paths.uw_landfill_distances),
            uw_recycler_distance_df = self._load_csv(self._paths.uw_recycler_distances),
            correct_mat_factor     = self._load_csv(self._paths.correct_mat_factor),
            states_adjacency_matrix = self._load_csv(
                                        self._paths.states_adjacency_matrix),
            pvice_waste_eol_df     = self._load_pvice_waste_eol(),
            tclp_market_share_df   = self._load_csv(self._paths.tclp_market_share),
            policy_schedule_by_state = self._load_policy_schedule(),
            uspvdb  = self._load_uspvdb(model_states) if resolution == "site" else None,
            reeds_data = self._load_reeds_balancing_areas(model_states)
                         if resolution == "site" else None,
        )

    # ── private helpers (one per file) ───────────────────────────────────────

    def _load_csv(self, path: Path) -> pd.DataFrame:
        self._assert_exists(path)
        return pd.read_csv(path)

    def _load_reeds(self) -> pd.DataFrame:
        self._assert_exists(self._paths.reeds_solar_futures)
        df = pd.read_excel(self._paths.reeds_solar_futures,
                           sheet_name="new installs PV")
        df.drop(columns=["Tech"], inplace=True)
        df.set_index(["Scenario", "Year", "PCA", "State"], inplace=True)
        return df

    def _load_gis(self) -> pd.DataFrame:
        df = self._load_csv(self._paths.gis_centroids)
        return df.set_index("id")

    def _load_recycler_data(self) -> pd.DataFrame:
        return self._load_csv(self._paths.recycler_data)

    def _load_landfill_data(self) -> pd.DataFrame:
        return self._load_csv(self._paths.landfill_data)

    def _load_hazardous_landfill_data(self) -> pd.DataFrame:
        return self._load_csv(self._paths.hazardous_landfill_data)

    def _load_pvice_waste_eol(self) -> pd.DataFrame:
        path = self._paths.pvice_pca_merged_dir / "PVICE_PCA_WasteEOL_by_Year_and_PCA.csv"
        return self._load_csv(path)

    def _load_uspvdb(self, model_states: list[str] | None) -> pd.DataFrame:
        self._assert_exists(self._paths.uspvdb)
        df = pd.read_excel(self._paths.uspvdb)
        if model_states is not None:
            df = df[df["p_state"].isin(model_states)]
        return df

    def _load_reeds_balancing_areas(self, model_states: list[str] | None) -> pd.DataFrame:
        self._assert_exists(self._paths.reeds_std_scen24)
        df = pd.read_excel(self._paths.reeds_std_scen24)
        if model_states is not None:
            df = df[df["state"].isin(model_states)]
        df["utility_scale_pv_contribution_factor"] = (
            df["upv_MW"] / (df["upv_MW"] + df["distpv_MW"])
        )
        return df

    def _load_policy_schedule(self) -> dict:
        """Return the same structure that _load_policy_schedule_by_state() produces today."""
        import yaml
        path = self._paths.policy_schedule
        if not path.exists():
            return {}
        with open(path) as f:
            config = yaml.safe_load(f) or {}
        raw_policies = config.get("policies") or {}
        schedule_by_state: dict[str, dict] = {}
        for policy_name, entries in raw_policies.items():
            if not entries:
                continue
            for entry in entries:
                states = entry.get("states", [])
                entry_schedule = {k: v for k, v in entry.items() if k != "states"}
                for state in states:
                    schedule_by_state.setdefault(state, {})[policy_name] = entry_schedule
        return schedule_by_state

    @staticmethod
    def _assert_exists(path: Path) -> None:
        if not path.exists():
            raise FileNotFoundError(
                f"Required data file not found: {path}\n"
                f"Check your data_paths.yaml or DataPathsConfig."
            )
```

---

## Phase 4 — Update `ABM_CE_PV.__init__` to Accept Config Objects

Once Phases 1–3 are done, the model signature becomes clean and simple:

```python
# Before (current state):
model = ABM_CE_PV(
    seed=42,
    num_consumers=1000,
    w_sn_eol=0.23,
    w_pbc_eol=0.44,
    # ... 65 more arguments ...
)

# After (target state):
from absice.schemas.simulation_config import SimulationConfig
from absice.schemas.data_paths_config import DataPathsConfig
from absice.data.data_loader import DataLoader

config = SimulationConfig.from_yaml("config/default_simulation.yaml")
paths  = DataPathsConfig.make_default(project_root=Path("."))
data   = DataLoader(paths).load_all(
    resolution=config.consumer.resolution,
    model_states=config.consumer.model_states,
)
model  = ABM_CE_PV(config=config, data=data)
```

### Change to `ABM_CE_PV.__init__`:

```python
def __init__(self, config: SimulationConfig, data: LoadedData) -> None:
    """
    Initialise the model.

    Parameters
    ----------
    config : SimulationConfig
        All simulation parameters loaded from YAML.
    data : LoadedData
        All pre-loaded DataFrames (produced by DataLoader.load_all()).
    """
    super().__init__(seed=config.run.seed)
    self.config = config   # keep a reference for agents that need it
    self.data   = data     # keep a reference for data access

    # replace 70+ local variable assignments with attribute access:
    self.seed      = config.run.seed
    self.timestep  = config.run.timestep   # already a TIMESTEP enum
    self.last_step = config.run.last_step
    # ... etc.

    # Data frames are already loaded — no pd.read_csv() here:
    self.recycler_data          = data.recycler_data
    self.landfill_cost_df       = data.landfill_cost_df
    self.recycler_distance_df   = data.recycler_distance_df
    # ... etc.
```

---

## Phase 5 — Update `ABM_CE_PV_MultipleRun.py`

The runner file becomes very small:

```python
# ABM_CE_PV_MultipleRun.py  (new version)

from pathlib import Path
from absice.schemas.simulation_config import SimulationConfig
from absice.schemas.data_paths_config import DataPathsConfig
from absice.data.data_loader import DataLoader
from ABM_CE_PV_Model import ABM_CE_PV

PROJECT_ROOT = Path(__file__).parent

def run(config_path: str = "config/default_simulation.yaml") -> None:
    config = SimulationConfig.from_yaml(config_path)
    paths  = DataPathsConfig.make_default(PROJECT_ROOT)
    data   = DataLoader(paths).load_all(
        resolution=config.consumer.resolution.value,
        model_states=config.consumer.model_states,
    )
    model = ABM_CE_PV(config=config, data=data)
    for step in range(config.run.last_step):
        model.step()

if __name__ == "__main__":
    run()
```

---

## Phase 6 — Write Tests

For each new component write at least one test. Tests live in `test/`.

```python
# test/test_simulation_config.py

from absice.schemas.simulation_config import SimulationConfig, CostConfig

def test_default_config_is_valid():
    """SimulationConfig can be constructed with all defaults."""
    # CostConfig requires the cost list explicitly (no default), so we pass it.
    cost = CostConfig(landfill_cost=[500.0] * 48, scd_mat_prices={...}, ...)
    cfg = SimulationConfig(cost=cost)
    assert cfg.run.last_step == 31

def test_round_trip_yaml(tmp_path):
    """Config serialised to YAML and loaded back is identical."""
    cost = CostConfig(landfill_cost=[500.0] * 48, ...)
    cfg = SimulationConfig(cost=cost)
    yaml_path = str(tmp_path / "test_config.yaml")
    cfg.to_yaml(yaml_path)
    loaded = SimulationConfig.from_yaml(yaml_path)
    assert loaded == cfg

def test_invalid_tpb_weight_raises():
    """Pydantic rejects a TPB weight outside [0, 1]."""
    from pydantic import ValidationError
    import pytest
    with pytest.raises(ValidationError):
        from absice.schemas.simulation_config import TpbConfig
        TpbConfig(w_sn_eol=2.5)   # > 1.0 should fail


# test/test_data_loader.py

from pathlib import Path
import pandas as pd
from unittest.mock import patch
from absice.data.data_loader import DataLoader
from absice.schemas.data_paths_config import DataPathsConfig

def test_load_all_calls_correct_files(tmp_path):
    """DataLoader.load_all uses the paths from DataPathsConfig."""
    # This test uses mock files so it works without real data.
    # Create minimal stand-in CSV files:
    (tmp_path / "recycler_data.csv").write_text("Recycler Name,Latitude,Longitude\nFoo,30,-90\n")
    # ... create other minimal CSVs ...
    
    paths = DataPathsConfig(
        recycler_data=tmp_path / "recycler_data.csv",
        # ... other paths ...
    )
    loader = DataLoader(paths)
    df = loader._load_recycler_data()
    assert "Recycler Name" in df.columns
```

---

## Step-by-Step Checklist

Work through this in order. Each step can be a separate commit.

- [x] **Step 1**: Install Pydantic v2 (`pip install pydantic>=2.0`) and add it to
  `environment.yml` / `pv_abm_env.yml`
- [ ] **Step 2**: Create `absice/schemas/__init__.py` (empty file to make it a package)
- [ ] **Step 3**: Implement `SimulationConfig` in `absice/schemas/simulation_config.py`
  — start with just `RunConfig` and `NetworkConfig`, get them to load from YAML, then
  add the remaining sub-models one at a time
- [ ] **Step 4**: Generate `config/default_simulation.yaml` by calling
  `SimulationConfig(...).to_yaml(...)` in a throw-away script
- [ ] **Step 5**: Implement `DataPathsConfig` in `absice/schemas/data_paths_config.py`
- [ ] **Step 6**: Create `config/data_paths.yaml` with default paths
- [ ] **Step 7**: Create `absice/data/__init__.py` (empty file to make it a package)
- [ ] **Step 8**: Implement `DataLoader` + `LoadedData` in `absice/data/data_loader.py`
  — start with two or three methods, make sure they work, then add the rest
- [ ] **Step 9**: Add the `ABM_CE_PV(config, data)` constructor, replacing the old
  70-argument signature
- [ ] **Step 10**: Update `ABM_CE_PV_MultipleRun.py` to use the new interface
- [ ] **Step 11**: Write tests in `test/` for config round-trip and DataLoader
- [ ] **Step 12**: Run the full simulation **before** starting the refactor and save the
  output (e.g. `results/baseline_before_refactor.csv`). After the refactor is complete,
  run again and confirm the two outputs match numerically (use `compare_results.py` or
  a quick `pandas` diff)
- [ ] **Step 13**: Open a PR and request a code review

---

## Coding Standards to Follow

These apply to all new files you create.

1. **Type-annotate everything** — every function parameter and return value.
   ```python
   # Bad
   def load(path):
       return pd.read_csv(path)
   
   # Good
   def load(path: Path) -> pd.DataFrame:
       return pd.read_csv(path)
   ```

2. **One responsibility per class** — `DataLoader` loads data, it does not transform it.
   Transformations (timestep scaling, unit conversion) stay in the model or a dedicated
   `transforms.py`.

3. **Fail fast with clear messages** — the `_assert_exists` helper in `DataLoader` is a
   good pattern. Use it everywhere a missing file would otherwise produce a confusing
   error 100 lines later.

4. **No magic strings** — file column names that appear in multiple places should be
   constants:
   ```python
  # absice/schemas/constants.py
   RECYCLER_NAME_COL = "Recycler Name"
   ```

5. **Commit often with descriptive messages**:
   ```
   feat(config): add RunConfig and NetworkConfig Pydantic models
   feat(config): add default_simulation.yaml generated from defaults
   feat(data): implement DataLoader._load_recycler_data
   test(config): add round-trip YAML serialisation test
   ```

6. **Keep commits small and reviewable** — one logical change per commit, not
   "refactor everything in one go".

---

## Out of Scope (Do NOT Do in This PR)

- Implementing a CLI (`argparse`, `click`, etc.) — that is a separate issue
- Changing agent logic in `ABM_CE_PV_ConsumerAgents.py` etc.
- Performance optimizations
- Adding new simulation features
- Moving files to a new directory structure (optional, discuss first)

---

## Definition of Done

- [ ] All new code passes `flake8` with no errors
- [ ] Simulation results with the new interface match a baseline run produced by the
  old interface numerically (run before/after and diff with `compare_results.py`)
- [ ] At least five passing tests in `test/`
- [ ] `config/default_simulation.yaml` contains all parameters with inline comments
- [ ] `README.md` updated with a short "How to configure a run" section
- [ ] PR description explains the before/after for anyone reading the diff

---

## References

- [Pydantic v2 docs — Getting started](https://docs.pydantic.dev/latest/concepts/models/)
- [Pydantic v2 — `Field()` validators](https://docs.pydantic.dev/latest/concepts/fields/)
- [Python `dataclasses` module](https://docs.python.org/3/library/dataclasses.html)
  (useful background reading; we use Pydantic throughout this project instead)
- [YAML in Python with `PyYAML`](https://pyyaml.org/wiki/PyYAMLDocumentation)
- [Separation of concerns (Wikipedia)](https://en.wikipedia.org/wiki/Separation_of_concerns)
- Existing example in this repo: `policy_regulation/policy_schedule.yaml` + the
  `_load_policy_schedule_by_state()` method in `ABM_CE_PV_Model.py` — this is already
  a good pattern you can extend.


Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ABM Task 3 #28

Separate Data Interface from Model Logic

Background

Goals

Proposed Architecture

Phase 1 — Extract Configuration into Pydantic Models and YAML

1a. Understand what a Pydantic model is

1b. Create `absice/schemas/simulation_config.py`

1c. Create `config/default_simulation.yaml`

Phase 2 — Separate Data Paths Config

2a. Create `absice/schemas/data_paths_config.py`

2b. Create `config/data_paths.yaml`

Phase 3 — Create a `DataLoader` Class

Phase 4 — Update `ABM_CE_PV.init` to Accept Config Objects

Change to `ABM_CE_PV.init`:

Phase 5 — Update `ABM_CE_PV_MultipleRun.py`

Phase 6 — Write Tests

Step-by-Step Checklist

Coding Standards to Follow

absice/schemas/constants.py

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Uh oh!

ABM Task 3 #28

Description

Separate Data Interface from Model Logic

Background

Goals

Proposed Architecture

Phase 1 — Extract Configuration into Pydantic Models and YAML

1a. Understand what a Pydantic model is

1b. Create absice/schemas/simulation_config.py

1c. Create config/default_simulation.yaml

Phase 2 — Separate Data Paths Config

2a. Create absice/schemas/data_paths_config.py

2b. Create config/data_paths.yaml

Phase 3 — Create a DataLoader Class

Phase 4 — Update ABM_CE_PV.__init__ to Accept Config Objects

Change to ABM_CE_PV.__init__:

Phase 5 — Update ABM_CE_PV_MultipleRun.py

Phase 6 — Write Tests

Step-by-Step Checklist

Coding Standards to Follow

absice/schemas/constants.py

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions

1b. Create `absice/schemas/simulation_config.py`

1c. Create `config/default_simulation.yaml`

2a. Create `absice/schemas/data_paths_config.py`

2b. Create `config/data_paths.yaml`

Phase 3 — Create a `DataLoader` Class

Phase 4 — Update `ABM_CE_PV.init` to Accept Config Objects

Change to `ABM_CE_PV.init`:

Phase 5 — Update `ABM_CE_PV_MultipleRun.py`