Architectural Refactor (Tiered Storage, Builder Pattern & Routing)

[PWhite-Eng]

Context & Motivation

Following the extensive architectural review in Discussion #191, this issue tracks the implementation of the modernized ramses_rf storage and device discovery architecture.

The goal is to eliminate read-after-write race conditions (the "Async Gap"), reduce SD card wear, centralize state into a Single Source of Truth, and transition device class generation to a dynamic Strategy/Builder pattern.

The work is divided into four sequential phases. Individual PRs will be opened for each phase to ensure scoped, reviewable, and safe merges.

---

Phase 1: Rotating Packet Logs & Write Buffer (The Flight Recorder)

Implement an independent, highly efficient logger for raw hex strings to preserve hardware and guarantee forensic data integrity.

• 1.1 Configurable Log Paths: Implement packet_log_path and packet_log_prefix configuration options.
• 1.2 Midnight Rotation & Cleanup: Implement automated log rotation at 00:00:00 and deletion of files exceeding the packet_log_retention_days threshold.
• 1.3 Application-Level Write Buffering: Accumulate raw packet strings in RAM and flush to disk only upon a defined size threshold, a time interval (packet_log_flush_interval), or system shutdown.

---

Phase 2: Database & State Architecture (The Storage Tier)

Deprecate distributed legacy state (device._msgs) in favor of a centralized, tiered storage engine utilizing SQLite and a RAM-First Write-Behind Cache.

• 2.1 Implement the "Fat" SQLite Backend: Refactor MessageIndex to MessageStore, adding a column to serialize and store fully decoded payloads using orjson.
• 2.2 Strategy Lookup Table: Implement a static configuration table to dynamically generate composite database keys (DeviceID, Command Code, ContextIndex), correctly handling complex codes like 2411 vs. standard codes.
• 2.3 RAM-First Write-Behind Cache (Middleware): Implement the "Hot State" RAM Hash Map (Tier 1) for $O(1)$ get() lookups. Wire the StorageWorker to update RAM synchronously and SQLite asynchronously.
• 2.4 Hydration & Disk Snapshots (Persistence): Create a background thread to snapshot the In-Memory SQLite DB to a physical ramses.db file periodically. Implement startup logic to rehydrate the RAM cache from this file.
• 2.5 Cutover & Legacy Cleanup: Update all entity properties to query MessageStore. Delete legacy device._msgs and device._msgz.

---

Phase 3: Device Identity & Discovery (The Builder Pattern)

Replace static inheritance with a Dynamic Strategy Pattern, allowing devices to safely upgrade their capabilities as RF evidence is gathered.

• 3.1 Dynamic Strategy Implementation: Create the immutable Device "shell" class that holds a mutable Strategy object (e.g., GenericStrategy upgrading to HoneywellTRVStrategy).
• 3.2 Event Sourcing for Replay: Ensure raw hex packets are stored in the SQLite persistence layer to allow newly promoted strategies to retroactively re-parse historical device states.
• 3.3 Known-List Seeding (Hydration): Integrate the known_list with the Builder. If a device is explicitly defined by the user in the config, immediately initialize it with its specialized strategy at startup.

---

Phase 4: Isolated Routing Logic

Separate transport decisions from state data to support reliable multi-gateway environments.

• 4.1 Routing Table Separation: Implement an internal routing table inside ramses_rf to track the best gateway for each device based purely on RSSI.
• 4.2 Write Path Protection: Enforce logic where the State DB prioritizes data freshness (latest timestamp) over signal quality, ensuring state is decoupled from transport metrics.