Problem
Currently, the Pipes Store (which holds the named SQL queries) only loads data from the NATS Key-Value (KV) store when a WaveHouse instance first boots up. In a clustered environment with multiple API or Worker nodes, this creates a synchronization lag: if an admin updates a Pipe on Node A, Node B will continue running the old, outdated SQL until it is manually restarted. This prevents the "Live Coding" experience where query changes should take effect across the entire cluster instantly.
Proposed Solution
Implement a KV Watcher pattern within the internal/pipes/store.go. Instead of a one-time "Get" during initialization, the Store should establish a long-lived subscription to the NATS KV bucket.
When a change occurs (Put or Delete), NATS will push a notification to every WaveHouse node. The Store will then reactively update its local in-memory cache. This ensures that every node in the cluster is perfectly synchronized within milliseconds of a Pipe being updated, without requiring any manual restarts or coordination.
Alternatives Considered
None
Additional Context
None
Problem
Currently, the Pipes Store (which holds the named SQL queries) only loads data from the NATS Key-Value (KV) store when a WaveHouse instance first boots up. In a clustered environment with multiple API or Worker nodes, this creates a synchronization lag: if an admin updates a Pipe on Node A, Node B will continue running the old, outdated SQL until it is manually restarted. This prevents the "Live Coding" experience where query changes should take effect across the entire cluster instantly.
Proposed Solution
Implement a KV Watcher pattern within the internal/pipes/store.go. Instead of a one-time "Get" during initialization, the Store should establish a long-lived subscription to the NATS KV bucket.
When a change occurs (Put or Delete), NATS will push a notification to every WaveHouse node. The Store will then reactively update its local in-memory cache. This ensures that every node in the cluster is perfectly synchronized within milliseconds of a Pipe being updated, without requiring any manual restarts or coordination.
Alternatives Considered
None
Additional Context
None