Cross-stream correlation & historical search (correlate by request_id + free-text search + api/sys persistence)

[ashish-jabble]

Background

fasten emits three observability streams, each tagged with a shared request_id that threads through a single logical operation:

Stream	Today's sink	History
audit	persistent store (SQLite)	full
api (api-log)	in-memory ring buffer	recent window only
sys (syslog)	in-memory ring buffer	recent window only

The reader router already exposes /logs/{sys,api,audit} with per-stream request_id filtering. There is no unified correlation read, no free-text search, and no persistence for api/sys — so as those rings churn, recent rows are overwritten and become unrecoverable.

Problem

A consumer holding one request_id cannot, in a single call, retrieve the correlated rows from all three streams. Worse, because api/sys live only in bounded rings, any correlation or search against past api/sys activity is impossible — the depth of correlation silently shrinks as log volume grows. Consumers are forced to either stitch multiple calls together themselves (logic that belongs in the library) or accept that "find everything for this operation" only works for the last few seconds of traffic.

Goals

• Correlation and search are fasten-owned primitives, exposed by the reader. Consumers call one API and render results — no stitching, no filtering logic on the consumer side.
• A request_id resolves to a complete correlated view across all three streams, bounded only by configured retention — not by ring size.
• Discovery (free-text) and correlation (request_id) are both reachable from the library, against persisted history, not just the live window.
• Library remains usable with zero new infrastructure for consumers that only want recent-window behavior (persistence is opt-in).

Non-goals (v1)

• Full-text indexing / relevance ranking (substring/ILIKE-class matching is sufficient).
• Distributed/multi-node aggregation (single fasten instance scope).
• Changing the request_id propagation mechanism itself (already works).

Functional requirements

FR1 — Optional persistence for api and sys streams

fasten must support persisting the api and sys streams to the same durable backend it already uses for audit, so they gain queryable history. Configurable (e.g. persist_streams = {audit, api, sys}) with retention per stream. When persistence is off, the stream falls back to ring-only (today's behavior) — fully backward compatible.

FR2 — Unified correlation read

A single reader endpoint returns all streams correlated by request_id:

GET /logs/correlate?request_id=<id>[&limit=N]
200 → {
  "request_id": "<id>",
  "audit": [ …rows ],
  "api":   [ …rows ],
  "sys":   [ …rows ],
  "counts": { "audit": n, "api": n, "sys": n },
  "completeness": {            // honest about source depth
    "audit": "store",
    "api":   "store" | "ring",
    "sys":   "store" | "ring"
  }
}

Internally reuses the existing per-stream query paths; adds no new query semantics beyond fan-out + assembly.

FR3 — Free-text search across streams

Each stream read (and a unified search) accepts an optional q= substring filter, applied server-side over the persisted records (not just the ring):

GET /logs/search?q=<text>[&streams=api,sys,audit][&since=…&until=…&limit=N]
200 → { "matches": [ { stream, request_id, ts, summary, row } … ], "counts": {…}, "completeness": {…} }

Purpose: a consumer can locate a row by something a human knows (error text, actor, path, device), obtain its request_id, and hand that to FR2. q is also accepted on the individual /logs/{sys,api,audit} endpoints.

FR4 — Completeness / truncation signaling

Every read that can be served from a ring must report, per stream, whether results came from the durable store or a bounded ring (see completeness above), so consumers can be honest about gaps instead of silently under-reporting.

FR5 — Reader parity (Python + Go)

The reader exists in both the Python and Go implementations. FR2–FR4 must be available in both (or, if phased, the requirement states which lands first and tracks the other). Storage/config (FR1) likewise.

Storage & API design considerations

• Persisted api/sys rows need indexes on request_id and ts (and ideally a text column for q), mirroring how audit is queried.
• Retention is per-stream (audit typically longest; api/sys shorter, higher volume). Expose as config.
• Ring buffers remain as the hot path / fallback; persistence is a write-through sink, not a replacement, so emit latency is unaffected.
• limit/offset and since/until consistent across all read endpoints.

Configuration knobs (proposed)

Key	Meaning	Default
persist_streams	which streams write to the durable store	{audit} (back-compat)
retention.<stream>	max age/rows per persisted stream	per-stream
search.enabled	enable q= / /logs/search	false until persistence on

Open questions

1. One table or per-stream tables? Unified events(stream, request_id, ts, level, summary, payload) vs separate tables per stream. Unified simplifies /search and /correlate; per-stream keeps schemas tight.
2. Backend beyond SQLite? Audit uses SQLite; high-volume api/sys persistence may warrant a pluggable store (Postgres/Timescale) — does v1 stay SQLite-only?
3. q matching scope — which columns are searchable (message/summary only, or full payload)? Case-insensitive substring assumed.
4. Python vs Go — land Python reader first, Go follows, or both together?
5. Naming — /logs/correlate & /logs/search, vs a combined /logs/investigate?request_id|q.

Acceptance criteria

• With persistence enabled, a request_id from hours/days ago returns full audit + api + sys via /logs/correlate, with completeness all store.
• With persistence disabled, the same call returns audit-from-store + api/sys-from-ring, with completeness honestly marking ring.
• /logs/search?q=… finds matching rows across persisted streams and returns their request_ids, suitable for a follow-up correlate.
• Existing /logs/{sys,api,audit} behavior is unchanged when the new config is absent.
• Tests cover: correlate-with-persistence, correlate-ring-fallback, search-by-text, completeness flags, retention pruning. Python + Go parity per FR5.

Use cases (acceptance scenarios)

① Recent correlation (works today, formalized): an operation just ran → search a known term, click the row → all three panes scope to that one request_id.

② Historical investigation (needs FR1): correlate a request_id from last week → audit is found on disk; with persistence the api/sys technical detail is also recoverable instead of ring-empty.

③ High-volume depth (needs FR1): under sustained traffic, correlate an operation from N minutes ago and still recover its api/sys rows rather than watching correlation depth shrink as the rings churn.

[praveen-garg]

Let's do this phase-wise. Split into sub-issues per phase so each can ship independently.

Phase 0: FR4 (completeness flags). One PR, no persistence work. Every read marks store vs ring per stream.

Phase 1: FR1 (per-stream persistence, default off) + FR2 (/logs/correlate fan-out) + indexed fields on api and sys (method, path, status, level, event). SQLite only. Per-stream tables, not a unified events table.

Phase 2: spec/persistent-streams.md (schema + canonical_form_id discipline), Postgres adapter, NL translator interface (NL parses to structured chips, not a new endpoint).

Phase 3: FR3 (free-text q=). Constrained or killed. If we ship: sys only, since= mandatory, result cap, no ranking, opt-in.

Two things the proposal should also call out:

Every row in every stream carries a request_id as a v1 invariant. Today sys has an orphan subset (boot, init, background workers, lost-context async). The logger shim should mint a sentinel id (boot-..., sched-..., bg-..., lib-..., orphan-...) when no real context is active. Otherwise /logs/correlate?request_id=X silently degrades to "fan out, hope the relevant sys lines weren't orphans."

Discovery is structured-fields-first. Operators arrive at a request_id from target/actor/code/method/path/time, not by pasting one. Indexing those fields on api and sys (cheap at write time) handles roughly 80% of cases. q= is the escape hatch for "I only have an error string" and should be sized accordingly.