domains/spf: fall through to tunnels.json when DB has no tunnel rows

Operators who bootstrap the relay sidecar by hand-editing
`/etc/mailcue-sidecar/tunnels.json` (or via env vars) never populate
the `tunnels` table.  `_list_active_tunnel_hosts` would then return an
empty list and `_build_spf_expected` would fall back to the single-host
default (`v=spf1 mx a:<hostname> ~all`) even when the live apex DNS
correctly authorised every relay (`a:relay-us... a:relay-de... -all`).

Result: every page load showed a phantom drift on the apex SPF record
forever, on any deployment that didn't go through the API path.

Fix: when the DB has no enabled tunnels, read `tunnels.json` directly
and extract the enabled `host` fields.  This keeps the API path as the
preferred source of truth (DB wins when populated) but lets externally
managed sidecars surface the right SPF guidance without forcing the
operator to mirror their JSON into the API.

The reader is defensive: missing file, unparseable JSON, schema
mismatch all return an empty list.  SPF generation must never depend
on the sidecar's filesystem being available.

Also: scrub example.com placeholders into tunnel/README.md and the
new test fixtures (the previous draft used a contributor's real
production domain as illustrative data).

Author: ibnbd

backend/app/domains/service.py

@@ -4,6 +4,7 @@
 
 import asyncio
 import contextlib
+import json
 import logging
 import re
 import subprocess
@@ -462,12 +463,65 @@ def _reload() -> None:
 # ── High-level API ───────────────────────────────────────────────
 
 
-async def _list_active_tunnel_hosts(db: AsyncSession) -> list[str]:
-    """Return the deduped, ordered list of enabled tunnel ``endpoint_host``s.
+def _read_tunnel_hosts_from_json(path: str | Path) -> list[str]:
+    """Read enabled relay hosts from the sidecar's ``tunnels.json``.
+
+    The sidecar (``mailcue-relay-sidecar``) treats this file as its single
+    source of truth — it loads it on boot and reloads on inotify.  Any
+    relay listed here is one that outbound mail might actually egress
+    through, so the apex SPF must authorise it.
 
-    These are the relay hostnames that an outbound MAIL FROM uses, so they
-    are exactly the ``a:`` mechanisms the apex SPF record must authorise.
-    Empty list when no tunnels are configured (single-host deployment).
+    Returns an empty list when the file is missing, unreadable, or its
+    schema doesn't match.  Never raises — SPF generation must not depend
+    on the sidecar's filesystem being available.
+    """
+    try:
+        raw = Path(path).read_text(encoding="utf-8")
+    except (OSError, FileNotFoundError):
+        return []
+    try:
+        data = json.loads(raw)
+    except json.JSONDecodeError:
+        logger.warning("Could not parse tunnels.json at %s — invalid JSON.", path)
+        return []
+    entries = data.get("tunnels") if isinstance(data, dict) else None
+    if not isinstance(entries, list):
+        return []
+    seen: set[str] = set()
+    hosts: list[str] = []
+    for entry in entries:
+        if not isinstance(entry, dict):
+            continue
+        # ``enabled`` defaults to True so a hand-written file that omits
+        # the field is treated as live (matching the sidecar's behaviour).
+        if entry.get("enabled", True) is False:
+            continue
+        host_value = entry.get("host")
+        if not isinstance(host_value, str):
+            continue
+        host = host_value.strip().rstrip(".").lower()
+        if host and host not in seen:
+            seen.add(host)
+            hosts.append(host)
+    return hosts
+
+
+async def _list_active_tunnel_hosts(db: AsyncSession) -> list[str]:
+    """Return the deduped, ordered list of relay hostnames outbound mail
+    may egress through.
+
+    These are the relay hostnames an outbound MAIL FROM uses, so they are
+    exactly the ``a:`` mechanisms the apex SPF record must authorise.
+    Source-of-truth precedence:
+
+      1. The ``tunnels`` DB table — populated when an admin configures
+         tunnels via the API.
+      2. ``/etc/mailcue-sidecar/tunnels.json`` — populated by hand or by
+         older deployments that bootstrap the sidecar config out-of-band.
+         Read directly so externally-managed sidecars still get accurate
+         SPF guidance without forcing operators to mirror their config
+         into the DB.
+      3. Empty list — single-host deployment with no relays.
     """
     stmt = select(Tunnel).where(Tunnel.enabled.is_(True)).order_by(Tunnel.name)
     result = await db.execute(stmt)
@@ -478,7 +532,9 @@ async def _list_active_tunnel_hosts(db: AsyncSession) -> list[str]:
         if host and host not in seen:
             seen.add(host)
             hosts.append(host)
-    return hosts
+    if hosts:
+        return hosts
+    return await asyncio.to_thread(_read_tunnel_hosts_from_json, settings.tunnels_config_path)
 
 
 def _build_spf_expected(hostname: str, tunnel_hosts: list[str]) -> str:

backend/tests/test_domains.py

@@ -28,6 +28,7 @@
     _join_txt_rdata,
     _normalize_dkim_txt,
     _parse_zonefile_txt,
+    _read_tunnel_hosts_from_json,
 )
 from app.tunnels.models import Tunnel
 
@@ -498,6 +499,105 @@ async def test_spf_expected_includes_every_enabled_tunnel(
     assert body["has_drift"] is False
 
 
+# ── Tests: tunnels.json fallback for externally-managed sidecars ─
+
+
+def test_read_tunnel_hosts_from_json_returns_enabled_hosts(tmp_path: Any) -> None:
+    """``tunnels.json`` is the sidecar's source of truth — operators who
+    bootstrap it directly (rather than via the API) must still get an
+    SPF that authorises every relay."""
+    import json as _json
+
+    cfg = tmp_path / "tunnels.json"
+    cfg.write_text(
+        _json.dumps(
+            {
+                "version": 1,
+                "tunnels": [
+                    {"name": "us", "host": "Relay-US.EXAMPLE.com.", "enabled": True},
+                    {"name": "de", "host": "relay-de.example.com", "enabled": True},
+                    {"name": "old", "host": "relay-old.example.com", "enabled": False},
+                    # Duplicate host across entries — must dedupe.
+                    {"name": "us-2", "host": "relay-us.example.com"},
+                ],
+            }
+        )
+    )
+    hosts = _read_tunnel_hosts_from_json(cfg)
+    # Order preserved, dot-normalised + lowercased, disabled dropped, duplicate dropped.
+    assert hosts == ["relay-us.example.com", "relay-de.example.com"]
+
+
+def test_read_tunnel_hosts_from_json_handles_missing_and_garbage(tmp_path: Any) -> None:
+    """Must never raise: SPF generation has to keep working even when the
+    sidecar isn't deployed or its config is mid-rewrite."""
+    assert _read_tunnel_hosts_from_json(tmp_path / "does-not-exist.json") == []
+
+    bad = tmp_path / "tunnels.json"
+    bad.write_text("not json {")
+    assert _read_tunnel_hosts_from_json(bad) == []
+
+    # Schema-mismatched JSON (top-level list, no ``tunnels`` key, etc).
+    bad.write_text("[1, 2, 3]")
+    assert _read_tunnel_hosts_from_json(bad) == []
+    bad.write_text('{"tunnels": "not-a-list"}')
+    assert _read_tunnel_hosts_from_json(bad) == []
+
+
+async def test_spf_falls_through_to_tunnels_json_when_db_empty(
+    client: AsyncClient,
+    seed_domain: Domain,
+    monkeypatch: pytest.MonkeyPatch,
+    _engine_and_session: Any,
+    tmp_path: Any,
+) -> None:
+    """The actual user-reported regression: a deployment that bootstrapped
+    the sidecar by hand (no API calls) had zero ``Tunnel`` rows, so SPF
+    fell back to the single-host default even though the live DNS listed
+    multiple relays.  With the JSON fallback in place, the expected SPF
+    must reflect the relays declared in ``tunnels.json``."""
+    import json as _json
+
+    cfg = tmp_path / "tunnels.json"
+    cfg.write_text(
+        _json.dumps(
+            {
+                "version": 1,
+                "tunnels": [
+                    {"name": "us", "host": "relay-us.example.com", "enabled": True},
+                    {"name": "de", "host": "relay-de.example.com", "enabled": True},
+                ],
+            }
+        )
+    )
+
+    from app.config import settings as _settings
+
+    monkeypatch.setattr(_settings, "tunnels_config_path", str(cfg), raising=False)
+
+    expected_spf = "v=spf1 mx a:relay-us.example.com a:relay-de.example.com -all"
+    plan = _build_full_plan(
+        domain=seed_domain.name,
+        hostname="mail.example.com",
+        selector=seed_domain.dkim_selector,
+        dkim_value=seed_domain.dkim_public_key_txt or "",
+        spf_value=expected_spf,
+    )
+    _pin_hostname(monkeypatch)
+    _install_resolver_stub(monkeypatch, plan)
+
+    resp = await client.get(f"/api/v1/domains/{seed_domain.name}/dns-state")
+    assert resp.status_code == 200, resp.text
+    body = resp.json()
+    spf_rec = next(
+        r
+        for r in body["records"]
+        if r["record_type"] == "TXT" and r["hostname"] == seed_domain.name
+    )
+    assert spf_rec["expected_value"] == expected_spf
+    assert spf_rec["drift"] is False
+
+
 # ── Tests: MTA-STS id is operator-chosen ─────────────────────────
 
 

tunnel/README.md

@@ -133,15 +133,15 @@ Append (replace with the FQDN you set as rDNS for **this** VPS — see
 
 ```ini
 [Service]
-Environment=MAILCUE_EDGE_HELO_HOSTNAME=relay-us.olib.email
+Environment=MAILCUE_EDGE_HELO_HOSTNAME=relay-us.example.com
 ```
 
 Then `sudo systemctl restart mailcue-relay-edge`.
 
 You can also set it in `/etc/mailcue-edge/config.toml`:
 
 ```toml
-helo_hostname = "relay-us.olib.email"
+helo_hostname = "relay-us.example.com"
 ```
 
 ---
@@ -154,15 +154,15 @@ relay through needs **all five** of the following set up before it can
 deliver mail. Skipping any one of them produces a 5xx rejection that
 looks like a tunnel bug but isn't.
 
-For each VPS hostname (e.g. `relay-us.olib.email`):
+For each VPS hostname (e.g. `relay-us.example.com`):
 
 | # | Direction | What | Where set | Verify |
 |---|---|---|---|---|
-| 1 | **Forward (A)**: hostname → IPv4 | Your DNS provider | `dig +short A relay-us.olib.email` |
-| 2 | **Forward (AAAA)**: hostname → IPv6 (if the VPS has IPv6 — most do) | Your DNS provider | `dig +short AAAA relay-us.olib.email` |
-| 3 | **Reverse (PTR) v4**: IPv4 → hostname | OVH (or other VPS provider) manager | `dig +short -x 51.81.202.4` |
-| 4 | **Reverse (PTR) v6**: IPv6 → hostname | OVH manager (separate row from v4!) | `dig +short -x 2604:2dc0:202:300::24de` |
-| 5 | **SPF** for the apex domain (`olib.email`) covering both IP families of every relay | Your DNS provider | `dig +short TXT olib.email` |
+| 1 | **Forward (A)**: hostname → IPv4 | Your DNS provider | `dig +short A relay-us.example.com` |
+| 2 | **Forward (AAAA)**: hostname → IPv6 (if the VPS has IPv6 — most do) | Your DNS provider | `dig +short AAAA relay-us.example.com` |
+| 3 | **Reverse (PTR) v4**: IPv4 → hostname | OVH (or other VPS provider) manager | `dig +short -x 192.0.2.4` |
+| 4 | **Reverse (PTR) v6**: IPv6 → hostname | OVH manager (separate row from v4!) | `dig +short -x 2001:db8::24de` |
+| 5 | **SPF** for the apex domain (`example.com`) covering both IP families of every relay | Your DNS provider | `dig +short TXT example.com` |
 
 Two recurring traps:
 
@@ -171,14 +171,14 @@ Two recurring traps:
   rDNS unset or pointing at a generic `vps-…` domain. Gmail will use
   whichever family the VPS picks for the outbound connection (often v6
   if available) and reject if PTR doesn't match.
-- **SPF is set on the relay hostname (`relay-us.olib.email`) but not
-  on the apex (`olib.email`).** Gmail's SPF check runs against the
+- **SPF is set on the relay hostname (`relay-us.example.com`) but not
+  on the apex (`example.com`).** Gmail's SPF check runs against the
   envelope `MAIL FROM` domain. Real Mailcue traffic will use
-  `From: user@olib.email`, so the apex must authorize *every* relay
+  `From: user@example.com`, so the apex must authorize *every* relay
   IP. Use the `a:` mechanism — it covers both A and AAAA records:
 
   ```text
-  olib.email.   IN  TXT  "v=spf1 mx a:relay-us.olib.email a:relay-de.olib.email -all"
+  example.com.   IN  TXT  "v=spf1 mx a:relay-us.example.com a:relay-de.example.com -all"
   ```
 
   This automatically authorizes all IPs of all named relay hosts; you
@@ -349,7 +349,7 @@ Common causes (read these in order):
   Your apex SPF doesn't authorize the IP that actually carried the
   outbound connection. Re-check that the SPF record covers *both*
   IPv4 and IPv6 of the relay (use the `a:` mechanism, see "DNS
-  prerequisites" above). Then verify with `dig +short TXT olib.email`.
+  prerequisites" above). Then verify with `dig +short TXT example.com`.
 - **`550 5.7.1 ... does not meet IPv6 sending guidelines regarding
   PTR records`** — the IPv6 PTR for the source IP is missing or wrong.
   Set it in OVH manager (separate row from the IPv4 PTR), then re-test.