Serve the workspace UI over HTTP/2 (TLS + h2 at the mngr forward proxy)#2410
Merged
Conversation
Places the blueprint-generated implementation plan for terminating TLS + HTTP/2 at the `mngr forward` proxy under blueprint/mngr-forward-http2/, following the repo's blueprint/<slug>/plan-<slug>.md convention. Paths are repo-relative (libs/mngr_forward, apps/minds). Co-authored-by: Sculptor <sculptor@imbue.com>
Corrects the plan against the real code and folds in the Q&A decisions: single Python-spawned forward subprocess (not per-workspace); hypercorn asyncio worker with in-memory SSLContext override, fd:// handoff (os.dup to avoid double-close), shutdown_trigger=None; Secure on mngr_forward_session only (minds HTTP backend untouched); precise scheme-flip sites excluding the proxy->container WS rewrite; Python loopback probes use verify=False; hardcode --use-http2 on; existing failure surfacing plus a TLS/h2 cause log line; CLI docs regen; explicit cryptography dep; drop uvicorn. Co-authored-by: Sculptor <sculptor@imbue.com>
The workspace origin was served over plain HTTP/1.1, which Chromium caps at ~6 held-open connections per origin. Once a workspace had enough long-lived streams (one SSE per open chat tab, plus any `/service/` app's streams), the pool was exhausted and every further request queued indefinitely, hanging the UI even though the backend was healthy. `mngr forward` gains a `--use-http2` flag (off by default): when set it generates a fresh in-memory self-signed cert (SANs localhost, *.localhost, 127.0.0.1), terminates TLS, and negotiates HTTP/2 via ALPN, served by hypercorn (replacing uvicorn) with an fd:// socket handoff. Client-facing URLs flip to https/wss and the `mngr_forward_session` cookie is marked Secure; the proxy->container hop and WebSocket transport are unchanged. Minds runs its single proxy with `--use-http2` hardcoded on: the Python probe/ redirect URLs flip to https (loopback probes use verify=False), and the Electron shell builds https/wss workspace URLs, marks the cookie Secure, and silently trusts the self-signed cert for loopback hosts in the workspace-content partition only. The minds bare-origin backend stays plain HTTP (a deliberate, non-blocked mixed transport). Co-authored-by: Sculptor <sculptor@imbue.com>
The e2e workspace-ready check (`_AGENT_SUBDOMAIN_PATTERN`) was anchored to `http://`, so once the workspace origin flipped to `https` it never matched the ready URL and `build-minds-snapshot` hung the full 600s at a workspace that had actually loaded. Accept `https?://` (the bare minds backend origin stays plain http). Add an https regression test to `_wait_for_workspace_ready`. `_get_mngr_forward_origin()` (used by the sidebar/chrome/sharing templates to render `/goto/<agent>/` links) still hardcoded `http://localhost:<port>`, which would send a plaintext request at the now-TLS proxy. Derive the scheme from `MNGR_FORWARD_USE_HTTP2` so the rendered links reach the proxy. Also correct the blueprint plan's "never written to disk" wording: stdlib `ssl.load_cert_chain` needs a path, so the cert goes to a transient 0600 mkstemp file, loaded and unlinked immediately (never persisted). Co-authored-by: Sculptor <sculptor@imbue.com>
Running the app on this branch, the terminal / `/api/ws` sockets crashed the
`mngr forward` proxy with `TypeError: data must be str, bytes, iterable, or
async iterable` in the client->backend forwarder.
Root cause: `_forward_client_to_backend` selected the frame payload by ASGI
key presence (`if "text" in data`). hypercorn's `websocket.receive` events
ALWAYS include both `text` and `bytes` keys, setting the unused one to `None`
(uvicorn omitted it), so a binary frame arrives as `{"bytes": b"...", "text":
None}` -- the key-presence check picked `text=None` and forwarded `None`,
which the `websockets` client rejects. This surfaced now because Chromium
carries WebSockets over the h2 connection (hypercorn advertises RFC 8441
Extended CONNECT), so every WS frame flows through hypercorn's ASGI layer.
Fix: extract `_select_ws_receive_payload`, which selects by value (per the
ASGI spec: exactly one of text/bytes is non-None), and unit-test it with the
hypercorn dual-key shape. Correct the plan/changelog claim that WebSockets
negotiate a separate http/1.1 connection -- they ride the h2 connection.
Co-authored-by: Sculptor <sculptor@imbue.com>
The prior wording asserted as fact that Chromium carries WebSockets over the h2 connection (RFC 8441 Extended CONNECT). Only the server side was verified (hypercorn advertises SETTINGS_ENABLE_CONNECT_PROTOCOL); Chromium's actual choice was not confirmed, and the dual-key ASGI bug manifests identically whether the WS rides h2 or a separate connection, so the traceback is not evidence for the over-h2 claim. Reword the plan and changelog to state only what is verified: hypercorn is the ASGI server either way, and its receive events always carry both text and bytes keys (one None), which is why the forwarder must select by value. Co-authored-by: Sculptor <sculptor@imbue.com>
gnguralnick
commented
Jul 10, 2026
| "Check the logs above for its stderr and retry." | ||
| ) | ||
| logger.info(" mngr forward: http://127.0.0.1:{}", mngr_forward_port) | ||
| forward_scheme = "https" if MNGR_FORWARD_USE_HTTP2 else "http" |
Contributor
Author
There was a problem hiding this comment.
not sure if we need an env var for this or if it should just be a full update
| """Ephemeral in-memory TLS for ``mngr forward`` (self-signed cert + hypercorn config). | ||
|
|
||
| Used only when ``--use-http2`` is set: the proxy terminates TLS and negotiates | ||
| HTTP/2 (via ALPN) so the workspace origin is not capped by Chromium's |
Contributor
Author
There was a problem hiding this comment.
this docstring ("the workspace origin...") doesn't make sense to have here - mngr forward stands on its own
gnguralnick
marked this pull request as ready for review
July 10, 2026 19:36
…ls docstring Remove the MNGR_FORWARD_USE_HTTP2 toggle constant and hardcode the https/wss + Secure-cookie path everywhere on the minds side (the plugin --use-http2 flag stays off-by-default, so mngr_forward stands on its own). minds now always passes --use-http2 and the mngr_forward_started event no longer carries a redundant use_http2 field; Electron unconditionally builds https URLs, marks the cookie Secure, and trusts the loopback cert. Also rewrite the mngr_forward/tls.py module docstring to drop app-specific rationale (workspace origin, Chromium, minds Electron) that does not belong in the standalone plugin. Co-authored-by: Sculptor <sculptor@imbue.com>
gnguralnick
temporarily deployed
to
minds-ci-test
July 10, 2026 20:04 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
gnguralnick
temporarily deployed
to
minds-ci-test
July 12, 2026 23:11 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Problem: This branch changed libs/mngr_forward/pyproject.toml to add hypercorn (and declare cryptography explicitly) and drop uvicorn, and regenerated the root uv.lock to match, but left the minds electron bundle's committed lockfile (apps/minds/electron/pyproject/uv.lock) stale. That lock resolves the editable imbue-mngr-forward package, so it still listed uvicorn as a mngr_forward dependency and contained no hypercorn entry -- inconsistent with the source pyproject. A --frozen / --locked consumer of that lock would build a bundle missing hypercorn, and `mngr forward --use-http2` (which minds always runs) would fail to import hypercorn.asyncio. Fix: Ran `uv lock` in apps/minds/electron/pyproject (the same command build.js uses) to regenerate the lock against the current editable pyprojects. For mngr_forward this removes uvicorn (its sole consumer in this subset) and adds hypercorn plus its transitive deps (priority, wsproto). The regeneration also incidentally caught the lock up to two pre-existing direct minds deps that it had drifted behind (argon2-cffi, spectree); no existing package versions changed. `uv lock --check` passes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
gnguralnick
temporarily deployed
to
minds-ci-test
July 12, 2026 23:54 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Contributor
|
lgtm, thanks for fixing! this feels likely to break something in the installed version, will flag to weishi on this one |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
agent-<id>.localhost:<port>), which Chromium caps at ~6 held-open connections per origin. Once a workspace had enough long-lived streams (one SSE per open chat tab, plus any/service/app's own streams), the pool was exhausted and every further request — including the plain GETs that bootstrap a new terminal/app iframe — queued indefinitely, hanging the UI even though the backend was healthy.mngr forwardgains a--use-http2flag (off by default). When set, the proxy generates a fresh in-memory self-signed cert at startup (SANslocalhost,*.localhost,127.0.0.1), terminates TLS, and negotiates HTTP/2 via ALPN — served by hypercorn (replacing uvicorn), with the pre-bound listen socket handed off by fd (fd://, viaos.dupso hypercorn's shutdown-close doesn't double-close). HTTP/2 multiplexes many streams over one connection, so the per-origin ceiling stops binding.https/wss(login URL,/gotobridge, unauthenticated-subdomain redirect) and itsmngr_forward_sessioncookie is markedSecure. The proxy→container hop and the WebSocket transport are unchanged; WS upgrades negotiatehttp/1.1per-connection via ALPN.--use-http2: the Python readiness/recovery probes and/gotoredirect URLs arehttps(loopback probes useverify=False, since they dial127.0.0.1with aHost: agent-<hex>.localhostheader against a self-signed cert), and the Electron shell unconditionally buildshttps/wssURLs, marks the cookieSecure, and trusts the loopback cert.https/wssworkspace URLs, marks the cookieSecure, and registers asetCertificateVerifyProcthat trusts the self-signed cert only for loopback hosts (localhost,*.localhost,127.0.0.1) in thepersist:workspace-contentpartition — every real https origin still gets Chromium's normal verification. The minds bare-origin backend stays plain HTTP (a deliberate, non-blocked mixed transport).cryptographyis declared explicitly anduvicorndropped frommngr_forward. CLI docs regenerated; per-project changelog entries added formngr_forward,minds,mngr, anddev. Two ratchets bumped with documented justification (asyncio import1 to 2 forasyncio.run(hypercorn.serve(...));__init__ in non-exception classes1 to 2 for theInMemoryTLSConfigsubclass of hypercorn's third-partyConfig).Implementation notes
serve(..., shutdown_trigger=None)installs SIGINT/SIGTERM handlers (matching uvicorn's graceful behavior minds'terminate()relies on),create_socketshandlesfd://by wrapping the fd without re-binding, andssl_enabled/create_ssl_contextare the hooks theConfigsubclass overrides.ssl.SSLContext.load_cert_chainrequires a filesystem path, so "never written to disk" isn't achievable — the PEM is written to a privatemkstemp(0600) file, loaded, and unlinked immediately. It is never persisted.Verification
origin/main.h2, a real HTTP/2 request succeeds withverify=False, and the fd handoff does not double-close on shutdown. A second check served the real forward app over TLS+h2:GET /returns 200 HTTP/2 (login page),/authenticatereturns 307 HTTP/2 with aSecurecookie, clean shutdown.wss) forwarding in a running workspace, and the reported symptom itself — opening more than 6 streaming tabs and then loading a new terminal/app tab without a hang. These require the full Electron app plus a live backend.