[Spike] grpc + Nuitka compatibility on Windows (throwaway) - #2081
Draft
QMalcolm wants to merge 7 commits into
Draft
[Spike] grpc + Nuitka compatibility on Windows (throwaway)#2081QMalcolm wants to merge 7 commits into
QMalcolm wants to merge 7 commits into
Conversation
Phase 2 of the Nuitka sidecar architecture (see pyo3-vs-nuitka-analysis.md). mf_entry.py is the long-lived subprocess that Fusion (dbt-core v2) spawns and communicates with over NDJSON stdin/stdout to compile metric queries to SQL without executing them. Design decisions: stdout protection: sys.stdout is redirected to sys.stderr at startup before writing the ready message. Any accidental print() call from MetricFlow internals or a library would corrupt the NDJSON framing Fusion reads; this makes the IPC channel the only writer to fd 1. Engine caching: manifest is cached by (path, mtime, sql_engine). Engine construction (manifest load + semantic validation) is the expensive operation; explain() itself is fast. Cache invalidation on mtime change means Fusion gets fresh SQL after a dbt parse run without a process restart. Engine-parametric stub: _SqlClientStub accepts any SqlEngine and selects the right SqlPlanRenderer, so the same binary produces correct dialect-specific SQL for all seven supported warehouses. Manifest loading: both YAML directories (dev/testing) and manifest.json files (production) are supported via a path.is_dir() branch. The JSON path uses mf_load_manifest_from_json_file from manifest_helpers, which is the same deserializer the Semantic Layer Gateway uses. Protocol v1 is strictly sequential (one request at a time). Fusion provides concurrency by pooling N sidecar processes. Request IDs are included for future multiplexing but not exercised here. JSON-RPC 2.0 compliance was considered and rejected: our use case only needs explain/ping/shutdown and the added error-code overhead and batch-request handling of full JSON-RPC would serve no practical purpose. Smoke-tested: ping, explain (bookings metric via sg_00_minimal_manifest), unknown method, shutdown, and invalid metric name all return correct NDJSON responses. The process exits cleanly on EOF.
Tests communicate with a real mf_entry.py process over stdin/stdout, mirroring exactly how Fusion uses the sidecar — no mocking of MetricFlowEngine or the IPC layer. This is the only test strategy that catches stdout contamination bugs, framing errors, or process lifecycle issues. A module-scoped fixture spawns one pre-warmed process and shares it across the seven protocol tests (ping, explain happy path, explain error, unknown method, protocol version mismatch, malformed JSON). Two additional tests spawn their own processes to verify clean exit on shutdown and EOF, since those tests consume the process. Unit tests of the dispatch logic were considered but rejected: the subprocess boundary is where the real failure modes live (framing, exit codes, stdout contamination), and the protocol functions are thin enough that subprocess tests cover them completely without duplication.
Updates the nuitka-build hatch environment to compile poc/mf_entry.py (the production IPC server) instead of poc/nuitka_poc.py (the throwaway PoC that demonstrated Nuitka feasibility). The validate script required a new approach: nuitka_poc.py printed SQL directly to stdout, so the old validate simply diffed files. mf_entry.py is a persistent server, so validation must send an IPC request via stdin and extract the SQL field from the JSON response. Embedding that logic as a shell pipeline in pyproject.toml would be unreadable due to quote escaping; a standalone validate_mf_entry.py keeps it clear and testable. The binary path on macOS is poc/mf_entry.dist/mf_entry.bin; validate checks for binary existence upfront and prints an actionable error rather than crashing with FileNotFoundError.
No developer landing in sidecar/ has a clear picture of what this binary is, how to build it, or how to speak to it. The README fills that gap: what the sidecar is and why it exists, the full mf-ipc v1 protocol reference (ready message, all three methods, error shape, and error type table), CLI flags, the stdout protection rationale, and the hatch commands for running tests, compiling, and validating.
The PoC run left nuitka.sql, reference.sql, nuitka_poc.dist/, and nuitka_poc.build/ untracked in sidecar/. These are one-time validation snapshots that don't belong in the repo; adding gitignore patterns for the build output directories cleans the working tree without losing the compile/validate hatch scripts that produced them. The two existing hatch scripts (compile, validate) were intentionally kept separate for CI use — CI will wire them as independent steps so failures are attributable. A build-and-validate convenience script is added for local use so developers don't need to remember the two-command sequence. Refs DI-4631
Investigating Paul's suggestion on PR #2072 to use gRPC for the mf-ipc protocol instead of NDJSON. Already validated locally on macOS arm64: grpc alone survives Nuitka standalone freezing (including its roots.pem data file, handled automatically by Nuitka's data-files plugin), grpc compiles and runs correctly alongside real MetricFlow deps (pydantic-core, duckdb) in the same binary with no C-extension conflicts, and real protobuf message serialization works through a live RPC. Windows is the one untested platform, and it's already bitten this project once on something unrelated (the Dependency Walker prompt) — grpc's C-core does platform-conditional DNS/socket handling that could behave differently there. Throwaway: not part of any real feature, just a risk-investigation spike. Delete this workflow and spike-grpc-nuitka/ once the signal lands.
|
Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide. |
Nuitka names the compiled binary grpc_probe.exe on Windows, not grpc_probe.bin (which validate_mf_entry.py already handles correctly for mf_entry). The compile step itself succeeded on Windows — this was just a path mistake in the run step, not a real finding.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Throwaway spike, not a real feature PR. Investigating the suggestion on #2072
to use gRPC for the mf-ipc protocol instead of NDJSON.
Already validated locally on macOS arm64:
roots.pemdata file, handled automatically by Nuitka's
data-filesplugin)(pydantic-core, duckdb) in the same binary, no C-extension conflicts
This PR exists only to exercise the same probe on a real Windows runner via
the
pull_requesttrigger (workflow_dispatchcan't be invoked until theworkflow exists on the default branch). Will close without merging once the
signal lands — see
spike-grpc-nuitka/grpc_probe.py.