POC: Nuitka compilation of MetricFlow for Fusion (dbt-core v2) integration - #2069
Draft
QMalcolm wants to merge 5 commits into
Draft
POC: Nuitka compilation of MetricFlow for Fusion (dbt-core v2) integration#2069QMalcolm wants to merge 5 commits into
QMalcolm wants to merge 5 commits into
Conversation
Fusion (dbt-core v2) is Rust-based. The existing crates/dbt-metricflow crate is explicitly non-viable long-term — two independent query compiler implementations will inevitably diverge. Python MetricFlow must remain authoritative. Two embedding strategies were evaluated: PyO3 embedding (link libpython into Fusion, call MetricFlow in-process) and Nuitka sidecar (compile MetricFlow to a standalone native binary, invoke via subprocess with JSON IPC). Full analysis in pyo3-vs-nuitka-analysis.md. Nuitka sidecar is recommended. Decisive factors against PyO3: - CPython must init on Fusion's main thread before Tokio starts — UB if called from a worker thread, requiring Fusion startup changes. - pydantic-core ABI conflict risk: two PyO3 runtimes in one process. - GIL serializes all concurrent MetricFlow compilations. - A MetricFlow crash kills the entire Fusion process. Nuitka sidecar avoids all of these: crash isolation via subprocess EOF detection, true concurrency via a process pool, hermetic CPython bundle, and it extends Fusion's existing SidecarClient infrastructure rather than introducing a new pattern. poc/nuitka_poc.py is the empirical validation step. It loads sg_00_minimal_manifest, calls explain() via a _StubSqlClient (DuckDB renderer only, no real DB connection), and prints compiled SQL to stdout. Validation: diff Python output against the Nuitka binary output — a non-empty diff indicates silent miscompilation in pydantic v2 validators or fast_frozen_dataclass dynamic __hash__ replacement. The PoC is intentionally minimal: no IPC, no execution, no Fusion integration. Nuitka compilation and diff validation are the next step.
Adds a dedicated hatch env for the Nuitka compilation PoC rather than installing Nuitka into the main dev environment. Nuitka is a build tool (invokes a C compiler to produce a standalone binary) — it has no role in day-to-day MetricFlow development or testing, so it doesn't belong in dev-env alongside ruff/black/mypy. The nuitka-build env templates off dev-env (inheriting the full MetricFlow dependency set) and adds nuitka via extra-dependencies, so the compiled binary sees the same package versions as the Python reference run. Scripts: hatch run nuitka-build:compile — build the standalone binary hatch run nuitka-build:validate — run reference + binary, diff output
The sync_dsi.py migration renamed all dbt_semantic_interfaces imports to metricflow_semantic_interfaces, but missed this docstring. Stale package name left after the MSI extraction.
Nothing in the metricflow codebase imports dbt_semantic_interfaces directly — the MSI extraction replaced all such imports with metricflow_semantic_interfaces. Passing --include-package for a package that isn't installed (and isn't needed) caused Nuitka to fail hard at startup. Removed; --follow-imports handles all reachable packages automatically under --standalone.
Nuitka appends .bin to the output binary on macOS (nuitka_poc.bin), not the extensionless name used on Linux. The validate script was referencing the Linux path and failing with "No such file or directory".
|
Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide. |
1 similar comment
|
Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
dbt-core v2 (Fusion) is written in Rust. We need a way to call Python MetricFlow from Fusion without maintaining a separate Rust implementation of the query compiler — two implementations will inevitably diverge.
This PR validates the Nuitka sidecar as the integration strategy: compile Python MetricFlow into a standalone native binary (bundling CPython + all dependencies), then have Fusion spawn it as a subprocess and communicate over JSON IPC. See `pyo3-vs-nuitka-analysis.md` for the full PyO3 vs. Nuitka comparison and rationale.
The PoC answers one binary question: does a Nuitka-compiled MetricFlow binary produce identical SQL to the Python reference? It passed on macOS arm64.
What's in this PR
How to run the PoC yourself
Prerequisites: the `nuitka-build` hatch env will be created automatically on first use. First run takes ~5 minutes as Nuitka compiles ~1500 C files (subsequent runs use ccache).
What this is NOT
Next steps (not in this PR)