entity role: cross-model join matching via entity.role - #2127
Draft
QMalcolm wants to merge 3 commits into
Draft
Conversation
|
Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide. |
Widens cross-model join matching from requiring the exact same entity name on both sides to matching on join key (entity.role if set, else entity.name). This lets a model declare multiple, differently-named entities (e.g. buyer, seller) that each join to the same target entity elsewhere (e.g. user) under their own local name - closing a gap the project had already hit and documented: scd_bookings.yaml carried a "Hack around lack of support for entity/role functionality" duplicate entity, with its own TODO to remove it once role support existed. That hack is removed here as the proof case: `guest` now declares `role: user` directly instead of bookings_source also declaring a redundant `user` entity that aliased the same guest_id column. The `Entity.role` field already existed in the schema/protocol/impl, but was dead - never read anywhere in join-building. This makes it do something. Investigation before writing any code found the real scope was larger than "widen a few dict keys": there are two separate, independently implemented join-matching systems in this codebase, both keyed on literal entity-name equality, and both needed updating for a role-based join to actually work end-to-end rather than validate-then-crash: 1. Query resolution (GroupByItemResolver -> the semantic_graph module: manifest_object_lookup.py, join_lookup.py, entity_lookup.py) - decides whether a query like `--group-by guest__attribute` is valid. entity_lookup.py gained `join_key_to_entities`; join_lookup.py now matches left/right entities by join key and, critically, records the *right* model's own entity name in JoinModelOnRightDescriptor (never the left's) - entity_join_subgraph.py builds a ConfiguredEntityNode from that name, which must equal the same model's own self-reference node built when it's later processed as a left model, for the graph to stay connected. Confirmed empirically that dunder-path resolution is unaffected: querying still uses the *target* entity's own name (`user__x`), not the role-aliased local name (`guest__x`) - full query-surface role names (`guest__x` and `host__x` resolving independently) is a separate, larger piece of work, deliberately out of scope here. 2. Dataflow plan building (DataflowPlanBuilder -> node_evaluator.py -> semantic_model_join_evaluator.py) - decides how to actually build the join once a query is deemed valid. This is a wholly separate, legacy implementation dataflow_plan_builder.py never routes through the semantic_graph module at all. Confirmed by direct testing: with only (1) fixed, a role-based query resolved successfully and then failed at plan-building with "Unable to join all items in request" - validate-then-crash, exactly the risk flagged before starting. SemanticModelJoinEvaluator's is_valid_semantic_model_join now takes separate left_entity_reference/right_entity_reference (previously one shared on_entity_reference assumed both sides matched by name). node_evaluator.py's candidate-matching loop, and a separate early "is this even joinable" classification inside evaluate_node found by empirical debugging (a fourth, easy-to-miss call site with the same literal-name assumption), both now match by join key. dataflow_join_validator.py's multi-hop-only JoinDataflowOutputValidator is unchanged (still single-shared-reference) - multi-hop role-based joins remain unsupported, out of scope, same as the query-surface piece above. 3. SQL predicate construction (sql_join_builder.py) - the deepest layer. join_on_entity (a single shared EntityReference) is used to find the join column on *both* the left and right SQL datasets via column_associations_for_entity. With genuinely different left/right entity names, the left-side lookup for the right side's name finds nothing. Added `join_on_left_entity: Optional[EntityReference] = None` to JoinDescription/JoinLinkableInstancesRecipe - None means "same as join_on_entity" (every existing, non-role join, unchanged), set only when a role-based join used a different left-side entity. Verified the actual generated SQL is correct end-to-end for the single-role case (`subq.guest = users_latest.user_id`, using guest_id, not some wrong or ambiguous column). Also found and fixed a real correctness risk before shipping: if more than one local entity shares a role (e.g. both guest and host declared role: user), the naive implementation silently picked whichever one was iterated first, with zero indication of ambiguity - a query that looks correct but silently joins on an arbitrary entity. Added an explicit check in node_evaluator.py that raises UnableToSatisfyQueryError naming the competing entities instead. There is no "default" mechanism yet to disambiguate (unlike the POC design that motivated this work, which has one) - deliberately deferred; this is why scd_bookings.yaml only adds `role: user` to `guest`, not `host`. Test fixture updates: two existing unit tests (test_semantic_model_join_evaluator.py, test_join_validator.py) constructed their own foreign/natural entity pairings using bookings_source + a literal "user" reference, which no longer exists post-migration - fixed by using the entities that actually exist on each side now (guest+user via role, or a completely different already-natural pairing where a shared literal name still works, for the one legacy validator that doesn't support role at all). Golden SQL snapshot diffs for existing SCD tests are the expected, purely mechanical removal of the redundant `user`/`booking__user` columns bookings_source no longer emits - confirmed via the real-data DuckDB integration suite (itest_scd.yaml, which already covers the exact `listing__user__home_state_latest` path) that nothing about correctness changed, only the SQL text. Added tests: a passing single-role join (asserting join_on_left_entity is set correctly) and the ambiguous-multi-role rejection, both in test_node_evaluator.py alongside the existing SCD/multi-hop coverage that file already had.
CI caught this on the previous commit (dd2646f): the new join_key_to_entities property was added to EntityLookup's debug _attribute_mapping, which pretty-prints full Entity objects - including their PydanticMetadata.repo_file_path, an absolute filesystem path. That path differs between my machine and the CI runner (and would differ for every contributor's checkout), so the committed snapshot (generated locally) could never match what CI generates. The property itself (join_key_to_entities, returning real Entity objects) is still needed and unchanged - join_lookup.py depends on it for the actual feature. Only the debug-display side needed fixing: added join_key_to_entity_names, names only (mirroring the existing entity_type_to_names pattern), and pointed _attribute_mapping at that instead. Also regenerates the Postgres-dialect golden SQL snapshots for the same SCD tests fixed for DuckDB in the previous commit - I'd only verified DuckDB locally and missed that CI runs the same golden-SQL tests against Postgres too, with its own separate snapshot files. Confirmed the diff is identical in shape to the DuckDB one (just the redundant `user`/`booking__user` columns disappearing, nothing dialect-specific) by spinning up a local Postgres instance and regenerating for real rather than hand-editing the SQL text. Note: BigQuery, Databricks, Redshift, Snowflake, and Trino each have their own golden SQL snapshots for these same SCD tests that will need the same regeneration. Those dialects are skipped in this PR's CI (gated behind a label), and I don't have credentials for any of the real warehouses to regenerate them for real - flagging this explicitly as a known gap rather than hand-editing SQL text I can't verify.
Required by the changelog-existence CI check. References this PR directly (2127) since there's no separate tracked issue for this exploratory work.
QMalcolm
force-pushed
the
entity-role-support
branch
from
September 3, 2026 14:24
0e0c1e2 to
ca60999
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #2126 (
mfsql-where-translator) — this PR targets that branch, notmain.Widens cross-model join matching from requiring the exact same entity name on both sides
to matching on join key (
entity.roleif set, elseentity.name). This lets a modeldeclare multiple, differently-named entities (e.g.
buyer,seller) that each join tothe same target entity elsewhere (e.g.
user) under their own local name.Entity.rolealready existed in the schema/protocol/impl but was dead code - never readanywhere in join-building. This PR makes it do something, and removes a hack the project
had already hit and documented:
scd_bookings.yamlcarried a duplicateuserentityaliasing
guest_id, commented"Hack around lack of support for entity/role functionality"with its ownTODO: Remove this when full support ... is available.guestnow declaresrole: userdirectly instead.What this touches, and why it's bigger than "widen a few dict keys"
Investigation before writing code found two separate, independently-implemented
join-matching systems, both keyed on literal entity-name equality, both required for a
role-based join to actually work end-to-end rather than validate-then-crash:
join_lookup.py,manifest_object_lookup.py,entity_lookup.py) - decides whether a query is valid. Now matches by join key;JoinModelOnRightDescriptoralways carries the right model's own entity name (neverthe left's), since that's the identity
entity_join_subgraph.pyneeds for graphconnectivity.
node_evaluator.py,semantic_model_join_evaluator.py) - a wholly separate, legacy implementation thatdataflow_plan_builder.pyroutes through instead of the semantic_graph module.Confirmed by direct testing: with only (1) fixed, a role-based query resolved
successfully and then failed at plan-building with "Unable to join all items in
request" - exactly the validate-then-crash risk flagged before starting.
is_valid_semantic_model_joinnow takes separateleft_entity_reference/right_entity_reference. A fourth call site insideevaluate_node(an early"is this even joinable" classification, easy to miss) had the same literal-name
assumption and needed the same fix.
sql_join_builder.py) - the deepest layer.join_on_entity(a single shared reference) is used to find the join column onboth sides via
column_associations_for_entity; with genuinely different left/rightnames, the left-side lookup finds nothing. Added
join_on_left_entitytoJoinDescription/JoinLinkableInstancesRecipe-Nonemeans "same asjoin_on_entity" (every existing non-role join, unchanged). Verified the actualgenerated SQL end-to-end (
subq.guest = users_latest.user_id, usingguest_idcorrectly).
Also found and fixed a real correctness risk before shipping: if more than one local
entity shares a role (e.g. both
guestandhostdeclaredrole: user), the naiveimplementation silently picked whichever was iterated first - a query that looks correct
but joins on an arbitrary entity, no error, no way to ask for the other one. Added an
explicit check that raises a clear error naming the competing entities instead. There's
no "default" disambiguation mechanism yet (unlike the POC design that motivated this
work) - deliberately deferred, which is why
scd_bookings.yamlonly addsrole: usertoguest, nothost.Explicitly out of scope (separate, larger pieces)
guest__x/host__xresolving independently asdistinct dunder paths. Dunder-path resolution today still uses the target entity's
own name (
user__x), confirmed empirically, not the role-aliased local name.dataflow_join_validator.py'sJoinDataflowOutputValidatoris unchanged, still single-shared-reference).
default-style disambiguation mechanism for the multi-role-per-model ambiguity case.Test plan
test_node_evaluator.py: a passing single-role join (assertingjoin_on_left_entityis set correctly) and the ambiguous-multi-role rejection.tests_metricflow_semanticssuite (287 passed) andtests_metricflowsuite(1159 passed, 4 pre-existing/unrelated
release_validationfailures confirmedpresent on the base commit too via
git stashcomparison) - zero regressions.expected, purely mechanical removal of the redundant
user/booking__usercolumns
bookings_sourceno longer emits.itest_scd.yaml,which already covers the exact
listing__user__home_state_latestpath) thatnothing about correctness changed, only SQL text/aliasing.
🤖 Generated with Claude Code