Skip to content

entity role: cross-model join matching via entity.role - #2127

Draft
QMalcolm wants to merge 3 commits into
mfsql-where-translatorfrom
entity-role-support
Draft

entity role: cross-model join matching via entity.role#2127
QMalcolm wants to merge 3 commits into
mfsql-where-translatorfrom
entity-role-support

Conversation

@QMalcolm

@QMalcolm QMalcolm commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Summary

Stacked on #2126 (mfsql-where-translator) — this PR targets that branch, not main.

Widens cross-model join matching from requiring the exact same entity name on both sides
to matching on join key (entity.role if set, else entity.name). This lets a model
declare multiple, differently-named entities (e.g. buyer, seller) that each join to
the same target entity elsewhere (e.g. user) under their own local name.

Entity.role already existed in the schema/protocol/impl but was dead code - never read
anywhere in join-building. This PR makes it do something, and removes a hack the project
had already hit and documented: scd_bookings.yaml carried a duplicate user entity
aliasing guest_id, commented "Hack around lack of support for entity/role functionality" with its own TODO: Remove this when full support ... is available.
guest now declares role: user directly instead.

What this touches, and why it's bigger than "widen a few dict keys"

Investigation before writing code found two separate, independently-implemented
join-matching systems
, both keyed on literal entity-name equality, both required for a
role-based join to actually work end-to-end rather than validate-then-crash:

  1. Query resolution (join_lookup.py, manifest_object_lookup.py,
    entity_lookup.py) - decides whether a query is valid. Now matches by join key;
    JoinModelOnRightDescriptor always carries the right model's own entity name (never
    the left's), since that's the identity entity_join_subgraph.py needs for graph
    connectivity.
  2. Dataflow plan building (node_evaluator.py,
    semantic_model_join_evaluator.py) - a wholly separate, legacy implementation that
    dataflow_plan_builder.py routes through instead of the semantic_graph module.
    Confirmed by direct testing: with only (1) fixed, a role-based query resolved
    successfully and then failed at plan-building with "Unable to join all items in
    request" - exactly the validate-then-crash risk flagged before starting.
    is_valid_semantic_model_join now takes separate left_entity_reference/
    right_entity_reference. A fourth call site inside evaluate_node (an early
    "is this even joinable" classification, easy to miss) had the same literal-name
    assumption and needed the same fix.
  3. SQL predicate construction (sql_join_builder.py) - the deepest layer.
    join_on_entity (a single shared reference) is used to find the join column on
    both sides via column_associations_for_entity; with genuinely different left/right
    names, the left-side lookup finds nothing. Added join_on_left_entity to
    JoinDescription/JoinLinkableInstancesRecipe - None means "same as
    join_on_entity" (every existing non-role join, unchanged). Verified the actual
    generated SQL end-to-end (subq.guest = users_latest.user_id, using guest_id
    correctly).

Also found and fixed a real correctness risk before shipping: if more than one local
entity shares a role (e.g. both guest and host declared role: user), the naive
implementation silently picked whichever was iterated first - a query that looks correct
but joins on an arbitrary entity, no error, no way to ask for the other one. Added an
explicit check that raises a clear error naming the competing entities instead. There's
no "default" disambiguation mechanism yet (unlike the POC design that motivated this
work) - deliberately deferred, which is why scd_bookings.yaml only adds role: user to
guest, not host.

Explicitly out of scope (separate, larger pieces)

  • Full query-surface role names - guest__x/host__x resolving independently as
    distinct dunder paths. Dunder-path resolution today still uses the target entity's
    own name (user__x), confirmed empirically, not the role-aliased local name.
  • Multi-hop role-based joins (dataflow_join_validator.py's JoinDataflowOutputValidator
    is unchanged, still single-shared-reference).
  • A default-style disambiguation mechanism for the multi-role-per-model ambiguity case.

Test plan

  • New tests in test_node_evaluator.py: a passing single-role join (asserting
    join_on_left_entity is set correctly) and the ambiguous-multi-role rejection.
  • Full tests_metricflow_semantics suite (287 passed) and tests_metricflow suite
    (1159 passed, 4 pre-existing/unrelated release_validation failures confirmed
    present on the base commit too via git stash comparison) - zero regressions.
  • Existing golden-SQL snapshot diffs for SCD tests reviewed and confirmed as the
    expected, purely mechanical removal of the redundant user/booking__user
    columns bookings_source no longer emits.
  • Cross-checked against the real-data DuckDB integration suite (itest_scd.yaml,
    which already covers the exact listing__user__home_state_latest path) that
    nothing about correctness changed, only SQL text/aliasing.
  • ruff/black/mypy clean.
  • No changelog entry yet - stacked on an unmerged PR, not yet ready for release.

🤖 Generated with Claude Code

@cla-bot cla-bot Bot added the cla:yes label Sep 2, 2026
@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide.

Widens cross-model join matching from requiring the exact same entity
name on both sides to matching on join key (entity.role if set, else
entity.name). This lets a model declare multiple, differently-named
entities (e.g. buyer, seller) that each join to the same target entity
elsewhere (e.g. user) under their own local name - closing a gap the
project had already hit and documented: scd_bookings.yaml carried a
"Hack around lack of support for entity/role functionality" duplicate
entity, with its own TODO to remove it once role support existed. That
hack is removed here as the proof case: `guest` now declares `role:
user` directly instead of bookings_source also declaring a redundant
`user` entity that aliased the same guest_id column.

The `Entity.role` field already existed in the schema/protocol/impl,
but was dead - never read anywhere in join-building. This makes it do
something.

Investigation before writing any code found the real scope was larger
than "widen a few dict keys": there are two separate, independently
implemented join-matching systems in this codebase, both keyed on
literal entity-name equality, and both needed updating for a role-based
join to actually work end-to-end rather than validate-then-crash:

1. Query resolution (GroupByItemResolver -> the semantic_graph module:
   manifest_object_lookup.py, join_lookup.py, entity_lookup.py) -
   decides whether a query like `--group-by guest__attribute` is valid.
   entity_lookup.py gained `join_key_to_entities`; join_lookup.py now
   matches left/right entities by join key and, critically, records
   the *right* model's own entity name in JoinModelOnRightDescriptor
   (never the left's) - entity_join_subgraph.py builds a
   ConfiguredEntityNode from that name, which must equal the same
   model's own self-reference node built when it's later processed as
   a left model, for the graph to stay connected. Confirmed empirically
   that dunder-path resolution is unaffected: querying still uses the
   *target* entity's own name (`user__x`), not the role-aliased local
   name (`guest__x`) - full query-surface role names (`guest__x` and
   `host__x` resolving independently) is a separate, larger piece of
   work, deliberately out of scope here.

2. Dataflow plan building (DataflowPlanBuilder -> node_evaluator.py ->
   semantic_model_join_evaluator.py) - decides how to actually build
   the join once a query is deemed valid. This is a wholly separate,
   legacy implementation dataflow_plan_builder.py never routes through
   the semantic_graph module at all. Confirmed by direct testing: with
   only (1) fixed, a role-based query resolved successfully and then
   failed at plan-building with "Unable to join all items in request" -
   validate-then-crash, exactly the risk flagged before starting.
   SemanticModelJoinEvaluator's is_valid_semantic_model_join now takes
   separate left_entity_reference/right_entity_reference (previously
   one shared on_entity_reference assumed both sides matched by name).
   node_evaluator.py's candidate-matching loop, and a separate early
   "is this even joinable" classification inside evaluate_node found
   by empirical debugging (a fourth, easy-to-miss call site with the
   same literal-name assumption), both now match by join key.
   dataflow_join_validator.py's multi-hop-only JoinDataflowOutputValidator
   is unchanged (still single-shared-reference) - multi-hop role-based
   joins remain unsupported, out of scope, same as the query-surface
   piece above.

3. SQL predicate construction (sql_join_builder.py) - the deepest
   layer. join_on_entity (a single shared EntityReference) is used to
   find the join column on *both* the left and right SQL datasets via
   column_associations_for_entity. With genuinely different left/right
   entity names, the left-side lookup for the right side's name finds
   nothing. Added `join_on_left_entity: Optional[EntityReference] =
   None` to JoinDescription/JoinLinkableInstancesRecipe - None means
   "same as join_on_entity" (every existing, non-role join, unchanged),
   set only when a role-based join used a different left-side entity.
   Verified the actual generated SQL is correct end-to-end for the
   single-role case (`subq.guest = users_latest.user_id`, using
   guest_id, not some wrong or ambiguous column).

Also found and fixed a real correctness risk before shipping: if more
than one local entity shares a role (e.g. both guest and host declared
role: user), the naive implementation silently picked whichever one
was iterated first, with zero indication of ambiguity - a query that
looks correct but silently joins on an arbitrary entity. Added an
explicit check in node_evaluator.py that raises UnableToSatisfyQueryError
naming the competing entities instead. There is no "default" mechanism
yet to disambiguate (unlike the POC design that motivated this work,
which has one) - deliberately deferred; this is why scd_bookings.yaml
only adds `role: user` to `guest`, not `host`.

Test fixture updates: two existing unit tests (test_semantic_model_join_evaluator.py,
test_join_validator.py) constructed their own foreign/natural entity
pairings using bookings_source + a literal "user" reference, which no
longer exists post-migration - fixed by using the entities that
actually exist on each side now (guest+user via role, or a completely
different already-natural pairing where a shared literal name still
works, for the one legacy validator that doesn't support role at all).
Golden SQL snapshot diffs for existing SCD tests are the expected,
purely mechanical removal of the redundant `user`/`booking__user`
columns bookings_source no longer emits - confirmed via the real-data
DuckDB integration suite (itest_scd.yaml, which already covers the
exact `listing__user__home_state_latest` path) that nothing about
correctness changed, only the SQL text.

Added tests: a passing single-role join (asserting join_on_left_entity
is set correctly) and the ambiguous-multi-role rejection, both in
test_node_evaluator.py alongside the existing SCD/multi-hop coverage
that file already had.
CI caught this on the previous commit (dd2646f): the new
join_key_to_entities property was added to EntityLookup's debug
_attribute_mapping, which pretty-prints full Entity objects - including
their PydanticMetadata.repo_file_path, an absolute filesystem path.
That path differs between my machine and the CI runner (and would
differ for every contributor's checkout), so the committed snapshot
(generated locally) could never match what CI generates.

The property itself (join_key_to_entities, returning real Entity
objects) is still needed and unchanged - join_lookup.py depends on it
for the actual feature. Only the debug-display side needed fixing:
added join_key_to_entity_names, names only (mirroring the existing
entity_type_to_names pattern), and pointed _attribute_mapping at that
instead.

Also regenerates the Postgres-dialect golden SQL snapshots for the
same SCD tests fixed for DuckDB in the previous commit - I'd only
verified DuckDB locally and missed that CI runs the same golden-SQL
tests against Postgres too, with its own separate snapshot files.
Confirmed the diff is identical in shape to the DuckDB one (just the
redundant `user`/`booking__user` columns disappearing, nothing
dialect-specific) by spinning up a local Postgres instance and
regenerating for real rather than hand-editing the SQL text.

Note: BigQuery, Databricks, Redshift, Snowflake, and Trino each have
their own golden SQL snapshots for these same SCD tests that will need
the same regeneration. Those dialects are skipped in this PR's CI
(gated behind a label), and I don't have credentials for any of the
real warehouses to regenerate them for real - flagging this explicitly
as a known gap rather than hand-editing SQL text I can't verify.
Required by the changelog-existence CI check. References this PR
directly (2127) since there's no separate tracked issue for this
exploratory work.
@QMalcolm
QMalcolm force-pushed the entity-role-support branch from 0e0c1e2 to ca60999 Compare September 3, 2026 14:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant