Feat(Workflows): Auto-inject inclusive gateways for batch routing#26918
Open
yan-3005 wants to merge 3 commits intoram/workflow-improvementsfrom
Open
Feat(Workflows): Auto-inject inclusive gateways for batch routing#26918yan-3005 wants to merge 3 commits intoram/workflow-improvementsfrom
yan-3005 wants to merge 3 commits intoram/workflow-improvementsfrom
Conversation
Check nodes (CheckEntityAttributes, CheckChangeDescription, DataCompleteness) now emit boolean flag variables (hasTrueEntities, hasFalseEntities, has_<band>_entities) instead of a single RESULT_VARIABLE. MainWorkflow auto-detects split/join points at BPMN build time via NodeInterface.getOutputPorts() and injects InclusiveGateways so both true and false branches fire simultaneously when a batch has mixed pass/fail entities. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
yan-3005
added
safe to test
yan-3005
changed the title
![gitar-bot[bot]](https://avatars.githubusercontent.com/in/827041?s=60&v=4){kind=small}
...ervice/governance/workflows/elements/nodes/automatedTask/impl/CheckEntityAttributesImpl.java
Outdated
Show resolved
Hide resolved
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates OpenMetadata governance workflows to support batch entity processing via a new entityList variable, adds inclusive gateway injection for batch “check” nodes (true/false and quality-band branching), and introduces a v1140 migration to update persisted workflow JSON accordingly.
Changes:
- Replace/augment legacy
relatedEntityusage withentityListacross workflow triggers, node schemas, and runtime variable handling. - Inject inclusive split/join gateways in Flowable BPMN generation for nodes that produce multiple output ports (e.g., check tasks, data completeness bands).
- Add v1140 migration + extensive unit/integration test updates for new variable semantics and batch entity fetching.
Reviewed changes
Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| openmetadata-ui/src/main/resources/ui/src/generated/governance/workflows/elements/nodes/automatedTask/sinkTask.ts | Generated UI type updates: InputNamespaceMap now uses entityList and allows additional properties. |
| openmetadata-ui/src/main/resources/ui/src/generated/governance/workflows/elements/nodes/automatedTask/setEntityAttributeTask.ts | Generated UI type updates to entityList-based input namespace map. |
| openmetadata-ui/src/main/resources/ui/src/generated/governance/workflows/elements/nodes/automatedTask/rollbackEntityTask.ts | Generated UI type updates + updated rollback description. |
| openmetadata-ui/src/main/resources/ui/src/generated/governance/workflows/elements/nodes/automatedTask/dataCompletenessTask.ts | Generated UI type updates to entityList-based input namespace map. |
| openmetadata-ui/src/main/resources/ui/src/generated/governance/workflows/elements/nodes/automatedTask/checkEntityAttributesTask.ts | Generated UI type updates + adds output?: string[] on task interface and entityList in namespace map. |
| openmetadata-ui/src/main/resources/ui/src/generated/governance/workflows/elements/nodes/automatedTask/checkChangeDescriptionTask.ts | Generated UI type updates + adds output?: string[] on task interface and entityList in namespace map. |
| openmetadata-ui/src/main/resources/ui/src/generated/api/governance/createWorkflowDefinition.ts | API types updated to include output?: string[] and allow entityList in trigger namespace mapping. |
| openmetadata-spec/src/main/resources/json/schema/governance/workflows/elements/triggers/periodicBatchEntityTrigger.json | Trigger schema output default now includes entityList; relaxes single-item constraint. |
| openmetadata-spec/src/main/resources/json/schema/governance/workflows/elements/triggers/eventBasedEntityTrigger.json | Trigger schema output default now includes entityList; relaxes single-item constraint. |
| openmetadata-spec/src/main/resources/json/schema/governance/workflows/elements/nodes/automatedTask/sinkTask.json | Node schema migrated to require entityList and allow additional properties for conditional keys. |
| openmetadata-spec/src/main/resources/json/schema/governance/workflows/elements/nodes/automatedTask/setEntityAttributeTask.json | Node schema migrated to require entityList and allow additional properties; input defaults/minItems adjusted. |
| openmetadata-spec/src/main/resources/json/schema/governance/workflows/elements/nodes/automatedTask/rollbackEntityTask.json | Node schema migrated to require entityList; description updated to fallback chain. |
| openmetadata-spec/src/main/resources/json/schema/governance/workflows/elements/nodes/automatedTask/dataCompletenessTask.json | Node schema migrated to require entityList; loosens input constraints. |
| openmetadata-spec/src/main/resources/json/schema/governance/workflows/elements/nodes/automatedTask/checkEntityAttributesTask.json | Node schema switches to entityList + adds explicit output defaults for derived entity lists. |
| openmetadata-spec/src/main/resources/json/schema/governance/workflows/elements/nodes/automatedTask/checkChangeDescriptionTask.json | Node schema switches to entityList + adds explicit output defaults for derived entity lists. |
| openmetadata-service/src/test/java/org/openmetadata/service/migration/utils/v1140/MigrationUtilTest.java | New unit tests for v1140 workflow JSON migration behavior. |
| openmetadata-service/src/test/java/org/openmetadata/service/migration/utils/v1105/MigrationUtilTest.java | Updates test fixtures to entityList in workflow JSON. |
| openmetadata-service/src/test/java/org/openmetadata/service/governance/workflows/flowable/MainWorkflowInclusiveGatewayTest.java | New tests verifying inclusive split/join gateway injection logic. |
| openmetadata-service/src/test/java/org/openmetadata/service/governance/workflows/flowable/builders/InclusiveGatewayBuilderTest.java | New unit tests for InclusiveGatewayBuilder. |
| openmetadata-service/src/test/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/sink/SinkTaskTest.java | Updates sink workflow JSON fixtures to entityList. |
| openmetadata-service/src/test/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/sink/SinkTaskDelegateTest.java | Updates delegate tests to batch entity fetching and entityList semantics. |
| openmetadata-service/src/test/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/SetEntityAttributeTaskTest.java | New unit tests for BPMN construction of SetEntityAttribute task with entityList. |
| openmetadata-service/src/test/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/RollbackEntityTaskTest.java | New unit tests for BPMN construction of RollbackEntity task with entityList. |
| openmetadata-service/src/test/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/impl/SetEntityAttributeImplTest.java | New unit tests for batch SetEntityAttributeImpl execution. |
| openmetadata-service/src/test/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/impl/CheckEntityAttributesImplTest.java | New unit tests for batch CheckEntityAttributesImpl execution and flags. |
| openmetadata-service/src/test/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/impl/CheckChangeDescriptionTaskImplTest.java | New unit tests for batch CheckChangeDescriptionTaskImpl execution and flags. |
| openmetadata-service/src/test/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/DataCompletenessTaskTest.java | New unit tests for BPMN construction of DataCompleteness task with entityList. |
| openmetadata-service/src/test/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/CheckEntityAttributesTaskTest.java | New unit tests for BPMN construction of CheckEntityAttributes task with entityList. |
| openmetadata-service/src/test/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/CheckChangeDescriptionTaskTest.java | New unit tests for BPMN construction of CheckChangeDescription task with entityList. |
| openmetadata-service/src/main/resources/json/data/governance/workflows/GlossaryApprovalWorkflow.json | Updates default workflow to emit/use entityList and conditional *_entityList wiring. |
| openmetadata-service/src/main/java/org/openmetadata/service/migration/utils/v1140/MigrationUtil.java | New v1140 migration utility to rewrite stored workflow JSON to entityList + conditional wiring. |
| openmetadata-service/src/main/java/org/openmetadata/service/migration/utils/v1105/MigrationUtil.java | Adds migration logic to rewrite glossary status node namespace map to entityList. |
| openmetadata-service/src/main/java/org/openmetadata/service/migration/postgres/v1140/Migration.java | New Postgres migration entrypoint invoking v1140 workflow migration. |
| openmetadata-service/src/main/java/org/openmetadata/service/migration/mysql/v1140/Migration.java | New MySQL migration entrypoint invoking v1140 workflow migration. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/WorkflowVariableHandler.java | Adds getEntityList() helper resolving entity lists from namespace maps (plain and conditional). |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/WorkflowEventConsumer.java | Ensures event-based trigger variables include both relatedEntity and entityList. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/Workflow.java | Adds shared workflow constants for true/false lists and retry configuration. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/flowable/MainWorkflow.java | Injects inclusive split/join gateways based on node output ports + conditional edges. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/flowable/builders/InclusiveGatewayBuilder.java | New builder for Flowable InclusiveGateway. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/triggers/PeriodicBatchEntityTrigger.java | Adjusts periodic trigger call-activity wiring to pass entityList for both single and parallel modes. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/triggers/impl/FilterEntityImpl.java | Sets global entityList for event-based flows after filtering. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/triggers/impl/FetchEntitiesImpl.java | Sets entityList, numberOfEntities, and entityToListMap for periodic triggers. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/triggers/EventBasedEntityTrigger.java | Always passes entityList into the main workflow; avoids duplicating trigger outputs. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/TriggerFactory.java | Refines batch-mode detection documentation and logic for trigger execution mode. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/SinkTask.java | Ensures sink task namespace map handling uses defined map and always includes entityList. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/sink/SinkTaskDelegate.java | Switches sink execution to entityList and introduces batched entity fetch + retries. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/SetEntityAttributeTask.java | Ensures BPMN wiring includes default entityList namespace. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/RollbackEntityTask.java | Ensures BPMN wiring includes default entityList namespace. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/impl/SetEntityAttributeImpl.java | Converts to batch processing over entityList with retry + partial failure recording. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/impl/RollbackEntityImpl.java | Converts to batch rollback over entityList with retry + fallback target version selection. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/impl/DataCompletenessImpl.java | Converts to batch completeness evaluation, per-band entity lists, and gateway-ready flags. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/impl/CheckEntityAttributesImpl.java | Converts to batch rule evaluation producing true/false entity lists and flags. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/impl/CheckChangeDescriptionTaskImpl.java | Converts to batch change-description evaluation producing true/false entity lists and flags. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/DataCompletenessTask.java | Declares output ports (bands) and defaults entityList mapping for gateway injection. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/CheckEntityAttributesTask.java | Declares output ports (true/false) and defaults entityList mapping for gateway injection. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/CheckChangeDescriptionTask.java | Declares output ports (true/false) and defaults entityList mapping for gateway injection. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/NodeInterface.java | Adds getOutputPorts() hook used for inclusive gateway detection. |
| openmetadata-service/src/main/java/org/openmetadata/service/Entity.java | Adds getEntitiesByLinks() helper for efficient batch entity fetch by entity link strings. |
| openmetadata-integration-tests/src/test/java/org/openmetadata/it/tests/WorkflowDefinitionResourceIT.java | Updates integration tests to build/validate workflows using entityList and new outputs. |
Comment on lines
+33
to
+47
| // Workflow: start → check → cert(true) / notify(false) → independent ends | ||
| WorkflowDefinition workflow = | ||
| createWorkflow( | ||
| "SplitOnlyWorkflow", | ||
| List.of( | ||
| startEvent("start"), | ||
| checkAttributesNode("check"), | ||
| endEvent("certEnd"), | ||
| endEvent("notifyEnd")), | ||
| List.of( | ||
| edge("start", "check", null), | ||
| edge("check", "cert", "true"), | ||
| edge("check", "notify", "false"), | ||
| edge("cert", "certEnd", null), | ||
| edge("notify", "notifyEnd", null))); |
There was a problem hiding this comment.
This test workflow defines edges to node ids (e.g. cert, notify) that are not included in the nodes list. That can produce a BPMN model with SequenceFlow targets/sources that don't exist, making the test less representative and potentially hiding issues if workflow validation becomes stricter. Consider adding minimal node definitions for cert/notify (or update edges to point to existing nodes) so the constructed workflow is structurally valid.
…ferences - Check nodes (CheckEntityAttributes, CheckChangeDescription) now set hasFalseEntities=true when entity list is empty, ensuring the split inclusive gateway always has at least one active branch and never stalls. - DataCompletenessImpl activates the lowest-score band flag when no entities are assigned to any band (empty input or all entities failed processing). - MainWorkflowInclusiveGatewayTest: replaced dangling edge targets (cert, notify, goldAction etc.) with actual declared end-event nodes so the constructed BPMN model is structurally valid. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Contributor
Jest test CoverageUI tests summary
|
|
Contributor
🔴 Playwright Results — 7 failure(s), 27 flaky✅ 3432 passed · ❌ 7 failed · 🟡 27 flaky · ⏭️ 223 skipped
Genuine Failures (failed on all attempts)❌
|
- DataCompletenessTask: guard against null getQualityBands() with Optional - DataCompletenessImpl: extract bandFlagVariable() static helper used by both the impl and MainWorkflow to avoid naming convention drift - InclusiveGatewayBuilder: remove misleading exclusive() setter; async job exclusivity is hardcoded true (not a routing concern, matches ExclusiveGatewayBuilder) - MainWorkflow: add @slf4j + warn log when a conditional edge targets a join node (condition is intentionally ignored); use DataCompletenessImpl.bandFlagVariable() - DataCompletenessImplTest: add tests for empty entity list and all-entities-fail scenarios, both verifying lowest-band flag is activated as deadlock fallback Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
| public DataCompletenessTask( | ||
| DataCompletenessTaskDefinition nodeDefinition, WorkflowConfiguration workflowConfig) { | ||
| this.outputPorts = | ||
| Optional.ofNullable(nodeDefinition.getConfig().getQualityBands()).orElse(List.of()).stream() |
There was a problem hiding this comment.
💡 Edge Case: Null guard on getQualityBands() but not on getConfig()
Line 44 wraps getQualityBands() with Optional.ofNullable() to handle a null band list, but getConfig() in the same call chain is not guarded. If getConfig() ever returns null, this will throw an NPE before the Optional even kicks in. While the JSON schema marks config as required (reducing the risk), other task nodes like CheckChangeDescriptionTask defensively null-check getConfig() as well. For consistency and resilience, the entire chain should be guarded.
Suggested fix:
Optional.ofNullable(nodeDefinition.getConfig())
.map(DataCompletenessTaskDefinition.DataCompletenessTaskConfig::getQualityBands)
.orElse(List.of()).stream()
.map(QualityBand::getName)
.collect(Collectors.toSet());
Was this helpful? React with 👍 / 👎 | Reply gitar fix to apply this suggestion
Code Review 👍 Approved with suggestions 2 resolved / 4 findingsAuto-inject inclusive gateways feature now handles empty entity lists without deadlocking and prevents silent failures in RollbackEntityImpl. Consider removing unnecessary retry logic from in-memory computation in DataCompletenessImpl and adding null guard on getConfig() for consistency with getQualityBands(). 💡 Performance: Retry wraps pure in-memory computation in DataCompletenessImpl📄 openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/impl/DataCompletenessImpl.java:63 📄 openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/impl/DataCompletenessImpl.java:74-80 In DataCompletenessImpl (lines 74-80), Other task impls (CheckEntityAttributes, SetEntityAttribute) correctly apply retry only around I/O operations like RuleEngine or EntityFieldUtils. 💡 Edge Case: Null guard on getQualityBands() but not on getConfig()Line 44 wraps Suggested fix✅ 2 resolved✅ Edge Case: Inclusive gateway deadlocks when entity list is empty
✅ Edge Case: RollbackEntityImpl silently continues on total failure
🤖 Prompt for agentsOptionsDisplay: compact → Showing less information. Comment with these commands to change:
Was this helpful? React with 👍 / 👎 | Gitar |
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe your changes:
Phase 2 of the batch workflow improvements. Builds on Phase 1 (PR #26715) which made all task nodes batch-aware with entity lists.
Problem: Check nodes previously emitted a single
RESULT_VARIABLEboolean, which drove exclusive gateway routing — only one branch (true OR false) could fire. In batch mode, a single check node can produce both passing and failing entities simultaneously, so both branches need to fire.Solution: Auto-inject Flowable inclusive gateways at BPMN build time based on graph structure — no changes needed to workflow JSON definitions.
Changes:
InclusiveGatewayBuilder(new) — mirrorsExclusiveGatewayBuilderforInclusiveGatewayelementsNodeInterface.getOutputPorts()(new default method) — nodes declare which output ports they produce. Only nodes overriding this with ≥2 ports get inclusive gateways injected.UserApprovalTaskand other single-result nodes return the default empty set and are unaffected.CheckEntityAttributesTask,CheckChangeDescriptionTask— overridegetOutputPorts()returning{"true", "false"}DataCompletenessTask— overridesgetOutputPorts()returning the configured quality band names (e.g.{"gold", "silver"})RESULT_VARIABLE; now emithasTrueEntities+hasFalseEntitiesboolean flags (check nodes), orhas_<band>_entitiesper band (DataCompleteness)MainWorkflow— detects split/join points at build time and injects inclusive gateways automatically:getOutputPorts().size() ≥ 2and ≥2 conditional outgoing edges; conditions map to${nodeName_hasTrueEntities},${nodeName_hasFalseEntities},${nodeName_has_<band>_entities}Before (exclusive — only one branch fires):
After (inclusive — both branches fire when batch has mixed results):
Type of change:
Checklist: