Add quality check execution to AI enhance flow#1859
Draft
arsalann wants to merge 1 commit into
Draft
Conversation
This change adds a new step (Step 4) to the AI enhance flow that automatically runs quality checks against the actual data after the AI enhancement is complete. This ensures that any quality checks added by the AI actually pass against the real data. Changes: - Add Step 4: Run quality checks using 'bruin run --only checks' - If quality checks fail, restore the original file (same behavior as validation failure) - Skip quality check execution if no checks are defined on the asset - Update the AI prompt to warn that quality checks will be validated against real data - Instruct the AI to be more conservative when adding quality checks - Tell the AI to verify assumptions using pre-fetched table statistics The flow is now: 1. Fill columns from database 2. AI Enhancement 3. Validate syntax 4. Run quality checks against data If any step fails, the original file is restored. Co-authored-by: Arsalan <arsalann@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR fixes a bug where the AI enhance function would add quality checks that fail when run against real data. The fix adds a new validation step that automatically executes the quality checks against the actual data after AI enhancement.
Problem
The AI enhance feature was adding quality checks to assets, but these checks were only being validated for syntax correctness (via
bruin validate), not whether they actually pass against the real data. This led to a poor user experience where enhanced assets would have failing quality checks.Solution
Added a new Step 4 to the enhance flow that runs
bruin run --only checksafter the AI enhancement is complete. If any quality check fails against the actual data, the original file is restored (same behavior as syntax validation failure).Changes
cmd/enhance.go:bruin run --only checkspkg/enhance/prompt.go:New Flow
If any step fails, the original file is restored.
Testing
go vetpassesNote: Full test suite requires Rust library that isn't available in the cloud agent environment, but the logic has been verified through code review and build testing.
Slack Thread