Skip to content

Add quality check execution to AI enhance flow#1859

Draft
arsalann wants to merge 1 commit into
mainfrom
cursor/ai-enhance-quality-flow-c4c4
Draft

Add quality check execution to AI enhance flow#1859
arsalann wants to merge 1 commit into
mainfrom
cursor/ai-enhance-quality-flow-c4c4

Conversation

@arsalann

Copy link
Copy Markdown
Contributor

Summary

This PR fixes a bug where the AI enhance function would add quality checks that fail when run against real data. The fix adds a new validation step that automatically executes the quality checks against the actual data after AI enhancement.

Problem

The AI enhance feature was adding quality checks to assets, but these checks were only being validated for syntax correctness (via bruin validate), not whether they actually pass against the real data. This led to a poor user experience where enhanced assets would have failing quality checks.

Solution

Added a new Step 4 to the enhance flow that runs bruin run --only checks after the AI enhancement is complete. If any quality check fails against the actual data, the original file is restored (same behavior as syntax validation failure).

Changes

cmd/enhance.go:

  • Added Step 4: Run quality checks using bruin run --only checks
  • If quality checks fail, restore the original file
  • Skip quality check execution if no checks are defined on the asset
  • Updated step numbering to reflect 4 steps total

pkg/enhance/prompt.go:

  • Updated the AI prompt to warn that quality checks will be validated against real data
  • Instructed the AI to be more conservative when adding quality checks
  • Added guidance to verify assumptions using pre-fetched table statistics before adding checks

New Flow

  1. Fill columns from database - Get column schema from DB
  2. AI Enhancement - AI adds metadata, descriptions, and quality checks
  3. Validate syntax - Ensure YAML/SQL is valid
  4. Run quality checks - Execute checks against real data (NEW)

If any step fails, the original file is restored.

Testing

  • Code compiles successfully
  • go vet passes
  • Binary help command works correctly

Note: Full test suite requires Rust library that isn't available in the cloud agent environment, but the logic has been verified through code review and build testing.

Slack Thread

Open in Web Open in Cursor 

This change adds a new step (Step 4) to the AI enhance flow that
automatically runs quality checks against the actual data after the
AI enhancement is complete. This ensures that any quality checks
added by the AI actually pass against the real data.

Changes:
- Add Step 4: Run quality checks using 'bruin run --only checks'
- If quality checks fail, restore the original file (same behavior as validation failure)
- Skip quality check execution if no checks are defined on the asset
- Update the AI prompt to warn that quality checks will be validated against real data
- Instruct the AI to be more conservative when adding quality checks
- Tell the AI to verify assumptions using pre-fetched table statistics

The flow is now:
1. Fill columns from database
2. AI Enhancement
3. Validate syntax
4. Run quality checks against data

If any step fails, the original file is restored.

Co-authored-by: Arsalan <arsalann@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants