Skip to content

CI_WORKFLOW_STRUCTURE.md

Latest commit

 

History

History
212 lines (171 loc) · 6.3 KB

CI_WORKFLOW_STRUCTURE.md

File metadata and controls

212 lines (171 loc) · 6.3 KB
Raw

CI Workflow Structure

This document describes the new component-based CI/CD workflow structure for FIML.

Overview

The CI/CD pipeline has been reorganized from a single monolithic workflow into 9 specialized workflows. This provides:

  • Faster feedback - Core tests run on every commit
  • Better diagnostics - Easy to identify which component is failing
  • Parallel execution - Independent test suites run in parallel
  • Efficient resource usage - Only run relevant tests based on changed files

Workflow Files

1. ci.yml - Main CI Pipeline (Core Tests Only)

Triggers: Every push and PR to main/develop Tests: Core functionality and cache tests only Purpose: Fast feedback loop for fundamental functionality

Tests:
- test_core.py (15 tests)
- test_cache.py (18 tests) 
- test_cache_improved.py (14 tests)
- test_cache_optimizations.py (17 tests)
Total: 64 tests

2. test-core.yml - Core Component Tests

Triggers: All commits Coverage: fiml/core/, fiml/cache/ Badge: Core tests always run and should always pass

3. test-providers.yml - Data Provider Tests

Triggers: PRs that change provider code or tests Coverage: fiml/providers/ Tests:

  • test_providers.py
  • test_providers_advanced.py
  • test_provider_registry.py
  • test_new_providers.py
  • test_phase2_providers.py
  • test_newsapi_integration.py

4. test-arbitration.yml - Provider Arbitration Tests

Triggers: PRs that change arbitration code or tests Coverage: fiml/arbitration/ Tests:

  • test_arbitration.py
  • test_arbitration_advanced.py
  • test_arbitration_coverage.py

5. test-dsl.yml - DSL Parser Tests

Triggers: PRs that change DSL code or tests Coverage: fiml/dsl/ Tests:

  • test_dsl_advanced.py
  • test_dsl_coverage.py

6. test-mcp.yml - MCP Server Tests

Triggers: PRs that change MCP/server code or tests Coverage: fiml/mcp/, fiml/server.py Tests:

  • test_mcp_and_server.py
  • test_mcp_coverage.py
  • test_server.py

7. test-agents.yml - Agent Workflow Tests

Triggers: PRs that change agent/narrative code or tests Coverage: fiml/agents/, fiml/narrative/ Tests:

  • test_agents.py
  • test_agent_workflows.py
  • test_narrative.py
  • test_narrative_generation.py
  • test_mcp_narrative_integration.py

8. test-infrastructure.yml - Infrastructure Tests

Triggers: PRs that change infrastructure code or tests Coverage: fiml/compliance/, fiml/alerts/, fiml/watchdog/, fiml/sessions/, fiml/tasks/, fiml/websocket/ Tests:

  • test_compliance.py
  • test_alerts.py
  • test_watchdog.py
  • test_sessions.py
  • test_tasks.py
  • test_websocket.py
  • test_workers_integration.py

9. test-integration.yml - Integration Tests

Triggers:

  • All PRs
  • Daily schedule (midnight UTC)
  • Manual workflow dispatch Coverage: Full application integration Tests:
  • test_integration.py
  • test_e2e_api.py
  • test_dashboard.py
  • test_azure_openai.py

Note: test_live_system.py is excluded as it requires live external services.

Test Execution Matrix

Workflow On Push On PR On Schedule Path Triggers
ci.yml (core) ✅ ✅ ❌ All
test-core.yml ✅ ✅ ❌ All
test-providers.yml ❌ ✅ ❌ fiml/providers/**, tests/test_providers*
test-arbitration.yml ❌ ✅ ❌ fiml/arbitration/**, tests/test_arbitration*
test-dsl.yml ❌ ✅ ❌ fiml/dsl/**, tests/test_dsl*
test-mcp.yml ❌ ✅ ❌ fiml/mcp/**, fiml/server.py, tests/test_mcp*, tests/test_server.py
test-agents.yml ❌ ✅ ❌ fiml/agents/, fiml/narrative/, tests/test_agents*, tests/test_narrative*
test-infrastructure.yml ❌ ✅ ❌ fiml/{compliance,alerts,watchdog,sessions,tasks,websocket}/**
test-integration.yml ❌ ✅ ✅ Daily All

Benefits

1. Faster Feedback

  • Core tests complete in ~1 minute
  • No waiting for full test suite on every commit
  • Quick identification of core functionality issues

2. Better Resource Utilization

  • Only run tests relevant to changed code
  • Parallel execution of independent test suites
  • Reduced CI/CD costs

3. Clearer Status

  • Core workflow badge shows fundamental health
  • Component-specific failures are isolated
  • Easier to diagnose and fix issues

4. Developer Experience

  • Less noise from unrelated test failures
  • Can manually trigger specific test suites
  • Integration tests run on schedule to catch issues early

Usage

For Contributors

On every commit:

  • Core tests run automatically
  • Must pass for merge

On pull requests:

  • Core tests + relevant component tests run
  • Based on which files were changed
  • All applicable tests must pass

Manual testing: You can trigger specific test suites manually:

# Run core tests locally
pytest tests/test_core.py tests/test_cache*.py

# Run provider tests locally  
pytest tests/test_providers*.py tests/test_newsapi*.py

# Run all tests
pytest tests/

For Maintainers

Monitoring:

  • Check core workflow badge for overall health
  • Review component-specific workflow results in PRs
  • Daily integration test runs catch integration issues

Debugging:

  • Failed core tests → fundamental issue
  • Failed component tests → isolated to that component
  • Failed integration tests → cross-component issue

Coverage Reporting

Each workflow uploads coverage to Codecov with specific flags:

  • core - Core and cache coverage
  • providers - Provider coverage
  • arbitration - Arbitration coverage
  • dsl - DSL coverage
  • mcp - MCP server coverage
  • agents - Agent and narrative coverage
  • infrastructure - Infrastructure coverage
  • integration - Full integration coverage

This allows tracking coverage per component over time.

Migration Notes

Before:

  • Single ci.yml ran all 35 test files (~439 tests)
  • Any test failure blocked the entire pipeline
  • Long execution time (~5-10 minutes)

After:

  • Core workflow runs 4 test files (64 tests) in ~1 minute
  • Component tests run in parallel on PRs
  • Total time reduced, better parallelization
  • Isolated failure domains

Future Improvements

  • Add performance test workflows
  • Add security scanning workflows
  • Implement test result caching
  • Add benchmark comparison workflows
  • Create nightly comprehensive test runs