Agent Instructions

This project uses bd (beads) for issue tracking. Run bd prime to refresh context after compaction.

Agent Initialization

Before any task, read the constitution.

1. Read `docs/constitution.md` — understand principles and fences
2. Read task context (spec, plan, or issue)
3. Then execute

Agents must internalize:

Implementation order: UI → Service → API → CLI → Seeder
Quality gates: Which block, which don't
Refactor checklist: Apply each loop, not at the end

If constitution conflicts with task instructions, constitution wins.

Project Overview

eval-ai-models is an AI Model Evaluation Framework with an integrated LLM-as-Judge Training System. It enables multi-model evaluation (OpenAI, Anthropic, Google), accuracy measurement using multiple rubrics, and iterative judge training with human-in-the-loop feedback.

Tech Stack: Astro 5 (SSR), TypeScript, Tailwind CSS 4, DaisyUI 5, SQLite (better-sqlite3), Vitest, Playwright

Tech Stack

Runtime: Node.js >= 22.0.0
Language: TypeScript 5.6+
Framework: Astro 5.x (SSR with Node adapter)
Styling: Tailwind CSS 4.x, daisyui (v5 beta/latest compatible with TW v4)
Database: SQLite via better-sqlite3
SDKs: OpenAI SDK, Anthropic SDK, Google Generative AI SDK
Testing: Vitest (unit/integration), Playwright (E2E)

Architecture Overview

Request Flow: Evaluation

POST /api/evaluate
  → validateCreateEvaluation()
  → insertEvaluation() [status: 'pending']
  → insertResult() for each model [status: 'pending']
  → startEvaluation() (async background execution)
    → EvaluationExecutor.execute()
      → Parallel model execution (30s timeout per model, 5min total)
      → Model client (OpenAIClient/AnthropicClient/GoogleClient)
      → Accuracy scoring (exact match/partial credit/semantic similarity)
      → updateResult() [status: 'completed'/'failed']
  → Client polls GET /api/evaluation-status

Request Flow: Judge Training

POST /api/personas/[id]/training/upload
  → parseCSV() → insertTrainingPairs()

POST /api/personas/[id]/training/start
  → createTrainingIteration()
  → startTrainingLoop() (async)
    → For each training pair:
      → Task model generates output → training_pair_results
      → Judge model evaluates → judge_decisions
    → Human review via POST .../iterations/[num]/feedback
    → calculateMetrics() → iteration_metrics (F1, precision, recall, Cohen's Kappa)
    → promptEngineer.refinePrompt() → judge_prompt_versions
    → Check convergence (F1 ≥ 0.80) or continue

Core Architectural Patterns

Separation of Concerns:

Database Layer (src/lib/db/): Raw SQL, CRUD operations, transactions
Validation Layer (src/lib/validation/): Input validation before DB/API
Business Logic (src/lib/evaluation/, src/lib/training/): Orchestration, algorithms
API Layer (src/pages/api/): HTTP handlers, response formatting
UI Layer (src/components/, src/pages/): Presentation

Multi-Provider Architecture:

ClientFactory creates provider-specific clients via adapter pattern
Unified ModelClient interface: evaluate(), testConnection()
Configuration: provider + model_name + encrypted API key

Async Execution:

Evaluations run in background; API returns evaluation_id immediately
Client polls for status/results
Training loops use state machine with pause/resume via checkpoints

Database Transactions:

Use withTransaction() wrapper from persona-db.ts for multi-step operations
Foreign key constraints enabled with CASCADE/RESTRICT
WAL mode for concurrent read/write

Project Structure

openapi.yml
src/
tests/
db/

Development Commands

Essential Commands

npm run dev              # Start dev server on port 3000
npm run build            # Build for production
npm run preview          # Preview production build

# Database
npm run db:init          # Initialize database from schema.sql
npm run db:reset         # Delete and reinitialize database
npm run db:e2e:clean     # Clean E2E test database

# Testing
npm test                 # Run unit/integration tests (Vitest)
npm run test:coverage    # Run tests with coverage (target >80% on critical paths)
npm run test:e2e         # Run E2E tests (Playwright)
npm run test:e2e:clean   # Clean E2E DB and run tests

# Quality Gates (MUST run before commits)
npm run typecheck        # TypeScript strict mode check + Astro component check
npm run lint             # ESLint check
npm run lint:fix         # Auto-fix lint issues
npm run format           # Format code with Prettier
npm run format:check     # Check formatting

# Development Tools
npm run storybook        # Component documentation (port 6006)
npm run check            # Astro type checking

Running Single Tests

# Vitest (unit/integration)
npm test -- path/to/test.test.ts
npm test -- --grep "specific test name"

# Playwright (E2E)
npx playwright test path/to/test.spec.ts
npx playwright test --grep "specific test name"
npx playwright test --debug  # Debug mode

Key File Locations

Core Business Logic

src/lib/evaluation/evaluator.ts - Evaluation orchestration, concurrency, timeouts
src/lib/evaluation/accuracy.ts - Rubric scoring (exact match, partial credit, semantic)
src/lib/evaluation/metrics.ts - F1, precision, recall, Cohen's Kappa calculation
src/lib/training/prompt-engineer.ts - LLM-based prompt refinement
src/lib/training/judge-runner.ts - Judge evaluation execution
src/lib/utils/api-clients.ts - Provider abstraction (OpenAI, Anthropic, Google)

Database

db/schema.sql - 13 tables (models, evaluations, personas, training)
src/lib/db/db.ts - Core database access layer
src/lib/db/persona-db.ts - Judge persona operations with transactions

Validation

src/lib/validation/validators.ts - Manual runtime validators
src/lib/validation/persona-validator.ts - Persona-specific validation

API Routes

src/pages/api/evaluate.ts - Create evaluation
src/pages/api/personas/ - Persona CRUD
src/pages/api/personas/[id]/training/ - Training operations
src/pages/api/personas/[id]/iterations/[num]/ - Iteration feedback/metrics

Types

src/lib/utils/types.ts - Core domain types (Evaluation, Result, ModelConfiguration)
src/types/training.ts - Judge training types (Persona, TrainingIteration)

Critical Development Patterns

Path Aliases (Vite/Astro)

import { evaluateInstruction } from '@lib/evaluation/evaluator';
import { Button } from '@components/ui/Button.astro';
import { database } from '@db/init';

API Error Handling

import { createErrorResponse, badRequest } from '@lib/api/api-error-handler';

export async function POST({ request }) {
  const validation = validateInput(data);
  if (!validation.valid) {
    return badRequest(validation.error.message);
  }

  try {
    // ... operation
  } catch (error) {
    return createErrorResponse(error);
  }
}

Database Transactions

import { withTransaction } from '@lib/db/persona-db';

const result = withTransaction(db, () => {
  db.prepare('INSERT ...').run();
  db.prepare('UPDATE ...').run();
  return computedValue;
});

API Key Encryption

import { encryptApiKey, decryptApiKey } from '@lib/utils/encryption';

// Before DB insert
const encrypted = encryptApiKey(plainKey);

// On retrieval
const plain = decryptApiKey(model.api_key_encrypted);

Evaluation Status Polling Pattern

// Server: Return evaluation_id immediately
return new Response(JSON.stringify({ evaluation_id }), { status: 201 });

// Client: Poll for completion
const pollStatus = async () => {
  const res = await fetch(`/api/evaluation-status?id=${evaluationId}`);
  const data = await res.json();
  if (data.status === 'completed' || data.status === 'failed') {
    // Handle completion
  } else {
    setTimeout(pollStatus, 1000);
  }
};

Testing Requirements

Coverage Targets

Critical paths (validators, accuracy, evaluator, metrics): >80% coverage
Current coverage: validators 84.29%, accuracy 92.85%, evaluator 93.05%

Test Structure

Unit tests (tests/unit/): Pure logic, metrics calculation, encryption
Integration tests (tests/integration/): Database operations, API handlers
E2E tests (tests/e2e/): Full user workflows with Playwright

E2E Database Isolation

# E2E tests use separate DB via EVAL_DB_PATH environment variable
# Database: db/evaluation.e2e-test.db
# Clean before test runs: npm run test:e2e:clean

Test-First Development

Per project constitution: Tests are written first for all critical paths. When adding features:

Write test cases first
Implement functionality
Verify coverage meets targets

Environment Configuration

Required Environment Variables

# .env (copy from .env.example)
ENCRYPTION_KEY=<32-byte-hex>  # Generate: node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
GOOGLE_API_KEY=...

# Optional
LOG_LEVEL=info|debug|warn|error
DEBUG=false  # Log model responses to console
MOCK_JUDGE_MODE=true  # Use mock data for training (dev only)
EVAL_DB_PATH=./db/evaluation.db  # Override database path

Mock Judge Mode

When MOCK_JUDGE_MODE=true, training loop uses mock responses instead of real LLM calls to reduce token costs during development.

Code Style & Quality

Style Rules

Formatter: Prettier (double quotes, semicolons, 2-space indent, 100 char line width)
Linter: ESLint 9 flat config with TypeScript, Astro, JSDoc plugins
TypeScript: Strict mode (strictNullChecks, noImplicitAny, noImplicitReturns)
Styling: Tailwind CSS v4 utility classes (prefer utilities over custom CSS)
JSDoc: Required for public functions/methods/classes

Pre-Commit Checklist

MANDATORY before every commit:

npm run typecheck    # Must pass
npm run lint         # Must pass
npm run format       # Auto-format code

Pre-Commit Quality Gates

Automated Hooks (Local)

This project uses simple-git-hooks and lint-staged to enforce quality gates automatically:

Pre-commit: Runs lint-staged on staged files (*.ts, *.tsx, *.astro, *.js, *.jsx)
- ESLint with auto-fix
- Prettier formatting
- Note: Typecheck is intentionally excluded from pre-commit for faster incremental development
Pre-push: Runs full test suite (optional, can be skipped with --no-verify)

Manual Quality Gate Runs

For manual verification before commits:

npm run lint         # ESLint check
npm run typecheck    # TypeScript strict mode check + Astro component check (run before PR)
npm run format       # Prettier auto-format
npm test             # Run full test suite

CI Quality Gates

GitHub Actions runs the following checks on every PR and push to main:

Lint: Full ESLint check on src/ and tests/
Type Check: TypeScript strict mode + Astro component check
Test: Full test suite with test database
Format: Prettier format check

All CI checks must pass before merging. Configure branch protection rules to require these status checks.

To reinstall hooks:

npm run prepare  # Sets up git hooks via simple-git-hooks

Or use the direct command:

npx simple-git-hooks  # Install git hooks from package.json config

Testing Status

Latest coverage (vitest npm test -- --coverage):

Overall line coverage: 9.24% (all files), 37.99% (evaluation module)
Critical path coverage (verified 2026-01-10 per eval-z5f):
- validators.ts: 86.01% lines (84.17% stmts, 82.42% branch, 95% funcs) ✅ >80%
- accuracy.ts: 92.85% lines (91.11% stmts, 80.76% branch, 100% funcs) ✅ >80%
- evaluator.ts: 93.05% lines (91.02% stmts, 70.58% branch, 93.75% funcs) ✅ >80%
Other coverage: api-clients.ts 64.38%, db.ts 62.62%
Constitution Principle III (line 48) SATISFIED: All critical paths >80% coverage

Styling Conventions

TOP PRIORITY: Always use Tailwind CSS v4 utility classes for styling.

Use Tailwind v4 syntax and features (e.g., @theme directive, CSS variables)
Leverage daisyUI component classes when appropriate
Only fall back to custom CSS when Tailwind utilities cannot achieve the desired result
When custom CSS is necessary, document why Tailwind was insufficient

Development Workflow

ALWAYS FOLLOW THIS IMPLEMENTATION WORKFLOW

Find work: Run bd ready to find unblocked tasks
Claim: Run bd update <id> --status in_progress for each task
Branch: git checkout -b feature/<descriptive-name>
Implement: For each task:
- Write code following component guidelines and design tokens
- Run npm run typecheck && npm run lint && npm run format:fix
- Commit with clear message: git commit -m "feat: descriptive message"
- Push: git push
PR: Create Pull Request following the GitHub PR template
Review: Invoke code-review-specialist agent to review (comments only, no code changes)
Complete: After merge:
- Run bd close <id> for each completed task
- Run bd sync
Report: Provide summary with PR link, tasks completed, review status, and any recommendations

Landing the Plane (Session Completion)

When ending a work session, you MUST complete ALL steps below. Work is NOT complete until git push succeeds.

MANDATORY WORKFLOW:

File issues for remaining work - Create issues for anything that needs follow-up
Run quality gates (if code changed) - Tests, linters, builds
Update issue status - Close finished work, update in-progress items

PUSH TO REMOTE - This is MANDATORY:

git pull --rebase
bd sync
git push
git status  # MUST show "up to date with origin"

Clean up - Clear stashes, prune remote branches
Verify - All changes committed AND pushed
Hand off - Provide context for next session

CRITICAL RULES:

Work is NOT complete until git push succeeds
NEVER stop before pushing - that leaves work stranded locally
NEVER say "ready to push when you are" - YOU must push
If push fails, resolve and retry until it succeeds

Database Schema Notes

Key Tables

ModelConfiguration: AI provider configs with encrypted API keys
Evaluation: Evaluation sessions with status tracking
Result: Model outputs with metrics (accuracy, latency, tokens)
personas: Judge configurations (task/judge/engineer models)
training_pairs: Input/output pairs for training
training_iterations: Iteration cycles (generation → judgment → metrics)
judge_decisions: Judge model assessments
human_reviews: Human reviewer feedback (mandatory early iterations)
iteration_metrics: F1, precision, recall, Cohen's Kappa, confusion matrix
judge_prompt_versions: Judge prompt version history

Important Constraints

Foreign keys with CASCADE/RESTRICT for referential integrity
Unique constraints on (persona_id, version_number) for prompts
Indexes on status, created_at, persona_id, f1_score for query performance

Common Pitfalls

Database initialization: Always run npm run db:init after cloning or schema changes
Encryption key: Must be 32-byte hex. Generate with provided command in .env.example
E2E test isolation: Use npm run test:e2e:clean to avoid stale test data
API key storage: NEVER commit unencrypted API keys. Use encryptApiKey() before DB insert
Async evaluation: Don't wait for evaluation results in POST handler; return evaluation_id immediately
Transaction safety: Use withTransaction() for multi-step DB operations
Path aliases: Use @lib, @components, etc. Don't use relative imports for cross-directory references
Tailwind v4 syntax: Use modern Tailwind v4 features (e.g., @theme directive) not v3 patterns

API Contract

Full REST API specification in openapi.yml. Key endpoints:

Evaluations: /api/evaluate, /api/evaluation-status, /api/results
Models: /api/models (CRUD, test connection)
Personas: /api/personas (CRUD, reset)
Training: /api/personas/[id]/training/* (upload, start, pause, resume, status)
Iterations: /api/personas/[id]/iterations/[num]/* (feedback, metrics, refine)
Prompts: /api/prompts/* (versions, optimize)
Templates: /api/templates (CRUD, run, import/export)

Beads Workflow Integration

This project uses beads_viewer for issue tracking. Issues are stored in .beads/ and tracked in git.

Essential Commands

# View issues (launches TUI - avoid in automated sessions)
bv

# CLI commands for agents (use these instead)
bd ready              # Show issues ready to work (no blockers)
bd list --status=open # All open issues
bd show <id>          # Full issue details with dependencies
bd create --title="..." --type=task --priority=2
bd update <id> --status=in_progress
bd close <id> --reason="Completed"
bd close <id1> <id2>  # Close multiple issues at once
bd sync               # Commit and push changes

Key Concepts

Dependencies: Issues can block other issues. bd ready shows only unblocked work.
Priority: P0=critical, P1=high, P2=medium, P3=low, P4=backlog (use numbers, not words)
Types: task, bug, feature, epic, question, docs
Blocking: bd dep add <issue> <depends-on> to add dependencies

Session Protocol

Before ending any session, run this checklist:

git status              # Check what changed
git add <files>         # Stage code changes
bd sync                 # Commit beads changes
git commit -m "..."     # Commit code
bd sync                 # Commit any new beads changes
git push                # Push to remote

Best Practices

Check bd ready at session start to find available work
Update status as you work (in_progress → closed)
Create new issues with bd create when you discover tasks
Use descriptive titles and set appropriate priority/type
Always bd sync before ending session

FilesExpand file tree

AGENTS.md

Latest commit

History

AGENTS.md

File metadata and controls

Agent Instructions

Agent Initialization

Project Overview

Tech Stack

Architecture Overview

Request Flow: Evaluation

Request Flow: Judge Training

Core Architectural Patterns

Project Structure

Development Commands

Essential Commands

Running Single Tests

Key File Locations

Core Business Logic

Database

Validation

API Routes

Types

Critical Development Patterns

Path Aliases (Vite/Astro)

API Error Handling

Database Transactions

API Key Encryption

Evaluation Status Polling Pattern

Testing Requirements

Coverage Targets

Test Structure

E2E Database Isolation

Test-First Development

Environment Configuration

Required Environment Variables

Mock Judge Mode

Code Style & Quality

Style Rules

Pre-Commit Checklist

Pre-Commit Quality Gates

Automated Hooks (Local)

Manual Quality Gate Runs

CI Quality Gates

Testing Status

Styling Conventions

Development Workflow

Landing the Plane (Session Completion)

Database Schema Notes

Key Tables

Important Constraints

Common Pitfalls

API Contract

Beads Workflow Integration

Essential Commands

Key Concepts

Session Protocol

Best Practices