run_eval.py: claude -p never triggers skills/commands (0% trigger rate across all queries)

[dthau120391]

Bug Description

run_eval.py creates a command file in .claude/commands/ with the skill description in YAML frontmatter, then runs claude -p with test queries to check if Claude invokes the Skill or Read tool for that command. In practice, no query ever triggers the skill — all should-trigger queries get 0/3 trigger rate, including an explicit "run /lucas:memory-layer please".

Reproduction

Tested with 8 different skills (144 total queries across all eval sets). Every should-trigger query scored 0/3 triggers. Every should-not-trigger query also scored 0/3 (correctly, but vacuously).

Diagnostic test with maximally obvious trigger:

[{"query": "I need to use the lucas memory layer skill right now, run /lucas:memory-layer please", "should_trigger": true}]

Result: 0/3 triggers.

Environment

• Claude Code v2.1.71
• macOS Darwin 25.3.0
• Model: claude-opus-4-6
• Ran from external terminal (no active Claude Code session)

Analysis

The issue appears to be that claude -p does not auto-trigger commands placed in .claude/commands/ the same way an interactive Claude Code session triggers skills from ~/.claude/skills/. Possible causes:

1. Commands vs Skills: Commands in .claude/commands/ may not appear in Claude's available_skills list the same way skills do, or may only be invocable via explicit /command-name syntax
2. claude -p mode: The -p (pipe/prompt) mode may not load commands/skills at all, or may not present them to the model
3. Simple query threshold: Claude may not consult skills for queries it can handle directly — but even explicit "run this skill" failed, ruling this out as the sole cause

Expected Behavior

At minimum, an explicit request like "run /skill-name please" should trigger the command (rate > 0/3). Description-matched queries should trigger at a reasonable rate.

Suggested Fix

• Verify that claude -p actually loads and presents .claude/commands/ files to the model
• Consider placing test skills in ~/.claude/skills/ instead of .claude/commands/ if skills and commands are triggered differently
• Add a --debug flag to run_eval.py that dumps what Claude actually sees (system prompt, available skills list) to verify the command file is being presented

[Curt-Park]

What I found

I added a command under .claude/commands and checked /skills in claude code.
It lists the command as a project skill.

.claude/commands/test-trigger-abc123.md

---
description: Test skill for trigger detection. Use when user says "test-trigger-abc123".
---
# Test Skill
Say: "test-trigger-abc123 was triggered"

Test

 ▐▛███▜▌   Claude Code v2.1.71                                                                                                                                          
▝▜█████▛▘  Sonnet 4.6 · Claude Max                                                                                                                                      
  ▘▘ ▝▝    ~/Repositories/autology                                                                                                                                      
                                                                                                                                                                        
❯ /skills                                                                                                                                                               
                                                                                                                                                                        
──────────────────────────
  Skills                                                                                                                                                                
  18 skills                                                                                                                                                             
                                                                                                                                                                        
  Project skills (.claude/skills, .claude/commands)
  test-trigger-abc123 · ~24 description tokens

  Plugin skills (plugin)
  ....

However, it doesn't list it in a headless mode: claude -p "list skills"