feat(promptfoo): add crab pf serve Slack polling daemon (#37)

## Why

Team members — including non-technical people — need a way to run the
promptfoo target discovery agent without setting up a local dev
environment. A Slack-based interface lets anyone DM the Crab bot with a
target specification and get back a ready-to-use `promptfooconfig.yaml`.

## What

Adds `crab pf serve`, a local Slack polling daemon that:

- **Polls for DMs** to the Crab bot, filtering by a `pf:` trigger prefix
- **Runs the discovery agent** locally using the operator's own API keys
- **Posts results** (generated config files) back to the Slack thread
- **Handles file attachments** — users can attach API specs, curl
commands, etc.

### Files changed

| File | Change |
|------|--------|
| `plugins/promptfoo/src/slack.ts` | **New** — Slack Web API helpers
using native `fetch()`, zero new dependencies. Token resolution, user
lookup, DM channel management, message posting, reactions, file
download/upload (3-step flow). |
| `plugins/promptfoo/src/serve.ts` | **New** — Daemon loop with
`--setup` flow (prompts for Slack username and provider preferences),
poll loop, job directory management. Defaults to `openai:gpt-5` with
`reasoning: low`. |
| `plugins/promptfoo/src/cli.ts` | **Modified** — Routes `serve`
subcommand, adds `--reasoning` flag support, updates help text. |
| `plugins/promptfoo/src/agent/providers.ts` | **Modified** — Adds
`reasoningEffort` support for GPT-5/o-series models
(`max_completion_tokens` instead of `max_tokens`, skips `temperature`
for reasoning models). |
| `plugins/promptfoo/package-lock.json` | **Modified** — Minor peer
dependency annotation changes. |

## How to Test

1. Set `CRAB_SLACK_BOT_TOKEN` env var (or add `slack.bot_token` to
`~/.crabcode/config.yaml`)
2. Set `OPENAI_API_KEY` env var
3. Run `crab pf serve --setup` and follow prompts to configure your
Slack username
4. Run `crab pf serve -v` to start the daemon in verbose mode
5. DM the Crab bot in Slack with `pf: My API is at
http://localhost:8080/chat`
6. Verify the bot acknowledges with a reaction, runs the agent, and
posts back a `promptfooconfig.yaml`

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

Author: MrFlounder

plugins/promptfoo/package-lock.json

@@ -56,6 +56,7 @@ export class OpenAIProvider implements LLMProvider {
   }
 
   async chat(options: ChatOptions): Promise<ChatResponse> {
+    const isReasoning = this.model.startsWith('gpt-5') || this.model.startsWith('o1') || this.model.startsWith('o3');
     const response = await fetch(`${this.baseUrl}/chat/completions`, {
       method: 'POST',
       headers: {
@@ -66,13 +67,13 @@ export class OpenAIProvider implements LLMProvider {
         model: this.model,
         messages: options.messages.map((m) => this.toOpenAIMessage(m)),
         tools: options.tools,
-        ...(this.model.startsWith('gpt-5') || this.model.startsWith('o1') || this.model.startsWith('o3')
+        ...(isReasoning
           ? { max_completion_tokens: options.maxTokens || 4096 }
           : { max_tokens: options.maxTokens || 4096 }),
-        ...(this.model.startsWith('gpt-5') || this.model.startsWith('o1') || this.model.startsWith('o3')
-          ? {}
-          : { temperature: options.temperature ?? 0.7 }),
-        ...(options.reasoningEffort || this.reasoningEffort ? { reasoning_effort: options.reasoningEffort || this.reasoningEffort } : {}),
+        ...(isReasoning ? {} : { temperature: options.temperature ?? 0.7 }),
+        ...(options.reasoningEffort || this.reasoningEffort
+          ? { reasoning_effort: options.reasoningEffort || this.reasoningEffort }
+          : {}),
       }),
     });
 

plugins/promptfoo/src/agent/providers.ts

@@ -15,11 +15,18 @@ import * as path from 'node:path';
 import { parseArtifact, detectFormat } from './parsers/index.js';
 import { runDiscoveryAgent } from './agent/loop.js';
 import { createProvider } from './agent/providers.js';
+import { runServe } from './serve.js';
 
 const args = process.argv.slice(2);
 
 async function main() {
   try {
+    // Handle 'serve' subcommand
+    if (args[0] === 'serve') {
+      await runServe(args.slice(1));
+      return;
+    }
+
     // Parse arguments
     const filePath = getArg('--file') || getArg('-f');
     const urlArg = getArg('--url');
@@ -160,6 +167,11 @@ Examples:
   # Using Anthropic
   crab pf --file target.txt --provider anthropic:claude-sonnet-4-20250514
 
+Subcommands:
+  serve                  Run Slack polling daemon
+  serve --setup          Configure Slack username
+  serve --help           Show serve help
+
 Output:
   The agent will create:
   - promptfooconfig.yaml (promptfoo configuration)