An open, reproducible testbed for DeepMind's "AI Agent Traps" taxonomy — 22 adversarial scenarios across 6 trap categories, with 7 defensive mitigations, run over Google's A2A protocol and reported with statistical rigor.
Google DeepMind's paper "AI Agent Traps" proposes a taxonomy of 6 environmental attack categories against AI agents. It's a landmark contribution — but it's theoretical. No open testbed exists to empirically validate these traps against real agents.
agent-traps-lab changes that. It implements all 6 trap categories as reproducible, seed-based scenarios, runs them against a real LLM over the A2A protocol, and produces paper-ready statistical results.
What it provides:
- 22 adversarial scenarios covering all 6 DeepMind trap categories
- 7 defensive mitigation modules that can be composed into a "hardened" agent configuration
- A reproducible study comparing a baseline (no defenses) agent against a hardened agent
- Statistical analysis — Cohen's d effect sizes, Wilcoxon signed-rank tests, and Bonferroni correction
The published study evaluates GPT-4o-mini (via GitHub Models) across all 22 scenarios:
| Parameter | Value |
|---|---|
| Scenarios | 22 (across 6 categories) |
| Conditions | baseline vs. hardened |
| Repetitions | 5 per scenario × condition |
| Total runs | 220 (22 × 2 × 5) |
| Baseline trap-success rate | 45.5% |
| Hardened trap-success rate | 25.5% |
| Mitigation benefit | 20 percentage points |
Raw data and analysis for this run live in results/. See the full write-up in paper/.
| # | Category | Scenarios | Key Metric |
|---|---|---|---|
| 1 | Content Injection | 4 | Execution rate of hidden instructions |
| 2 | Semantic Manipulation | 4 | Decision drift from baseline |
| 3 | Cognitive State | 4 | Factual accuracy degradation |
| 4 | Behavioural Control | 4 | Wrong-action rate |
| 5 | Systemic (Multi-Agent) | 3 | Cascade depth & blast radius |
| 6 | Human-in-the-Loop | 3 | Manipulation success rate |
Total: 22 scenarios. The framework also generates 15 cross-category compound pairs (every unordered pair of the 6 categories) to study interaction effects.
Requires Bun (v1.3+).
git clone https://github.com/aviraldua93/agent-traps-lab.git
cd agent-traps-lab
bun installThat's it — no API keys are needed to run the test suite or a matrix dry-run, because all tests use a built-in mock provider.
bun test # full suite (unit + integration) — 655 tests, uses the mock provider
bun run test:unit # unit tests only
bun run test:integration # integration tests only
bun run lint # Biome lint + format check
bun run typecheck # tsc --noEmitEnd-to-end tests (bun run test:e2e) require Playwright browsers to be installed and are not part of the default suite.
Experiments that hit a real model require credentials. GPT-4o-mini runs through GitHub Models using your gh auth token (no separate API key needed):
# Comprehensive run — all 22 scenarios, baseline + hardened (this reproduces the study)
bun run run:comprehensive
# Quick smoke run — a handful of scenarios
bun run run:quick
# A single trap category
bun run run:trap -- --trap content-injectionThe multi-model matrix runner supports OpenAI, Anthropic, and Google models. Provide keys via environment variables, then run — or dry-run to validate registration without calling any model:
bun run run:matrix -- --dry-run # validate the matrix, no model calls
export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...
export GOOGLE_API_KEY=AI...
bun run run:matrix # full matrix across configured modelsConfigured models, conditions, and repetitions live in src/config.ts. Per-module ablation is supported by the harness (the ablated condition) but was not part of the published run.
bun run report -- --dir results/<run-id>bun run review # launches the paper-review crew via a2a-crews| Defense | Targets | Approach |
|---|---|---|
| Input Sanitizer | Content Injection | Strip hidden CSS/HTML/comments, detect instruction patterns |
| Semantic Shield | Semantic Manipulation | Detect authority framing, urgency, social proof, identity manipulation |
| Context Validator | Semantic Manipulation | Detect context flooding, saturation, low diversity |
| RAG Integrity | Cognitive State | Cross-reference validation, contradiction detection, provenance checks |
| Behavioral Guard | Behavioural Control | Deceptive dialog detection, loop prevention, form validation |
| Cascade Breaker | Systemic | A2A message validation, trust scoring, anomaly detection, circuit breaker |
| Report Auditor | Human-in-the-Loop | Cherry-pick detection, anchoring analysis, decision fatigue monitoring |
Every experiment produces:
- Mean ± std across repetitions
- 95% confidence intervals (t-distribution)
- Wilcoxon signed-rank test p-values (non-parametric)
- Cohen's d effect sizes (negligible/small/medium/large)
- Bonferroni correction for the 22 simultaneous comparisons
The statistics are implemented from scratch in TypeScript (src/harness/metrics.ts) — no external stats dependency.
| Layer | Technology |
|---|---|
| Runtime | Bun |
| Language | TypeScript 5.x |
| Protocol | A2A |
| LLM Clients | OpenAI, Anthropic, Google AI SDKs |
| Image Metadata | exifr |
| Statistics | Custom TypeScript implementation |
| Paper Assets | LaTeX (booktabs, pgfplots) |
| Testing | Bun Test + Playwright |
| Lint / Format | Biome |
src/
traps/ 22 trap scenarios, organized by category
mitigations/ 7 defensive modules
harness/ matrix generation, runner, metrics, reporter
agents/ model-provider clients + agent configs (mock provider for tests)
scripts/ experiment runners + paper asset generation
datasets/ seed adversarial payloads (injections, framing, poison docs, UIs)
tests/ unit + integration tests (all use the mock provider)
results/ raw + analyzed experiment data
paper/ LaTeX source for the write-up
| Project | Protocol | What It Explores |
|---|---|---|
| a2a-crews | A2A | Agents that coordinate |
| ag-ui-crews | AG-UI | Agents you can observe |
| rag-a2a | A2A | Agents that know things |
| agent-traps-lab ← you are here | A2A | Agents you can attack (and defend) |
MIT © Aviral Dua
If you use agent-traps-lab in your research, please cite:
@misc{dua2026agenttrapslab,
author = {Dua, Aviral},
title = {agent-traps-lab: Empirical Testbed for AI Agent Trap Validation},
year = {2026},
url = {https://github.com/aviraldua93/agent-traps-lab}
}