You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When an LLM is told "use bigfoot for testing" in a project, it installs from PyPI and learns the API via introspection (docstrings, type hints, error messages). It never clones the bigfoot repo or reads AGENTS.md.
In a real-world test, an LLM made these mistakes:
Created a bare verifier via object.__new__(StrictVerifier) to avoid built-in plugins, instead of configuring [tool.bigfoot] enabled_plugins
Used verifier.sandbox() directly instead of with bigfoot:
Skipped forced assertions - tests passed without calling assert_query() because the bare verifier broke the enforcement contract
The result: bigfoot was reduced to a fancy unittest.mock wrapper with none of the actual quality guarantees.
What LLMs actually read (in priority order)
Docstrings on classes/functions they import (BasePlugin.__doc__, StrictVerifier.__doc__)
Error messages when they misuse the API
PyPI README (rendered on pypi.org, sometimes fetched by web search)
Type hints that Pyright/mypy flag
They do NOT read: repo README (unless they web search), AGENTS.md, CONTRIBUTING.md, or source comments.
Proposed changes
1. Module-level docstring (bigfoot/__init__.py)
Add a concise usage guide that LLMs will see on import bigfoot:
"""bigfoot - Full-certainty test mocking.Quick start: # 1. Configure in pyproject.toml: # [tool.bigfoot] # guard = true # # 2. Mock, execute, assert: bigfoot.http.mock_response("GET", "/api", json={"ok": True}) with bigfoot: response = requests.get("/api") bigfoot.http.assert_request("GET", "/api", status=200) # 3. Every intercepted call MUST be asserted. # Unasserted interactions raise UnassertedInteractionsError. # This is the core guarantee. Do not bypass it.Anti-patterns: - NEVER create StrictVerifier directly. Use `with bigfoot:` context. - NEVER use verifier.sandbox() directly. Use `with bigfoot:`. - NEVER skip assert_* calls. Every mock MUST be asserted. - Configure plugins via [tool.bigfoot], not by code."""
2. StrictVerifier docstring
classStrictVerifier:
"""Manages plugin lifecycle, sandbox context, and assertion verification. Do NOT instantiate directly. Use bigfoot's pytest integration: # In your test: with bigfoot: do_something() bigfoot.assert_something(...) Direct instantiation bypasses forced assertion checking and will silently produce tests that pass without verifying anything. """
3. BasePlugin docstring
classBasePlugin:
"""Base class for bigfoot plugins. To write a custom plugin: 1. Subclass BasePlugin 2. Register via [tool.bigfoot] in pyproject.toml 3. Implement all abstract methods 4. Use `with bigfoot:` in tests (never verifier.sandbox()) See bigfoot documentation for the plugin authoring guide. """
4. Runtime guardrails
StrictVerifier.__init__: if called outside pytest context, emit a warning: "StrictVerifier instantiated directly. Use with bigfoot: for proper assertion enforcement."
Or make object.__new__(StrictVerifier) raise/warn if _plugins is never set
5. Error messages that teach
When UnmockedInteractionError is raised, include in the message:
Unmocked call to [source_id]. Add a mock before the sandbox:
bigfoot.http.mock_response(...)
with bigfoot:
...
When UnassertedInteractionsError is raised:
1 interaction was not asserted. Every intercepted call must be
verified with an assert_* call after the sandbox closes:
with bigfoot:
result = do_something()
bigfoot.http.assert_request(...) # <-- required
6. PyPI README
Add a "Common Mistakes" section that will be indexed by search engines and LLM training data:
## Common Mistakes### Don't skip assertions
Every intercepted interaction must be asserted. If you mock a call
but never assert it, bigfoot raises `UnassertedInteractionsError`.
This is intentional - it prevents tests that pass without testing.
### Don't create verifiers directly
Use `with bigfoot:`, not `StrictVerifier()` or `verifier.sandbox()`.
### Don't bypass the plugin system
Configure active plugins in pyproject.toml, not in code:
```toml
[tool.bigfoot]
enabled_plugins = ["http", "my_custom_plugin"]
## Why this matters
LLMs are increasingly writing tests. Bigfoot's value prop (forced assertions, no silent passes) only works if consumers use the API correctly. Docstrings are the primary documentation surface for LLM consumers. Investing here prevents every future LLM integration from repeating the same mistakes.
Problem
When an LLM is told "use bigfoot for testing" in a project, it installs from PyPI and learns the API via introspection (docstrings, type hints, error messages). It never clones the bigfoot repo or reads AGENTS.md.
In a real-world test, an LLM made these mistakes:
object.__new__(StrictVerifier)to avoid built-in plugins, instead of configuring[tool.bigfoot] enabled_pluginsverifier.sandbox()directly instead ofwith bigfoot:assert_query()because the bare verifier broke the enforcement contractbigfoot._base_plugin,bigfoot._timeline) instead of public API (separate issue: Expose plugin authoring internals as public API #46)The result: bigfoot was reduced to a fancy
unittest.mockwrapper with none of the actual quality guarantees.What LLMs actually read (in priority order)
BasePlugin.__doc__,StrictVerifier.__doc__)They do NOT read: repo README (unless they web search), AGENTS.md, CONTRIBUTING.md, or source comments.
Proposed changes
1. Module-level docstring (
bigfoot/__init__.py)Add a concise usage guide that LLMs will see on
import bigfoot:2. StrictVerifier docstring
3. BasePlugin docstring
4. Runtime guardrails
StrictVerifier.__init__: if called outside pytest context, emit a warning: "StrictVerifier instantiated directly. Usewith bigfoot:for proper assertion enforcement."object.__new__(StrictVerifier)raise/warn if_pluginsis never set5. Error messages that teach
When
UnmockedInteractionErroris raised, include in the message:When
UnassertedInteractionsErroris raised:6. PyPI README
Add a "Common Mistakes" section that will be indexed by search engines and LLM training data: