Skip to content

Improve docs for LLM consumers: docstrings, anti-patterns, runtime guardrails #47

Description

@elijahr

Problem

When an LLM is told "use bigfoot for testing" in a project, it installs from PyPI and learns the API via introspection (docstrings, type hints, error messages). It never clones the bigfoot repo or reads AGENTS.md.

In a real-world test, an LLM made these mistakes:

  1. Created a bare verifier via object.__new__(StrictVerifier) to avoid built-in plugins, instead of configuring [tool.bigfoot] enabled_plugins
  2. Used verifier.sandbox() directly instead of with bigfoot:
  3. Skipped forced assertions - tests passed without calling assert_query() because the bare verifier broke the enforcement contract
  4. Imported private modules (bigfoot._base_plugin, bigfoot._timeline) instead of public API (separate issue: Expose plugin authoring internals as public API #46)

The result: bigfoot was reduced to a fancy unittest.mock wrapper with none of the actual quality guarantees.

What LLMs actually read (in priority order)

  1. Docstrings on classes/functions they import (BasePlugin.__doc__, StrictVerifier.__doc__)
  2. Error messages when they misuse the API
  3. PyPI README (rendered on pypi.org, sometimes fetched by web search)
  4. Type hints that Pyright/mypy flag

They do NOT read: repo README (unless they web search), AGENTS.md, CONTRIBUTING.md, or source comments.

Proposed changes

1. Module-level docstring (bigfoot/__init__.py)

Add a concise usage guide that LLMs will see on import bigfoot:

"""bigfoot - Full-certainty test mocking.

Quick start:
    # 1. Configure in pyproject.toml:
    #    [tool.bigfoot]
    #    guard = true
    #
    # 2. Mock, execute, assert:
    bigfoot.http.mock_response("GET", "/api", json={"ok": True})
    with bigfoot:
        response = requests.get("/api")
    bigfoot.http.assert_request("GET", "/api", status=200)
    
    # 3. Every intercepted call MUST be asserted.
    #    Unasserted interactions raise UnassertedInteractionsError.
    #    This is the core guarantee. Do not bypass it.

Anti-patterns:
    - NEVER create StrictVerifier directly. Use `with bigfoot:` context.
    - NEVER use verifier.sandbox() directly. Use `with bigfoot:`.
    - NEVER skip assert_* calls. Every mock MUST be asserted.
    - Configure plugins via [tool.bigfoot], not by code.
"""

2. StrictVerifier docstring

class StrictVerifier:
    """Manages plugin lifecycle, sandbox context, and assertion verification.
    
    Do NOT instantiate directly. Use bigfoot's pytest integration:
    
        # In your test:
        with bigfoot:
            do_something()
        bigfoot.assert_something(...)
    
    Direct instantiation bypasses forced assertion checking and will
    silently produce tests that pass without verifying anything.
    """

3. BasePlugin docstring

class BasePlugin:
    """Base class for bigfoot plugins.
    
    To write a custom plugin:
    
    1. Subclass BasePlugin
    2. Register via [tool.bigfoot] in pyproject.toml
    3. Implement all abstract methods
    4. Use `with bigfoot:` in tests (never verifier.sandbox())
    
    See bigfoot documentation for the plugin authoring guide.
    """

4. Runtime guardrails

  • StrictVerifier.__init__: if called outside pytest context, emit a warning: "StrictVerifier instantiated directly. Use with bigfoot: for proper assertion enforcement."
  • Or make object.__new__(StrictVerifier) raise/warn if _plugins is never set

5. Error messages that teach

When UnmockedInteractionError is raised, include in the message:

Unmocked call to [source_id]. Add a mock before the sandbox:
    bigfoot.http.mock_response(...)
    with bigfoot:
        ...

When UnassertedInteractionsError is raised:

1 interaction was not asserted. Every intercepted call must be
verified with an assert_* call after the sandbox closes:
    with bigfoot:
        result = do_something()
    bigfoot.http.assert_request(...)  # <-- required

6. PyPI README

Add a "Common Mistakes" section that will be indexed by search engines and LLM training data:

## Common Mistakes

### Don't skip assertions
Every intercepted interaction must be asserted. If you mock a call
but never assert it, bigfoot raises `UnassertedInteractionsError`.
This is intentional - it prevents tests that pass without testing.

### Don't create verifiers directly  
Use `with bigfoot:`, not `StrictVerifier()` or `verifier.sandbox()`.

### Don't bypass the plugin system
Configure active plugins in pyproject.toml, not in code:
```toml
[tool.bigfoot]
enabled_plugins = ["http", "my_custom_plugin"]

## Why this matters

LLMs are increasingly writing tests. Bigfoot's value prop (forced assertions, no silent passes) only works if consumers use the API correctly. Docstrings are the primary documentation surface for LLM consumers. Investing here prevents every future LLM integration from repeating the same mistakes.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions