Skip to content

feat(01-01): config system, Fernet crypto, setup wizard#1

Merged
coder-ishan merged 25 commits intomainfrom
feature/01-01-config-crypto
Feb 26, 2026
Merged

feat(01-01): config system, Fernet crypto, setup wizard#1
coder-ishan merged 25 commits intomainfrom
feature/01-01-config-crypto

Conversation

@coder-ishan
Copy link
Owner

What Was Built

Fernet-encrypted config system, setup wizard CLI, and package scaffold — the shared config layer every other module will import.

Key Files Created

  • pyproject.toml — hatchling build, all deps, asyncio_mode = "auto", job-hunter entry point
  • src/ingot/__init__.py__version__ = "0.1.0"
  • src/ingot/config/crypto.py — PBKDF2HMAC key derivation (600k iterations), get_fernet(), encrypt_secret(), decrypt_secret()
  • src/ingot/config/schema.pyAppConfig, AgentConfig, SmtpConfig, ImapConfig (Pydantic v2)
  • src/ingot/config/manager.pyConfigManager.load()/save() with atomic write, __encrypted__: prefix for secrets
  • src/ingot/cli/setup.py — full setup wizard: interactive (questionary) + non-interactive (env vars), --preset fully_free/best_quality, skips existing values, Rich summary table
  • src/ingot/cli/__init__.py — Typer app with setup subcommand
  • src/ingot/logging_config.py — structlog dual handlers (stderr WARNING+, rotating file DEBUG+ JSON)

Verification

  • python3 -c "import ingot; print(ingot.__version__)"0.1.0
  • Fernet roundtrip: decrypt_secret(encrypt_secret("hello")) == "hello"
  • All module imports clean: ConfigManager, AppConfig, setup_app

Commits

  • a53d0a8 feat(01-01): project scaffold, pyproject.toml, and package structure
  • b4b1f02 feat(01-01): Fernet crypto module and ConfigManager
  • fa2d6b7 feat(01-01): setup wizard CLI with interactive + non-interactive modes

🤖 Generated with Claude Code

coder-ishan and others added 4 commits February 26, 2026 10:17
- pyproject.toml with hatchling build backend, all dependencies, entry point
- src/ingot/__init__.py with __version__ = "0.1.0"
- src/ingot/config/__init__.py package init
- src/ingot/logging_config.py with structlog dual handlers (stderr WARNING+, rotating file DEBUG+ JSON)
- src/ingot/cli/__init__.py with Typer app and setup subcommand
- src/ingot/cli/setup.py stub (full implementation in Task 3)
- crypto.py: PBKDF2HMAC key derivation (600k iterations), get_fernet(), encrypt_secret(), decrypt_secret()
- schema.py: AppConfig, AgentConfig, SmtpConfig, ImapConfig Pydantic v2 models with 7 default agents
- manager.py: ConfigManager.load()/save() with atomic write (.tmp then rename), __encrypted__: prefix for secret fields
- Full questionary prompts with secret masking for API keys and passwords
- Non-interactive mode reads ANTHROPIC_API_KEY, OPENAI_API_KEY, GMAIL_* env vars
- --preset fully_free / best_quality sets all 7 agent model fields
- Skips fields already configured in config.json
- Rich summary table with secrets masked at end of setup
- Creates ~/.outreach-agent/ directory structure (logs/, resume/, venues/)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR implements the foundational configuration system for INGOT, including Fernet-based encryption for secrets, a Pydantic-based configuration schema, and an interactive/non-interactive setup wizard CLI. This is Plan 01-01, providing the essential infrastructure that all subsequent modules will depend on for loading credentials, selecting LLM backends, and locating application data.

Changes:

  • Added Fernet encryption system with PBKDF2 key derivation for protecting API keys and passwords at rest
  • Created Pydantic v2 configuration schema with support for 7 core agents and email/LLM service credentials
  • Implemented interactive and non-interactive setup wizard with preset configurations for different LLM deployment strategies
  • Added structlog-based logging configuration with dual handlers (stderr and rotating file)
  • Configured project build system with hatchling, dependencies, and test settings

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 8 comments.

Show a summary per file
File Description
pyproject.toml Project metadata, dependencies, pytest configuration, and CLI entry point definition
src/ingot/init.py Package version identifier
src/ingot/config/init.py Config subsystem package marker
src/ingot/config/crypto.py Fernet encryption/decryption with PBKDF2 key derivation from machine-specific key file
src/ingot/config/schema.py Pydantic models for application configuration including agent, SMTP, and IMAP settings
src/ingot/config/manager.py Configuration persistence with atomic writes and automatic secret encryption/decryption
src/ingot/logging_config.py Structured logging setup with stderr console output and rotating JSON file logs
src/ingot/cli/init.py Typer-based CLI application entry point
src/ingot/cli/setup.py Interactive and non-interactive setup wizard with preset support and validation
.planning/phases/01-foundation-and-core-infrastructure/01-01-SUMMARY.md Plan completion summary
Comments suppressed due to low confidence (14)

src/ingot/cli/setup.py:217

  • When password prompt returns None (user cancels), the code checks if password is falsy but uses the generic error message "Gmail App Password cannot be empty." This treats None (cancelled) the same as an empty string (user pressed enter). Consider distinguishing between cancellation and empty input, or at least catch the cancellation earlier to provide a clearer message.
        password = questionary.password("Gmail App Password:").ask()
        if not password:
            _err.print("[red]Gmail App Password cannot be empty.[/red]")
            raise typer.Exit(code=1)

src/ingot/config/manager.py:77

  • If the config.json file contains invalid JSON, json.loads() will raise a JSONDecodeError which is not caught. This could happen if the file is corrupted or manually edited incorrectly. Consider catching JSONDecodeError and providing a helpful error message, or letting it bubble up with context about which file failed to parse.
        raw = json.loads(self.config_path.read_text(encoding="utf-8"))

src/ingot/logging_config.py:17

  • The comment references "~/.outreach-agent/" but should reference "log files" or describe what the base_dir parameter represents more generally. The hardcoded path in the docstring doesn't match the function's flexibility to accept any base_dir.
        base_dir: Base directory for log files (e.g., ~/.outreach-agent/).

src/ingot/cli/setup.py:194

  • The errors list is initialized but never populated with any error messages. The list is checked at the end but no errors are ever added to it. Either remove the unused error handling code or implement the validation that should add errors (e.g., when required environment variables are missing).
    errors: list[str] = []

    gmail_username = os.environ.get("GMAIL_USERNAME", "")
    gmail_password = os.environ.get("GMAIL_APP_PASSWORD", "")
    anthropic_key = os.environ.get("ANTHROPIC_API_KEY", "")
    openai_key = os.environ.get("OPENAI_API_KEY", "")

    if gmail_username:
        cfg.smtp.username = gmail_username
        cfg.imap.username = gmail_username
    if gmail_password:
        cfg.smtp.password = gmail_password
        cfg.imap.password = gmail_password
    if anthropic_key:
        cfg.anthropic_api_key = anthropic_key
    if openai_key:
        cfg.openai_api_key = openai_key

    # Apply preset or default to fully_free
    effective_preset = preset or _PRESET_FULLY_FREE
    _apply_preset(cfg, effective_preset)

    # Ensure all 7 agents exist
    for agent_name in _AGENT_NAMES:
        if agent_name not in cfg.agents:
            cfg.agents[agent_name] = AgentConfig()

    if errors:
        for error in errors:
            _err.print(f"[red]{error}[/red]")
        raise typer.Exit(code=1)

src/ingot/logging_config.py:31

  • The date format uses the current date at the time configure_logging is called, but if the application runs across midnight, logs will continue writing to the old date's file. Consider whether this is the intended behavior or if log rotation should happen at midnight. The RotatingFileHandler rotates based on size, not time.
    date_str = datetime.now().strftime("%Y-%m-%d")
    log_file = log_dir / f"run-{date_str}.log"

src/ingot/config/crypto.py:15

  • The stat module is imported but never used. The chmod call on line 59 uses the literal 0o600 instead of stat module constants. Either remove the unused import or use stat.S_IRUSR | stat.S_IWUSR for better clarity.
import stat

src/ingot/cli/init.py:21

  • The import of setup_app is placed after code execution (after the app callback function) to avoid circular imports, indicated by the noqa: E402 comment. While this works, consider restructuring the module to avoid this pattern - perhaps by moving the command registration to a separate function or using lazy imports. The current structure makes the code flow less intuitive.
from ingot.cli.setup import setup_app  # noqa: E402

src/ingot/cli/setup.py:66

  • Secrets are displayed in the summary table with partial masking via the _mask function. However, showing even the first 4 characters of API keys or passwords could aid in brute-force attacks if an attacker can see the output. Consider either masking more characters or only showing them for debugging purposes with a flag. This is especially important for the summary output which might be logged or shared in screenshots.
def _mask(value: str, show_chars: int = 4) -> str:
    """Return a masked version of a secret for display."""
    if not value:
        return "(not set)"
    if len(value) <= show_chars:
        return "*" * len(value)
    return value[:show_chars] + "*" * (len(value) - show_chars)

src/ingot/cli/setup.py:131

  • The traceback import is unused - it's imported but never called. Either use traceback.print_exc() or remove the import.
        import traceback

src/ingot/cli/setup.py:132

  • The import statement appears after code execution due to the late import inside the exception handler. This violates PEP 8 conventions which recommend imports at the top of the file. Move the logging and traceback imports to the top of the file with the other imports.
        import traceback
        import logging

src/ingot/cli/setup.py:143

  • The verbose parameter is accepted but never used in the setup wizard. It's passed through _run_setup but not utilized to control output verbosity or logging level. Either implement verbose output levels or remove the unused parameter.
    verbose: int = typer.Option(0, "-v", count=True, max=2),
) -> None:
    """Run the INGOT setup wizard to configure credentials and LLM backends."""
    try:
        _run_setup(non_interactive=non_interactive, preset=preset, verbose=verbose)
    except KeyboardInterrupt:
        _err.print("\n[yellow]Setup cancelled.[/yellow]")
        raise typer.Exit(code=1)
    except typer.Exit:
        raise
    except Exception as exc:
        log_path = Path.home() / ".outreach-agent" / "logs"
        _err.print(f"[red][Setup] Something went wrong. Full error logged to {log_path}[/red]")
        # Log the full traceback
        import traceback
        import logging
        logging.getLogger("ingot.cli.setup").error(
            "Setup wizard failed", exc_info=True
        )
        raise typer.Exit(code=1) from exc


def _run_setup(
    *,
    non_interactive: bool,
    preset: str | None,
    verbose: int,

pyproject.toml:16

  • The version constraint ">=44" for cryptography is unusual and may be incorrect. The cryptography library typically uses semantic versioning (e.g., 44.0.0). This constraint would match any version >= 44.0.0, but versions typically look like "44.0.0" not "44". Consider using a more standard constraint like ">=44.0.0" or ">=44.0" to be more explicit.
    "cryptography>=44",

src/ingot/config/manager.py:96

  • The atomic write uses with_suffix(".tmp") which replaces the file extension. For "config.json", this produces "config.tmp" which is correct. However, if config_path ever doesn't have a suffix (e.g., "config"), this would produce "config.tmp" which is still fine. Consider using a more explicit temp file pattern like config_path.with_name(config_path.name + ".tmp") or documenting why with_suffix is sufficient here.
        tmp_path = self.config_path.with_suffix(".tmp")

src/ingot/config/schema.py:72

  • The default factory lambda creates a new list each time, which is correct for mutable defaults. However, the list could be defined as a module-level constant to avoid recreating it. Consider: _DEFAULT_LLM_FALLBACK = ["claude", "openai", "ollama"] and then use default_factory=lambda: _DEFAULT_LLM_FALLBACK.copy()
    llm_fallback_chain: list[str] = Field(
        default_factory=lambda: ["claude", "openai", "ollama"]
    )

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

- crypto.py: eliminate TOCTOU race on key file creation by using
  os.open with O_CREAT|O_EXCL|0o600 instead of write_bytes + chmod;
  also removes unused `stat` import
- setup.py: loop on API key prompts when the required model provider
  needs a key — prevents silently storing an empty string that causes
  runtime failures
- setup.py: wire configure_logging into _run_setup after ensure_dirs
  so structlog handlers are active for the full wizard lifecycle;
  import moved to module level

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@coder-ishan
Copy link
Owner Author

Addressed the three valid findings from this review (commit 9162af0):

Fixed:

  • TOCTOU race (crypto.py) — replaced write_bytes + chmod with os.open(O_CREAT|O_EXCL, 0o600) so the key file is created with restricted permissions atomically. Also removed the unused stat import.
  • Empty API key silently stored (setup.py) — the Anthropic/OpenAI key prompts now loop until a non-empty value is entered, preventing a config save that will always fail at runtime.
  • configure_logging never called (setup.py) — wired into _run_setup after ensure_dirs() so structlog handlers are active for the full wizard lifecycle.

Disputed (code already handles these correctly):

  • Comments on setup.py:247, :208, :229, :309 — all four claim AttributeError from calling .strip() or .startswith() on a cancelled None prompt, but each call site is already guarded by if value: or if value and value.startswith(...). Replied inline on each thread.

coder-ishan and others added 13 commits February 26, 2026 11:01
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Rename CLI entry point from `job-hunter` to `ingot` (pyproject.toml
  and Typer app name)
- Replace all `~/.outreach-agent/` references with `~/.ingot/` across
  crypto.py, manager.py, schema.py, logging_config.py, and setup.py

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…#2)

* feat(01-02): async SQLite engine with WAL mode and all 11 SQLModel models

- engine.py: create_async_engine + WAL/synchronous/cache/fk PRAGMAs via sync_engine event listener
- models.py: UserProfile, Lead, IntelBrief, Match, Email, FollowUp, Campaign, AgentLog, Venue, OutreachMetric, UnsubscribedEmail
- JSON columns for list fields (skills, experience, education, signals, talking_points)
- str-backed enums for Lead/Email/FollowUp/Campaign status (SQLite has no native enum)
- repositories/base.py: BaseRepository[T] with add/get/list/delete using AsyncSession

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(01-02): Alembic async migration setup with initial schema

- alembic.ini: standard config with sqlite+aiosqlite URL
- alembic/env.py: async online migration runner; explicit model imports before target_metadata to prevent empty autogenerate
- alembic/script.py.mako: template with sqlmodel import included
- alembic/versions/149adcd94073_initial_schema.py: autogenerated CREATE TABLE for all 11 models

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* docs(01-02): plan complete — summary

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(01-02): address copilot review findings

- engine.py: use .as_posix() for cross-platform SQLite URL safety
- repositories/base.py: remove erroneous await from session.delete()
- models.py: add nullable=False to all JSON list/dict columns
- alembic/env.py: inject src/ into sys.path for non-editable installs

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------
- STATE.md: reflects 01-04 complete, Wave 4 (01-05) next
- Phase 2 plan files: 02-01 through 02-07 + CONTEXT.md
- outreach-agent-plan.md: original feature backlog reference

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… hierarchy (#3)

* feat(01-03): LLMClient, XML fallback, and typed exception hierarchy

- agents/exceptions.py: IngotError → LLMError, LLMValidationError, DBError, ConfigError, ValidationError, AgentError
- llm/fallback.py: xml_extract() — scalar and list fields from XML tags, raises LLMValidationError on failure
- llm/schemas.py: LLMMessage, LLMRequest, LLMResponse Pydantic envelopes
- llm/client.py: LLMClient with tool-call → JSON → XML fallback priority, tenacity 3-attempt exponential backoff
- No direct anthropic/openai imports anywhere — all routing via litellm.acompletion

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* docs(01-03): plan complete — summary

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(01-03): address copilot review findings

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(01-04): PydanticAI agent framework — 7 shells, registry, HTTP client, dispatcher (#4)

* feat(01-02): async SQLite engine with WAL mode and all 11 SQLModel models

- engine.py: create_async_engine + WAL/synchronous/cache/fk PRAGMAs via sync_engine event listener
- models.py: UserProfile, Lead, IntelBrief, Match, Email, FollowUp, Campaign, AgentLog, Venue, OutreachMetric, UnsubscribedEmail
- JSON columns for list fields (skills, experience, education, signals, talking_points)
- str-backed enums for Lead/Email/FollowUp/Campaign status (SQLite has no native enum)
- repositories/base.py: BaseRepository[T] with add/get/list/delete using AsyncSession

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(01-02): Alembic async migration setup with initial schema

- alembic.ini: standard config with sqlite+aiosqlite URL
- alembic/env.py: async online migration runner; explicit model imports before target_metadata to prevent empty autogenerate
- alembic/script.py.mako: template with sqlmodel import included
- alembic/versions/149adcd94073_initial_schema.py: autogenerated CREATE TABLE for all 11 models

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* docs(01-02): plan complete — summary

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(01-02): address copilot review findings

- engine.py: use .as_posix() for cross-platform SQLite URL safety
- repositories/base.py: remove erroneous await from session.delete()
- models.py: add nullable=False to all JSON list/dict columns
- alembic/env.py: inject src/ into sys.path for non-editable installs

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(01-04): PydanticAI agent framework — 7 shells, registry, HTTP client, dispatcher

- AgentDeps dataclass (llm_client, session, http_client) — no global state (AGENT-06)
- AGENT_REGISTRY with register_agent/get_agent/list_agents
- 6 PydanticAI v1.63.0 agent shells: scout, research, matcher, writer, outreach, analyst
  - All use defer_model_check=True — model injected from config at runtime
  - All use ollama:llama3.1 default — colon separator per v1.x API
  - AGENT-05 enforced: no cross-agent imports (AST-verified)
- Orchestrator skeleton: routes tasks via AGENT_REGISTRY, 70 lines (AGENT-07: <250)
- Shared httpx.AsyncClient singleton with connection pooling (INFRA-18)
- AsyncTaskDispatcher over asyncio.Queue, configurable worker pool (INFRA-17)
- aiosmtplib + aioimaplib imported in outreach.py — Phase 3 stubs (INFRA-19/20)

PydanticAI discovery: v1.x uses output_type= (not result_type=), provider:model
format (colon not slash), and defer_model_check=True for runtime-configured agents.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* docs(01-04): plan complete — summary and PydanticAI v1.x discoveries

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor(01-04): agents as pipeline classes with tools and step sequences

AgentBase protocol now enforces:
  - STEPS: ClassVar[list[str]] — ordered pipeline steps
  - run_step(step, deps) -> StepResult — single step execution
  - run(deps, steps=None) -> AgentRunResult — full or partial pipeline

New types in base.py:
  - StepResult: result of one step (step, success, output, error)
  - AgentRunResult: full pipeline result with per-step list + failed_step property

All 6 agent shells refactored from bare PydanticAI Agent instances to classes:
  - Module-level PydanticAI Agent for @tool decoration
  - Per-agent @tool stubs showing Phase 2/3/4 function-call surface:
      scout: fetch_venue_page, extract_company_list
      research: search_web, fetch_page
      matcher: get_user_profile, extract_requirements
      writer: load_intel_brief, get_tone_guide
      outreach: classify_reply
      analyst: query_campaign_stats, compare_cohorts
  - run_step() match/case dispatch per step
  - run() chains STEPS, supports partial execution via steps= param

Orchestrator gains:
  - run() returns AgentRunResult (was dict)
  - run_step() for single-step dispatch (checkpointing, retry)
  - list_steps() to inspect an agent's pipeline

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(db): add LeadContact model with ContactType enum

Adds DB-02b — LeadContact table for typed per-lead contact details.
Keeps Lead.person_email unchanged (v1 outreach still reads from there).

ContactType enum: email | linkedin | github | twitter | website | phone
LeadContact fields: lead_id (FK+idx), contact_type, value, is_primary, created_at

is_primary marks the preferred contact for a given type on a lead.
Only `email` contacts are used for sending in v1; all types stored for
research context and future channel extensibility.

Alembic migration: a3f2e1d4b5c6 (chains from 149adcd94073 initial schema)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 69 out of 72 changed files in this pull request and generated 9 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

coder-ishan and others added 5 commits February 26, 2026 13:02
96 tests across 17 modules covering all Phase 1 subsystems:
crypto, config, DB engine/repos, LLM client/fallback, agent
pipeline contracts, orchestrator delegation, dispatcher, HTTP
client, and logging config. Zero real network/LLM calls.

Bug fix: await session.delete() in BaseRepository.delete() —
AsyncSession.delete() is a coroutine in current SQLAlchemy version.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
All 5 Phase 1 plans complete. 96 tests, 80.17% coverage.
Phase 1 ready for verification.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Phase 1 fully verified: 96 tests, 80.17% coverage, all 6 subsystems
built and confirmed. Advancing STATE.md to Phase 2.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Planning files are gitignored and should not be tracked.
Untracked with git rm --cached to keep local files intact.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
coder-ishan and others added 2 commits February 26, 2026 15:03
- alembic/env.py: add LeadContact to autogenerate imports (was silently skipped)
- config/crypto.py: replace local ConfigError stub with canonical ingot.agents.exceptions.ConfigError
- logging_config.py: close handlers before removing to prevent FD leaks on reconfigure
- http_client.py: reset _config_snapshot in close_http_client() for clean test isolation
- cli/setup.py: move configure_logging before wizard so errors are always captured to file
- cli/setup.py: add validation to _run_non_interactive (Anthropic/OpenAI keys, partial Gmail config)
- llm/fallback.py: handle PEP-604 X | None unions alongside typing.Union in xml_extract()
- tests: add test_cli_setup_noninteractive.py + http_client snapshot-reset test; coverage 86%

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@coder-ishan coder-ishan merged commit 1d02046 into main Feb 26, 2026
1 check failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants