feat(01-01): config system, Fernet crypto, setup wizard#1
Conversation
- pyproject.toml with hatchling build backend, all dependencies, entry point - src/ingot/__init__.py with __version__ = "0.1.0" - src/ingot/config/__init__.py package init - src/ingot/logging_config.py with structlog dual handlers (stderr WARNING+, rotating file DEBUG+ JSON) - src/ingot/cli/__init__.py with Typer app and setup subcommand - src/ingot/cli/setup.py stub (full implementation in Task 3)
- crypto.py: PBKDF2HMAC key derivation (600k iterations), get_fernet(), encrypt_secret(), decrypt_secret() - schema.py: AppConfig, AgentConfig, SmtpConfig, ImapConfig Pydantic v2 models with 7 default agents - manager.py: ConfigManager.load()/save() with atomic write (.tmp then rename), __encrypted__: prefix for secret fields
- Full questionary prompts with secret masking for API keys and passwords - Non-interactive mode reads ANTHROPIC_API_KEY, OPENAI_API_KEY, GMAIL_* env vars - --preset fully_free / best_quality sets all 7 agent model fields - Skips fields already configured in config.json - Rich summary table with secrets masked at end of setup - Creates ~/.outreach-agent/ directory structure (logs/, resume/, venues/) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR implements the foundational configuration system for INGOT, including Fernet-based encryption for secrets, a Pydantic-based configuration schema, and an interactive/non-interactive setup wizard CLI. This is Plan 01-01, providing the essential infrastructure that all subsequent modules will depend on for loading credentials, selecting LLM backends, and locating application data.
Changes:
- Added Fernet encryption system with PBKDF2 key derivation for protecting API keys and passwords at rest
- Created Pydantic v2 configuration schema with support for 7 core agents and email/LLM service credentials
- Implemented interactive and non-interactive setup wizard with preset configurations for different LLM deployment strategies
- Added structlog-based logging configuration with dual handlers (stderr and rotating file)
- Configured project build system with hatchling, dependencies, and test settings
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| pyproject.toml | Project metadata, dependencies, pytest configuration, and CLI entry point definition |
| src/ingot/init.py | Package version identifier |
| src/ingot/config/init.py | Config subsystem package marker |
| src/ingot/config/crypto.py | Fernet encryption/decryption with PBKDF2 key derivation from machine-specific key file |
| src/ingot/config/schema.py | Pydantic models for application configuration including agent, SMTP, and IMAP settings |
| src/ingot/config/manager.py | Configuration persistence with atomic writes and automatic secret encryption/decryption |
| src/ingot/logging_config.py | Structured logging setup with stderr console output and rotating JSON file logs |
| src/ingot/cli/init.py | Typer-based CLI application entry point |
| src/ingot/cli/setup.py | Interactive and non-interactive setup wizard with preset support and validation |
| .planning/phases/01-foundation-and-core-infrastructure/01-01-SUMMARY.md | Plan completion summary |
Comments suppressed due to low confidence (14)
src/ingot/cli/setup.py:217
- When password prompt returns None (user cancels), the code checks if password is falsy but uses the generic error message "Gmail App Password cannot be empty." This treats None (cancelled) the same as an empty string (user pressed enter). Consider distinguishing between cancellation and empty input, or at least catch the cancellation earlier to provide a clearer message.
password = questionary.password("Gmail App Password:").ask()
if not password:
_err.print("[red]Gmail App Password cannot be empty.[/red]")
raise typer.Exit(code=1)
src/ingot/config/manager.py:77
- If the config.json file contains invalid JSON, json.loads() will raise a JSONDecodeError which is not caught. This could happen if the file is corrupted or manually edited incorrectly. Consider catching JSONDecodeError and providing a helpful error message, or letting it bubble up with context about which file failed to parse.
raw = json.loads(self.config_path.read_text(encoding="utf-8"))
src/ingot/logging_config.py:17
- The comment references "~/.outreach-agent/" but should reference "log files" or describe what the base_dir parameter represents more generally. The hardcoded path in the docstring doesn't match the function's flexibility to accept any base_dir.
base_dir: Base directory for log files (e.g., ~/.outreach-agent/).
src/ingot/cli/setup.py:194
- The errors list is initialized but never populated with any error messages. The list is checked at the end but no errors are ever added to it. Either remove the unused error handling code or implement the validation that should add errors (e.g., when required environment variables are missing).
errors: list[str] = []
gmail_username = os.environ.get("GMAIL_USERNAME", "")
gmail_password = os.environ.get("GMAIL_APP_PASSWORD", "")
anthropic_key = os.environ.get("ANTHROPIC_API_KEY", "")
openai_key = os.environ.get("OPENAI_API_KEY", "")
if gmail_username:
cfg.smtp.username = gmail_username
cfg.imap.username = gmail_username
if gmail_password:
cfg.smtp.password = gmail_password
cfg.imap.password = gmail_password
if anthropic_key:
cfg.anthropic_api_key = anthropic_key
if openai_key:
cfg.openai_api_key = openai_key
# Apply preset or default to fully_free
effective_preset = preset or _PRESET_FULLY_FREE
_apply_preset(cfg, effective_preset)
# Ensure all 7 agents exist
for agent_name in _AGENT_NAMES:
if agent_name not in cfg.agents:
cfg.agents[agent_name] = AgentConfig()
if errors:
for error in errors:
_err.print(f"[red]{error}[/red]")
raise typer.Exit(code=1)
src/ingot/logging_config.py:31
- The date format uses the current date at the time configure_logging is called, but if the application runs across midnight, logs will continue writing to the old date's file. Consider whether this is the intended behavior or if log rotation should happen at midnight. The RotatingFileHandler rotates based on size, not time.
date_str = datetime.now().strftime("%Y-%m-%d")
log_file = log_dir / f"run-{date_str}.log"
src/ingot/config/crypto.py:15
- The stat module is imported but never used. The chmod call on line 59 uses the literal 0o600 instead of stat module constants. Either remove the unused import or use stat.S_IRUSR | stat.S_IWUSR for better clarity.
import stat
src/ingot/cli/init.py:21
- The import of setup_app is placed after code execution (after the app callback function) to avoid circular imports, indicated by the noqa: E402 comment. While this works, consider restructuring the module to avoid this pattern - perhaps by moving the command registration to a separate function or using lazy imports. The current structure makes the code flow less intuitive.
from ingot.cli.setup import setup_app # noqa: E402
src/ingot/cli/setup.py:66
- Secrets are displayed in the summary table with partial masking via the _mask function. However, showing even the first 4 characters of API keys or passwords could aid in brute-force attacks if an attacker can see the output. Consider either masking more characters or only showing them for debugging purposes with a flag. This is especially important for the summary output which might be logged or shared in screenshots.
def _mask(value: str, show_chars: int = 4) -> str:
"""Return a masked version of a secret for display."""
if not value:
return "(not set)"
if len(value) <= show_chars:
return "*" * len(value)
return value[:show_chars] + "*" * (len(value) - show_chars)
src/ingot/cli/setup.py:131
- The traceback import is unused - it's imported but never called. Either use traceback.print_exc() or remove the import.
import traceback
src/ingot/cli/setup.py:132
- The import statement appears after code execution due to the late import inside the exception handler. This violates PEP 8 conventions which recommend imports at the top of the file. Move the logging and traceback imports to the top of the file with the other imports.
import traceback
import logging
src/ingot/cli/setup.py:143
- The verbose parameter is accepted but never used in the setup wizard. It's passed through _run_setup but not utilized to control output verbosity or logging level. Either implement verbose output levels or remove the unused parameter.
verbose: int = typer.Option(0, "-v", count=True, max=2),
) -> None:
"""Run the INGOT setup wizard to configure credentials and LLM backends."""
try:
_run_setup(non_interactive=non_interactive, preset=preset, verbose=verbose)
except KeyboardInterrupt:
_err.print("\n[yellow]Setup cancelled.[/yellow]")
raise typer.Exit(code=1)
except typer.Exit:
raise
except Exception as exc:
log_path = Path.home() / ".outreach-agent" / "logs"
_err.print(f"[red][Setup] Something went wrong. Full error logged to {log_path}[/red]")
# Log the full traceback
import traceback
import logging
logging.getLogger("ingot.cli.setup").error(
"Setup wizard failed", exc_info=True
)
raise typer.Exit(code=1) from exc
def _run_setup(
*,
non_interactive: bool,
preset: str | None,
verbose: int,
pyproject.toml:16
- The version constraint ">=44" for cryptography is unusual and may be incorrect. The cryptography library typically uses semantic versioning (e.g., 44.0.0). This constraint would match any version >= 44.0.0, but versions typically look like "44.0.0" not "44". Consider using a more standard constraint like ">=44.0.0" or ">=44.0" to be more explicit.
"cryptography>=44",
src/ingot/config/manager.py:96
- The atomic write uses with_suffix(".tmp") which replaces the file extension. For "config.json", this produces "config.tmp" which is correct. However, if config_path ever doesn't have a suffix (e.g., "config"), this would produce "config.tmp" which is still fine. Consider using a more explicit temp file pattern like config_path.with_name(config_path.name + ".tmp") or documenting why with_suffix is sufficient here.
tmp_path = self.config_path.with_suffix(".tmp")
src/ingot/config/schema.py:72
- The default factory lambda creates a new list each time, which is correct for mutable defaults. However, the list could be defined as a module-level constant to avoid recreating it. Consider: _DEFAULT_LLM_FALLBACK = ["claude", "openai", "ollama"] and then use default_factory=lambda: _DEFAULT_LLM_FALLBACK.copy()
llm_fallback_chain: list[str] = Field(
default_factory=lambda: ["claude", "openai", "ollama"]
)
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- crypto.py: eliminate TOCTOU race on key file creation by using os.open with O_CREAT|O_EXCL|0o600 instead of write_bytes + chmod; also removes unused `stat` import - setup.py: loop on API key prompts when the required model provider needs a key — prevents silently storing an empty string that causes runtime failures - setup.py: wire configure_logging into _run_setup after ensure_dirs so structlog handlers are active for the full wizard lifecycle; import moved to module level Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
Addressed the three valid findings from this review (commit 9162af0): Fixed:
Disputed (code already handles these correctly):
|
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Rename CLI entry point from `job-hunter` to `ingot` (pyproject.toml and Typer app name) - Replace all `~/.outreach-agent/` references with `~/.ingot/` across crypto.py, manager.py, schema.py, logging_config.py, and setup.py Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…#2) * feat(01-02): async SQLite engine with WAL mode and all 11 SQLModel models - engine.py: create_async_engine + WAL/synchronous/cache/fk PRAGMAs via sync_engine event listener - models.py: UserProfile, Lead, IntelBrief, Match, Email, FollowUp, Campaign, AgentLog, Venue, OutreachMetric, UnsubscribedEmail - JSON columns for list fields (skills, experience, education, signals, talking_points) - str-backed enums for Lead/Email/FollowUp/Campaign status (SQLite has no native enum) - repositories/base.py: BaseRepository[T] with add/get/list/delete using AsyncSession Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(01-02): Alembic async migration setup with initial schema - alembic.ini: standard config with sqlite+aiosqlite URL - alembic/env.py: async online migration runner; explicit model imports before target_metadata to prevent empty autogenerate - alembic/script.py.mako: template with sqlmodel import included - alembic/versions/149adcd94073_initial_schema.py: autogenerated CREATE TABLE for all 11 models Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * docs(01-02): plan complete — summary Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(01-02): address copilot review findings - engine.py: use .as_posix() for cross-platform SQLite URL safety - repositories/base.py: remove erroneous await from session.delete() - models.py: add nullable=False to all JSON list/dict columns - alembic/env.py: inject src/ into sys.path for non-editable installs Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> ---------
- STATE.md: reflects 01-04 complete, Wave 4 (01-05) next - Phase 2 plan files: 02-01 through 02-07 + CONTEXT.md - outreach-agent-plan.md: original feature backlog reference Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… hierarchy (#3) * feat(01-03): LLMClient, XML fallback, and typed exception hierarchy - agents/exceptions.py: IngotError → LLMError, LLMValidationError, DBError, ConfigError, ValidationError, AgentError - llm/fallback.py: xml_extract() — scalar and list fields from XML tags, raises LLMValidationError on failure - llm/schemas.py: LLMMessage, LLMRequest, LLMResponse Pydantic envelopes - llm/client.py: LLMClient with tool-call → JSON → XML fallback priority, tenacity 3-attempt exponential backoff - No direct anthropic/openai imports anywhere — all routing via litellm.acompletion Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * docs(01-03): plan complete — summary Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(01-03): address copilot review findings Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(01-04): PydanticAI agent framework — 7 shells, registry, HTTP client, dispatcher (#4) * feat(01-02): async SQLite engine with WAL mode and all 11 SQLModel models - engine.py: create_async_engine + WAL/synchronous/cache/fk PRAGMAs via sync_engine event listener - models.py: UserProfile, Lead, IntelBrief, Match, Email, FollowUp, Campaign, AgentLog, Venue, OutreachMetric, UnsubscribedEmail - JSON columns for list fields (skills, experience, education, signals, talking_points) - str-backed enums for Lead/Email/FollowUp/Campaign status (SQLite has no native enum) - repositories/base.py: BaseRepository[T] with add/get/list/delete using AsyncSession Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(01-02): Alembic async migration setup with initial schema - alembic.ini: standard config with sqlite+aiosqlite URL - alembic/env.py: async online migration runner; explicit model imports before target_metadata to prevent empty autogenerate - alembic/script.py.mako: template with sqlmodel import included - alembic/versions/149adcd94073_initial_schema.py: autogenerated CREATE TABLE for all 11 models Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * docs(01-02): plan complete — summary Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(01-02): address copilot review findings - engine.py: use .as_posix() for cross-platform SQLite URL safety - repositories/base.py: remove erroneous await from session.delete() - models.py: add nullable=False to all JSON list/dict columns - alembic/env.py: inject src/ into sys.path for non-editable installs Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(01-04): PydanticAI agent framework — 7 shells, registry, HTTP client, dispatcher - AgentDeps dataclass (llm_client, session, http_client) — no global state (AGENT-06) - AGENT_REGISTRY with register_agent/get_agent/list_agents - 6 PydanticAI v1.63.0 agent shells: scout, research, matcher, writer, outreach, analyst - All use defer_model_check=True — model injected from config at runtime - All use ollama:llama3.1 default — colon separator per v1.x API - AGENT-05 enforced: no cross-agent imports (AST-verified) - Orchestrator skeleton: routes tasks via AGENT_REGISTRY, 70 lines (AGENT-07: <250) - Shared httpx.AsyncClient singleton with connection pooling (INFRA-18) - AsyncTaskDispatcher over asyncio.Queue, configurable worker pool (INFRA-17) - aiosmtplib + aioimaplib imported in outreach.py — Phase 3 stubs (INFRA-19/20) PydanticAI discovery: v1.x uses output_type= (not result_type=), provider:model format (colon not slash), and defer_model_check=True for runtime-configured agents. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * docs(01-04): plan complete — summary and PydanticAI v1.x discoveries Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * refactor(01-04): agents as pipeline classes with tools and step sequences AgentBase protocol now enforces: - STEPS: ClassVar[list[str]] — ordered pipeline steps - run_step(step, deps) -> StepResult — single step execution - run(deps, steps=None) -> AgentRunResult — full or partial pipeline New types in base.py: - StepResult: result of one step (step, success, output, error) - AgentRunResult: full pipeline result with per-step list + failed_step property All 6 agent shells refactored from bare PydanticAI Agent instances to classes: - Module-level PydanticAI Agent for @tool decoration - Per-agent @tool stubs showing Phase 2/3/4 function-call surface: scout: fetch_venue_page, extract_company_list research: search_web, fetch_page matcher: get_user_profile, extract_requirements writer: load_intel_brief, get_tone_guide outreach: classify_reply analyst: query_campaign_stats, compare_cohorts - run_step() match/case dispatch per step - run() chains STEPS, supports partial execution via steps= param Orchestrator gains: - run() returns AgentRunResult (was dict) - run_step() for single-step dispatch (checkpointing, retry) - list_steps() to inspect an agent's pipeline Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(db): add LeadContact model with ContactType enum Adds DB-02b — LeadContact table for typed per-lead contact details. Keeps Lead.person_email unchanged (v1 outreach still reads from there). ContactType enum: email | linkedin | github | twitter | website | phone LeadContact fields: lead_id (FK+idx), contact_type, value, is_primary, created_at is_primary marks the preferred contact for a given type on a lead. Only `email` contacts are used for sending in v1; all types stored for research context and future channel extensibility. Alembic migration: a3f2e1d4b5c6 (chains from 149adcd94073 initial schema) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> ---------
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 69 out of 72 changed files in this pull request and generated 9 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
96 tests across 17 modules covering all Phase 1 subsystems: crypto, config, DB engine/repos, LLM client/fallback, agent pipeline contracts, orchestrator delegation, dispatcher, HTTP client, and logging config. Zero real network/LLM calls. Bug fix: await session.delete() in BaseRepository.delete() — AsyncSession.delete() is a coroutine in current SQLAlchemy version. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
All 5 Phase 1 plans complete. 96 tests, 80.17% coverage. Phase 1 ready for verification. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Phase 1 fully verified: 96 tests, 80.17% coverage, all 6 subsystems built and confirmed. Advancing STATE.md to Phase 2. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Planning files are gitignored and should not be tracked. Untracked with git rm --cached to keep local files intact. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- alembic/env.py: add LeadContact to autogenerate imports (was silently skipped) - config/crypto.py: replace local ConfigError stub with canonical ingot.agents.exceptions.ConfigError - logging_config.py: close handlers before removing to prevent FD leaks on reconfigure - http_client.py: reset _config_snapshot in close_http_client() for clean test isolation - cli/setup.py: move configure_logging before wizard so errors are always captured to file - cli/setup.py: add validation to _run_non_interactive (Anthropic/OpenAI keys, partial Gmail config) - llm/fallback.py: handle PEP-604 X | None unions alongside typing.Union in xml_extract() - tests: add test_cli_setup_noninteractive.py + http_client snapshot-reset test; coverage 86% Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
What Was Built
Fernet-encrypted config system, setup wizard CLI, and package scaffold — the shared config layer every other module will import.
Key Files Created
pyproject.toml— hatchling build, all deps,asyncio_mode = "auto",job-hunterentry pointsrc/ingot/__init__.py—__version__ = "0.1.0"src/ingot/config/crypto.py— PBKDF2HMAC key derivation (600k iterations),get_fernet(),encrypt_secret(),decrypt_secret()src/ingot/config/schema.py—AppConfig,AgentConfig,SmtpConfig,ImapConfig(Pydantic v2)src/ingot/config/manager.py—ConfigManager.load()/save()with atomic write,__encrypted__:prefix for secretssrc/ingot/cli/setup.py— full setup wizard: interactive (questionary) + non-interactive (env vars),--preset fully_free/best_quality, skips existing values, Rich summary tablesrc/ingot/cli/__init__.py— Typer app withsetupsubcommandsrc/ingot/logging_config.py— structlog dual handlers (stderr WARNING+, rotating file DEBUG+ JSON)Verification
python3 -c "import ingot; print(ingot.__version__)"→0.1.0✓decrypt_secret(encrypt_secret("hello")) == "hello"✓ConfigManager,AppConfig,setup_app✓Commits
a53d0a8feat(01-01): project scaffold, pyproject.toml, and package structureb4b1f02feat(01-01): Fernet crypto module and ConfigManagerfa2d6b7feat(01-01): setup wizard CLI with interactive + non-interactive modes🤖 Generated with Claude Code