Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
218 lines (180 loc) · 8.43 KB

ARCHITECTURE.md

File metadata and controls

218 lines (180 loc) · 8.43 KB
Raw

Architecture

hermes-cortex is a local retrieval layer for Hermes Agent. The Markdown vault is the source of truth; generated artifacts are rebuildable caches.

Data flow

Markdown vault (*.md)
  │
  ▼
indexer.py
  ├─ parses YAML frontmatter
  ├─ chunks notes by headings
  ├─ extracts wikilinks
  └─ writes ~/.hermes/cortex/chunks.jsonl
  │
  ├──────────────┐
  ▼              ▼
embedder.py      graph_index.py
  │              │
  ▼              ▼
Chroma           graph_nodes.jsonl / graph_edges.jsonl / graph_broken.jsonl
  │              │
  └──────┬───────┘
         ▼
search.py
  ├─ BM25 lexical ranking
  ├─ vector ranking
  ├─ wikilink graph expansion
  ├─ metadata filters
  └─ RRF fusion + recency/importance boosts
         │
         ▼
context.py
  └─ token-budgeted Markdown context with citations
         │
         ▼
Hermes tools
  ├─ vault_search
  ├─ vault_read_note
  └─ vault_build_context

Runtime integration

Hermes loads Cortex as a standalone directory plugin:

~/.hermes/plugins/cortex/
├── plugin.yaml
├── __init__.py
├── plugin_runtime.py
└── cortex/

plugin.yaml declares the plugin. Hermes imports root __init__.py, which calls register(ctx) and delegates tool, hook, and CLI registration to plugin_runtime.py.

The active runtime source is the plugin checkout itself. Keep that checkout in sync with the development repo via git pull --ff-only origin main inside ~/.hermes/plugins/cortex/; otherwise the hermes cortex ... command surface can lag behind python -m cortex.cli .... Hook code/config is also process-local: already-started TUI sessions, Hermes gateway processes, and Kanban workers may retain the old hook behavior until a new session/process starts or an operator-approved restart picks up the updated plugin checkout/config. The runtime smoke check is:

scripts/smoke-runtime-cortex-cli.sh

Knowledge sources

Cortex distinguishes between indexed vault knowledge and Hermes runtime memory.

Source Indexed? Purpose
Markdown vault Yes Durable facts, decisions, projects, runbooks
~/.hermes/memories/MEMORY.md No Compact runtime facts
~/.hermes/memories/USER.md No User preferences/profile
~/.hermes/SOUL.md No Agent persona/rules

The vault is embedded and searched. vault.path in the active Cortex config is the only durable runtime source of truth for that Vault. WIKI_PATH is accepted only by init as a defaulting input when no explicit --vault path or existing config wins; runtime commands, hooks, indexing, search, and lifecycle do not use WIKI_PATH, wiki.path, or a blind wiki-directory fallback.

Hermes memory files may be included as static bootstrap context via hooks.bootstrap_context.include_static_files (preferred) or the legacy context_builder.include_hermes_memory switch. They are read as context only; they are not copied into the vault or vector store.

Runtime hook context is split semantically:

  • skill_context — each-turn runtime rules and skill bootstrap
  • bootstrap_context — first-turn static context, including deterministic include_static_files
  • recent_context — disabled by default but SessionDB-capable deterministic recent-topic metadata context
  • dynamic_context — gated/off-by-default user-message Vault context

New semantic blocks take precedence over the legacy hooks.context_injection block. Legacy configs still parse for compatibility, but they are treated as a fallback path, not the primary model. hermes cortex status and hermes cortex config show render this as hook lifecycle rows with explicit enabled, effective, timing, origin, source/payload/target, and skipped reason. A legacy-only config reports legacy_context_injection as legacy-active; a mixed semantic+legacy config reports it as legacy-ignored; a config with no legacy block reports it as legacy-absent. when is validated per semantic block so impossible lifecycle combinations fail during config load rather than quietly drifting at runtime.

Vault seed and curated-source boundaries

Fresh init seeds Cortex canonical folders plus llm-wiki-compatible root material inside the same configured Vault root:

SCHEMA.md
index.md
log.md
raw/articles/
raw/papers/
raw/transcripts/
raw/assets/
raw/README.md

SCHEMA.md, index.md, and log.md are operational/orientation files, not curated answer chunks by default under the normal non-empty include_folders configuration. raw/ stores immutable source/provenance material. The default curated corpus includes 10_facts, 20_decisions, 30_projects, 40_runbooks, 50_people, and 60_maps, and excludes 00_inbox, 80_templates, and raw. Operators can diagnose root/raw drift with the read-only hermes cortex wiki-health command.

log.md is append-only operator/lifecycle history. Maintenance/nightly may append compact events when the file already exists; missing or non-file log.md is a wiki-health error and a lifecycle skip condition rather than runtime-created state.

Nightly/session promotion

NightlyPromotion uses Hermes SessionDB as the primary session source: it reads ~/.hermes/state.db first, falls back to legacy JSON/JSONL session files only when the database is missing, unreadable, schema-incompatible, or otherwise fails the documented selection rules, and ignores request_dump_*.json even if the file glob would match them.

Nightly report payloads should make the source choice visible: which backend was primary, how many sessions each backend saw, whether fallback was used, the fallback reason, the ignored request-dump count, and the lookback cutoff/ timezone. No empty-file-glob story is acceptable if state.db actually contains recent sessions.

Main components

Component Role
cortex.config Loads and validates Cortex config
cortex.indexer Parses Markdown and writes chunks.jsonl
cortex.embedder Computes embeddings and stores them in Chroma
cortex.search Hybrid search and ranked result fusion
cortex.graph_index Builds graph artifacts from notes and wikilinks
cortex.graph_diagnostics Reports broken links, orphans, hubs, stale notes
cortex.context Builds cited Markdown context under a token budget
cortex.plugin Pure Python tool API used by Hermes integration
plugin_runtime.py Hermes plugin registration layer
cortex.cli Standalone and Hermes-routed CLI commands

Search model

Search combines three channels:

  1. BM25 — exact/lexical matches over normalized chunk text
  2. Vector — semantic matches from Chroma embeddings
  3. Graph — nearby chunks via wikilink expansion

Ranks are fused with weighted reciprocal rank fusion (RRF). Optional recency and importance boosts are applied after fusion. Missing metadata stays neutral; it is not treated as medium importance by accident.

CLI guardrails

Two operator-facing commands are intentionally read-only or diagnostic-only:

  • hermes cortex validate-frontmatter
    • validates YAML frontmatter and vault metadata
    • exits 1 on validation errors, or on warnings only when --strict is set
    • --json emits a stable report with schema_version, counts, and per-file issues
    • --path scopes validation to explicit notes or directories inside the vault
  • hermes cortex wiki-health
    • read-only Cortex/llm-wiki contract diagnostics for configured vault.path
    • reports missing root/raw material, missing canonical folders, curated-source drift, raw frontmatter/hash issues, and missing/non-file log.md
    • never creates or repairs Vault files; --strict makes warnings non-zero
  • hermes cortex search-eval
    • runs fixed real-vault ranking cases
    • --output writes the JSON report, --json prints the same payload
    • --baseline adds compare metadata and rank deltas; update baselines only after hermes cortex validate-frontmatter and --lint-vault-files pass and index/embed maintenance has run
    • report cases include final_score, rrf_score, channel ranks, boost fields, and baseline comparison fields when a baseline is provided

Rebuild policy

Generated artifacts are disposable:

hermes cortex index --force
hermes cortex embed --force
hermes cortex graph build --force

If search behavior is suspicious, rebuild before theorizing. Machines enjoy humbling us.