docs: Add comprehensive messaging and memory lifecycle documentation

[elmorem]

Summary

This PR adds two comprehensive technical deep dive documentation files that explain the core engines of ContextIQ: the messaging system and the complete memory lifecycle from sessions to consolidated memories.

New Documentation Files

1. docs/MESSAGING.md (3,681 lines, 101KB)

Complete technical guide to the RabbitMQ messaging system including:

Architecture & Configuration:

• MessagingSettings with all environment variables
• RabbitMQClient connection management with auto-reconnection
• Queue and exchange declarations
• Dead letter queue configuration

Core Components:

• RabbitMQClient: Connection pooling, exchange/queue declaration, health checks
• MessagePublisher: Publishing with priority, persistence, correlation IDs, RPC pattern
• MessageConsumer: Consuming with prefetch, auto-ack, error handling, graceful shutdown
• Queue Definitions: EXTRACTION_REQUESTS, CONSOLIDATION_REQUESTS, events, DLQ

Message Flows:

• Session event → Memory extraction request
• Memory extraction → Consolidation trigger
• Event publishing and consumption

Queue Patterns:

• Work queue pattern for load balancing
• Event queue pattern for pub/sub
• Dead letter queue for failed messages

Reliability & Performance:

• Publisher confirms for guaranteed delivery
• Message persistence and durability
• Prefetch control for flow management
• Connection recovery and retry logic
• Batch operations and connection pooling
• Production clustering and HA setup

Documentation Quality:

• 100+ code examples covering all use cases
• ASCII diagrams of message flows
• Complete environment variable reference
• Production deployment patterns
• Monitoring and health check setup
• Comprehensive troubleshooting guide

2. docs/MEMORY_LIFECYCLE.md (3,848 lines, 122KB)

Complete technical guide to the memory lifecycle - how ContextIQ transforms sessions into memories:

The Six Phases:

1. Session Events Collection - Conversation capture with structured events
2. Memory Extraction - LLM-powered extraction with Anthropic Claude
3. Embedding Generation - OpenAI text-embedding-3-small (1536 dimensions)
4. Memory Storage - PostgreSQL database + Qdrant vector store
5. Memory Consolidation - Similarity detection and merging with cosine similarity
6. Memory Retrieval - Semantic search and relevance ranking

Memory Extraction Pipeline:

• MemoryGenerationWorker: Background worker consuming from RabbitMQ
• MemoryGenerationProcessor: 4-step pipeline orchestration
  1. Fetch events from Sessions Service (HTTP)
  2. Extract memories using ExtractionEngine (LLM)
  3. Generate embeddings using EmbeddingService
  4. Save to Memory Service with embeddings
• ExtractionEngine: LLM-powered extraction with structured response schema
  • Anthropic Claude 3.5 Sonnet
  • 8 memory categories: preference, fact, goal, habit, relationship, professional, location, temporal
  • Confidence scoring (0.0-1.0)
  • Memory validation
• LLM Integration: Structured extraction with response schema, few-shot examples

Memory Consolidation Pipeline:

• ConsolidationWorker: Background worker for deduplication
• ConsolidationProcessor: 5-step consolidation orchestration
  1. Fetch memories from Memory Service by scope
  2. Convert to consolidation format
  3. Run ConsolidationEngine (similarity detection)
  4. Generate embeddings for merged memories
  5. Save consolidated memories
• ConsolidationEngine: Similarity-based merging
  • Cosine similarity calculation from embeddings
  • Similarity threshold: 0.85 (configurable)
  • Merge strategies: highest_confidence, most_recent, longest
  • Conflict detection for contradictory memories (0.7-0.85 similarity)
  • Confidence boost for merged memories (+0.1)

Technical Implementation Details:

• Complete data models: MemoryGenerationRequest, ExtractionResult, Memory, ConsolidationRequest, MergedMemory
• Configuration reference for all tunable parameters
• Memory quality: confidence scores, importance scoring, category classification
• Performance considerations: batching, async operations, worker concurrency
• Database optimization: indexing, query patterns, pagination

Documentation Quality:

• End-to-end ASCII flow diagrams
• Complete data model examples
• 100+ comprehensive code examples
• Step-by-step extraction and consolidation walkthroughs
• Monitoring and debugging guides
• Production best practices (triggers, error handling, data quality)
• Troubleshooting guide with solutions for common issues

3. docs/README.md (updated)

• Added "Technical Deep Dives" section with:

  • Messaging System Guide
  • Memory Lifecycle Guide
  • Embeddings Guide
  • Vector Search Guide

• Added new use cases:

  • "...understand how messaging works" with navigation to messaging docs
  • "...understand how memories are created" with navigation to memory lifecycle docs

Implementation Coverage

Messaging System Documentation

Based on actual implementation from:

• shared/messaging/config.py - MessagingSettings configuration
• shared/messaging/rabbitmq_client.py - RabbitMQClient implementation
• shared/messaging/publisher.py - MessagePublisher
• shared/messaging/consumer.py - MessageConsumer
• shared/messaging/queues.py - Queue definitions
• workers/memory_generation/worker.py - Extraction worker
• workers/consolidation/worker.py - Consolidation worker

Memory Lifecycle Documentation

Based on actual implementation from:

• workers/memory_generation/worker.py - MemoryGenerationWorker
• workers/memory_generation/processor.py - Extraction pipeline
• shared/extraction/engine.py - ExtractionEngine with LLM
• shared/extraction/prompts.py - LLM prompts and schemas
• workers/consolidation/worker.py - ConsolidationWorker
• workers/consolidation/processor.py - Consolidation pipeline
• shared/consolidation/engine.py - Similarity and merging logic

Documentation Statistics

• Total Lines: 7,529 lines of technical documentation
• Code Examples: 200+ comprehensive examples
• Diagrams: Multiple ASCII diagrams for visual understanding
• File Size: 223KB total (101KB + 122KB)

Key Benefits

1. For Developers: Complete understanding of messaging infrastructure and memory processing
2. For Operators: Production deployment patterns, monitoring, and troubleshooting
3. For Architects: System design decisions, trade-offs, and optimization strategies
4. For Contributors: Clear documentation of implementation details

Test Plan

• Documentation files created and properly formatted
• All markdown linting checks passed
• Cross-references verified and working
• Code examples validated against actual implementation
• Technical accuracy verified by reviewing source code
• docs/README.md updated with new links and use cases

Related Documentation

This PR complements PR #40 (Embeddings and Vector Search) and PR #39 (Comprehensive Documentation) to complete the technical documentation suite.

Related docs:

• docs/README.md - Documentation index (updated)
• docs/EMBEDDINGS.md - Embeddings guide (PR docs: Add comprehensive embeddings and vector search documentation #40)
• docs/VECTOR_SEARCH.md - Vector search guide (PR docs: Add comprehensive embeddings and vector search documentation #40)
• docs/ARCHITECTURE.md - System architecture (PR docs: Add comprehensive documentation for all completed features #39)
• docs/API_USAGE.md - API usage guide (PR docs: Add comprehensive documentation for all completed features #39)

🤖 Generated with Claude Code