AI Content Factory

Production-oriented multi-agent RAG pipeline that discovers technology trends, researches the web, embeds knowledge into Qdrant, plans and writes articles with Qwen (Ollama), and stores results in PostgreSQL. All agents communicate asynchronously through RabbitMQ events.

Architecture

flowchart LR
  subgraph Agents
    TA[trend-agent]
    RA[research-agent]
    EA[embedding-agent]
    PA[planner-agent]
    WA[writer-agent]
    SA[storage-agent]
  end

  subgraph Infra
    RMQ[(RabbitMQ)]
    QD[(Qdrant)]
    PG[(PostgreSQL)]
    OL[Ollama Qwen3]
  end

  TA -->|TrendFound| RMQ
  RMQ --> RA
  RA -->|ResearchCompleted| RMQ
  RMQ --> EA
  EA --> QD
  EA -->|EmbeddingCompleted| RMQ
  RMQ --> PA
  PA --> QD
  PA --> OL
  PA -->|PlanCreated| RMQ
  RMQ --> WA
  WA --> QD
  WA --> OL
  WA -->|ArticleGenerated| RMQ
  RMQ --> SA
  SA --> PG

Loading

sequenceDiagram
  participant T as trend-agent
  participant R as research-agent
  participant E as embedding-agent
  participant P as planner-agent
  participant W as writer-agent
  participant S as storage-agent
  participant Q as Qdrant
  participant O as Ollama

  T->>R: TrendFound
  R->>E: ResearchCompleted
  E->>Q: upsert chunks
  E->>P: EmbeddingCompleted
  P->>Q: RAG search
  P->>O: outline JSON
  P->>W: PlanCreated
  W->>Q: RAG search
  W->>O: markdown article
  W->>S: ArticleGenerated

Loading

Event Pipeline

Event	Routing Key	Queue	Producer	Consumer
TrendFound	trend.found	trend-found	trend-agent	research-agent
ResearchCompleted	research.completed	research-completed	research-agent	embedding-agent
EmbeddingCompleted	embedding.completed	embedding-completed	embedding-agent	planner-agent
PlanCreated	plan.created	plan-created	planner-agent	writer-agent
ArticleGenerated	article.generated	article-generated	writer-agent	storage-agent

Exchange: content-pipeline (topic). Each queue has a dead-letter queue (<queue>.dlq).

Quick Start

Prerequisites

• Docker and Docker Compose
• 16 GB+ RAM recommended (embedding model + Ollama)

Run the stack

cd ai-content-factory
docker compose up --build -d

Services:

Service	Port	Purpose
RabbitMQ	5672, 15672	Event bus (management UI)
Qdrant	6333	Vector store
PostgreSQL	5432	Article persistence
Ollama	11434	Local LLM (qwen3:8b)

Verify

docker compose ps
docker compose logs -f trend-agent

Query stored articles:

docker exec -it acf-postgres psql -U content -d content_factory -c "SELECT title, slug, created_at FROM articles ORDER BY created_at DESC LIMIT 5;"

Configuration

• topics.json — seed technology topics for trend discovery
• .env — copy from .env.example
• shared/config/pipeline.yaml — documented pipeline topology and defaults
• shared/config/settings.py — Pydantic settings (env vars override YAML defaults)

Key environment variables

Variable	Default	Description
OLLAMA_HOST	http://ollama:11434	Ollama API endpoint
OLLAMA_MODEL	qwen3:8b	Model for planner/writer
EMBEDDING_MODEL	BAAI/bge-m3	HuggingFace embeddings
QDRANT_COLLECTION	articles	Qdrant collection name
RABBITMQ_HOST	rabbitmq	Message broker host

Agent Responsibilities

trend-agent

• Reads topics.json
• Uses GoogleTrendsProvider (pluggable TrendProvider)
• Publishes TrendFound every 7 days
• Deduplicates trends and skips topics processed in the last 30 days

research-agent

• Consumes TrendFound
• Builds search queries (tutorial, architecture, deployment, observability)
• Collects 5–20 sources, extracts clean text (trafilatura/newspaper3k)
• Stores raw JSON locally, publishes ResearchCompleted

embedding-agent

• Chunks text (512 tokens, 50 overlap)
• Embeds with BAAI/bge-m3 (1024-dim, cosine)
• Upserts into Qdrant collection articles
• Publishes EmbeddingCompleted

planner-agent

• RAG: top 20 chunks from Qdrant
• Prompts Qwen for JSON outline (validated, retried)
• Publishes PlanCreated

writer-agent

• RAG + outline → markdown article (1500–3000 words)
• Grounded in retrieved context with citations
• Publishes ArticleGenerated with SEO metadata

storage-agent

• Persists articles and source URLs to PostgreSQL

Qdrant Collection

• Name: articles
• Vector size: 1024
• Distance: COSINE
• Payload: topic, url, title, chunk_text, created_at

PostgreSQL Schema

articles: id, topic, title, slug, content, meta_description, created_at

sources: id, article_id, url, title

Development

Unit tests

pip install -r requirements-dev.txt
pip install -r shared/requirements.txt
pytest -m "not integration"

Integration tests (stack running)

export RABBITMQ_HOST=localhost
export QDRANT_HOST=localhost
export OLLAMA_HOST=http://localhost:11434
pytest -m integration

RabbitMQ Topology

graph TD
  EX[content-pipeline topic exchange]
  EX --> Q1[trend-found]
  EX --> Q2[research-completed]
  EX --> Q3[embedding-completed]
  EX --> Q4[plan-created]
  EX --> Q5[article-generated]
  Q1 --> D1[trend-found.dlq]
  Q2 --> D2[research-completed.dlq]
  Q3 --> D3[embedding-completed.dlq]
  Q4 --> D4[plan-created.dlq]
  Q5 --> D5[article-generated.dlq]

Loading

Extending the Pipeline

New agents (SEO, translator, newsletter, LinkedIn, podcast, video) can be added by:

1. Implementing AgentPlugin in shared/plugins/agent_plugin.py
2. Defining a new event contract in shared/contracts/events.py
3. Adding routing key + queue in shared/messaging/topology.py
4. Creating a new *-agent service with its own Dockerfile
5. Wiring the service in docker-compose.yml

No direct service-to-service calls — only events on RabbitMQ.

Project Layout

ai-content-factory/
├── docker-compose.yml
├── topics.json
├── shared/               # contracts, messaging, config, services
├── trend-agent/
├── research-agent/
├── embedding-agent/
├── planner-agent/
├── writer-agent/
└── storage-agent/

Troubleshooting

• Ollama model missing: ensure ollama-init completed (docker compose logs ollama-init)
• Embedding OOM: increase Docker memory or run embedding-agent alone first
• No articles in Postgres: check upstream agents and RabbitMQ queue depths in management UI (http://localhost:15672)