Skip to content

JoelJohnsonThomas/AnimeLoom

BranchesTags

Repository files navigation

AnimeLoom

Anime Character Consistency Engine — generate studio-quality anime video from a text story while keeping the same character face, identity, and style across every shot.

AnimeLoom orchestrates a multi-stage pipeline (story decomposition, identity-locked keyframes, motion synthesis, face restoration, post-processing) to turn a one-paragraph prompt into a smooth anime sequence on a single GPU. Built around the latest 2026 open-source video models with a face-lock pass that pastes the character's face from a reference image into every output frame.

What's New in v2

Stage v1 (older Colab path) v1.5 v2 (current)
Keyframes SDXL + LoRA SDXL + LoRA + IP-Adapter SDXL + img2img chaining unchanged from v1.5
Story decomposition Rule-based Gemini Flash Two-stage Gemini -> Claude refinement
Video model CogVideoX 1.5 Wan2.1-I2V-14B Wan2.2-I2V-A14B (MoE) + anime LoRA
Face consistency None GFPGAN per frame Wan2.2-Animate face lock (face pasted from keyframe at every frame)
Post-processing GFPGAN + Real-ESRGAN + two-pass temporal smoothing, every-2nd-frame GFPGAN unchanged from v1.5
Identity consistency ~70% ~85% ~95%+

The v2 path runs Wan2.2 in two passes per shot:

  1. Phase 3a — Wan2.2-I2V-A14B (with anime LoRA) generates a driving clip that captures motion only
  2. Phase 3b — Wan2.2-Animate-14B takes the SDXL keyframe as reference and the driving clip as motion source, producing a face-locked output where the character's face is literally pasted from the keyframe at every frame

This decoupled design comes from the Wan-Animate paper (arXiv 2509.14055) and is the single biggest available jump for "consistent anime faces" in open-source 2026.

Web Studio (new in v2)

A branded Vite + React + TypeScript web UI lives in frontend/ and ships alongside the API. It covers character management, script editing, live generation tracking, and result review — wired to the FastAPI backend with TanStack Query polling.

Stack: Vite 6 · React 18 · TypeScript · Tailwind v4 · TanStack Query · React Router · Zustand · Lucide icons.

Deploy modes:

  • Standalone dev — cd frontend && npm install && npm run dev opens :5173 and proxies /api to FastAPI on :8080.
  • Embedded — npm run build:embedded outputs to frontend/dist/; api/app.py auto-mounts it at /ui/ so the entire stack runs on one port.
  • RunPod — the notebook ships a Deploy Web Studio cell that installs Node, builds the frontend, starts FastAPI, and prints https://{pod_id}-8080.proxy.runpod.net/ui/.

Design tokens are wired through a single CSS source (frontend/src/styles/tokens.css), a global :focus-visible ring lands on every interactive element, and brand polish (gradient logo, pink-glow CTAs, sakura petal overlay, celebration spring) is baked in. The historical spec — token CSS, preview HTMLs, original JSX prototypes — is preserved in AnimeLoom Design System/.

See frontend/README.md for full setup, build modes, and a11y notes.

Pipeline

Story (text)
   |
   v
Phase 1: Decompose (Gemini story planning -> Claude cinematic refinement)
   |
   v
Phase 2: SDXL + LoRA + IP-Adapter -> identity-locked keyframes (img2img chaining,
                                     adaptive strength decay, dynamic anchor refresh,
                                     quality gate with drift detection)
   |
   v
Phase 3a: Wan2.2-I2V-A14B (+ anime LoRA) -> driving clips (motion source)
   |
   v
Phase 3b: Wan2.2-Animate-14B(reference=keyframe, driving=clip) -> face-locked frames
   |
   v
Phase 4: GFPGAN every-2nd-frame face restoration + two-pass temporal smoothing
   |
   v
Phase 5: RIFE temporal upscale (16fps -> 24fps) + Real-ESRGAN spatial upscale
   |
   v
Phase 6: Cross-dissolve assembly -> final mp4

Requirements

  • Recommended GPU: NVIDIA RTX A6000 (48GB VRAM). Each Wan2.2 14B variant peaks around 42-46GB with model CPU offload.
  • Minimum GPU: any 24GB+ card with sequential offload (slower; 480x640 max resolution).
  • Python 3.10+, PyTorch 2.5.1 + CUDA 12.4, ffmpeg, Redis (optional, for Celery).
  • Node.js 20+ (only required if you want to build/run the Web Studio frontend).
  • API keys (optional but recommended): Gemini (free, 1500 req/day at aistudio.google.com/apikey) and Anthropic Claude.

Quick Start (RunPod A6000 — primary path)

  1. Spin up an A6000 (48GB) pod on RunPod with the PyTorch image
  2. Open Jupyter, then notebooks/AnimeLoom_RunPod.ipynb
  3. Run cells in order:
    • Cell 1 — installs pinned deps (torch 2.5.1+cu124, diffusers 0.36, ftfy, gfpgan, facexlib, etc.)
    • Cell 2 — downloads a character LoRA from HuggingFace (default: AnimeLoom/sakura-haruno; also available: AnimeLoom/denji, AnimeLoom/yuki-nagato)
    • Cell 2.5 — patches the story decomposer for two-stage Gemini->Claude refinement
    • Cell 3 — runs the full v2 pipeline (Phase 1 -> 6) and renders the final video
    • Deploy Web Studio — installs Node.js, builds the React frontend, starts FastAPI on :8080, and prints https://{pod_id}-8080.proxy.runpod.net/ui/. Expose HTTP port 8080 in your pod settings first, otherwise the proxy URL will 502.

Configure in Cell 3:

  • STORY_TEXT — your one-paragraph story
  • CHARACTER_NAME — must match the LoRA from Cell 2
  • GEMINI_API_KEY / ANTHROPIC_API_KEY — both optional; falls back gracefully

Quick Start (CLI / standalone)

git clone https://github.com/JoelJohnsonThomas/AnimeLoom.git
cd AnimeLoom
chmod +x setup.sh
./setup.sh
python main.py --text "A girl walks through a cherry blossom forest at sunset"
python main.py --script script.txt --quality high
python main.py --api      # FastAPI server (auto-serves /ui/ if frontend/dist/ exists)
python main.py --test     # smoke test
# Web Studio
cd frontend && npm install
npm run dev               # http://localhost:5173 (proxies /api -> :8080)
npm run build:embedded    # then start FastAPI; visit http://localhost:8080/ui/

Architecture

+------------------------------------------------------------+
|                       DirectorAgent                         |
|  +-----------+  +---------------+  +--------------------+   |
|  | Story     |->| WorkflowGraph |->| Shot Executor      |   |
|  | Decomposer|  | (DAG)         |  | + Checkpointing    |   |
|  +-----------+  +---------------+  +---------+----------+   |
+------------------------------------------------+------------+
            |              |               |              |
            v              v               v              v
    +---------------+ +-----------+ +-----------+ +--------------+
    | Character     | | Animator  | | Evaluator | | Asset        |
    | Agent         | | Agent     | | Agent     | | MemoryBank   |
    |               | |           | |           | |              |
    | * LoRA train  | | * Wan2.2  | | * Identity| | * LoRAs      |
    | * IP-Adapter  | | * Animate | | * Motion  | | * Embeddings |
    | * Consistency | | * RIFE    | | * Visual  | | * Scenes     |
    +---------------+ +-----------+ +-----------+ +--------------+

Project Structure

animeloom/
├── director/
│   ├── agent.py                   # main orchestrator (script parsing, shot execution)
│   ├── workflow.py                 # shot dependency DAG with topological ordering
│   └── memory_bank.py              # persistent character/scene/shot storage
├── agents/
│   ├── story/
│   │   └── decomposer.py           # two-stage Gemini -> Claude story decomposer
│   ├── character/
│   │   ├── trainer.py              # LoRA fine-tuning (PEFT, rank 16-32)
│   │   ├── lora_manager.py         # adapter load/unload
│   │   ├── ip_adapter.py           # IPAdapterConditioner (SDXL face-image conditioning)
│   │   └── consistency.py          # GroundingDINO + SAM + CLIP identity validation
│   ├── animator/
│   │   ├── wan_wrapper.py          # multi-backend video wrapper
│   │   ├── wan_animate.py          # Wan2.2-Animate-14B face-lock wrapper (NEW in v2)
│   │   ├── cogvideo_wrapper.py     # CogVideoX fallback
│   │   ├── pixverse.py             # PixVerse external fallback
│   │   └── controlnet.py           # OpenPose pose conditioning
│   ├── postprocess/
│   │   ├── upscaler.py             # RIFE temporal + Real-ESRGAN spatial
│   │   ├── face_restore.py         # GFPGAN/CodeFormer face restoration
│   │   ├── color_grade.py          # anime LUT grading
│   │   └── transitions.py          # cross-dissolve assembly
│   └── evaluator/
│       ├── character_score.py       # CLIP-based identity consistency
│       ├── motion_score.py          # optical flow motion fidelity
│       └── visual_score.py          # sharpness, colour, smoothness
├── api/
│   ├── app.py                       # FastAPI application
│   ├── routes/{characters,generation}.py
│   └── schemas/models.py
├── jobs/
│   ├── worker.py                    # Celery async worker
│   └── tasks/{training,generation}.py
├── cloud/
│   ├── colab_survival.py            # 4-min keep-alive + 5-min checkpointing
│   ├── kaggle_trainer.py            # Kaggle P100 trainer
│   └── gcp_setup.sh                 # GCP T4 VM provisioning
├── frontend/                        # Vite + React + TypeScript Web Studio (v2)
│   ├── src/
│   │   ├── routes/                  # Dashboard, Characters, Script, Generate, Results, Settings, Docs
│   │   ├── components/              # ui/ (Button, Card, ...) + brand/ (LogoMark, SakuraOverlay)
│   │   ├── lib/                     # api.ts, types.ts, hooks/, store.ts
│   │   └── styles/tokens.css        # single source of truth for design tokens
│   └── README.md
├── AnimeLoom Design System/         # historical spec — token CSS, preview HTMLs, JSX prototypes
├── notebooks/
│   └── AnimeLoom_RunPod.ipynb       # primary v2 pipeline notebook (with Deploy Web Studio cell)
├── warehouse/                        # runtime asset storage
│   ├── models/                       # base model weights
│   ├── lora/                         # character LoRA adapters
│   ├── outputs/                      # generated videos
│   └── checkpoints/                  # resume checkpoints
├── main.py                           # CLI entry point
├── setup.sh
├── requirements.txt
└── sample_script.txt

Models Used (v2)

Stage Model Purpose VRAM
Keyframes cagliostrolab/animagine-xl-3.1 (SDXL) + character LoRA identity-locked anime stills ~12GB
Identity conditioning h94/IP-Adapter ip-adapter_sdxl.bin image-to-image face anchoring shares SDXL UNet
Story decomposer Gemini 2.5 Flash + Claude Sonnet 4.6 shot list + cinematic refinement API only
Driving clip Wan-AI/Wan2.2-I2V-A14B-Diffusers (MoE 14B) motion source for Phase 3b ~42GB peak
Anime style Wan 2.2 anime LoRA (Civitai community) anime aesthetic on Wan output <1GB
Face lock Wan-AI/Wan2.2-Animate-14B reference face + driving motion -> output ~46GB peak
Face restore GFPGAN v1.4 every-2nd-frame face cleanup ~3GB
Temporal upscale RIFE 4.x 16fps -> 24fps interpolation ~6GB
Spatial upscale Real-ESRGAN x4plus_anime_6B 480p -> 720p+ sharpening ~6GB

Each phase fully unloads before the next loads, so peak VRAM stays within A6000 limits.

Tech Stack

Category Tools
ML PyTorch 2.5.1+cu124, Diffusers 0.36, PEFT, Transformers, Accelerate
Video Wan2.2-I2V-A14B, Wan2.2-Animate-14B, CogVideoX-2B (fallback), AnimateDiff (fallback)
Identity IP-Adapter SDXL, character LoRA, GroundingDINO + SAM + CLIP
NLP Gemini 2.5 Flash, Claude Sonnet 4.6, rule-based fallback
Post RIFE, Real-ESRGAN, GFPGAN, OpenCV, ffmpeg
API FastAPI, Uvicorn, Pydantic
Frontend Vite 6, React 18, TypeScript, Tailwind v4, TanStack Query, React Router, Zustand, Lucide
Queue Celery, Redis
Infra RunPod (primary), Google Colab, Kaggle, GCP

Settings (Cell 3 of the notebook)

Parameter Default Description
IMAGE_WIDTH x IMAGE_HEIGHT 768 x 1152 SDXL keyframe resolution (portrait)
SDXL_STEPS 35 SDXL inference steps
SDXL_GUIDANCE 7.0 SDXL guidance scale
LORA_SCALE 1.15 character LoRA scale (early shots; relaxed to 1.0 after shot 2)
WAN_W x WAN_H 480x832 (auto-detected from VRAM) Wan2.2 output resolution
NUM_FRAMES 33 frames per Wan2.2 clip
WAN_STEPS 30 Wan2.2 inference steps
WAN_GUIDANCE 3.0 lower = more motion freedom
FPS -> TARGET_FPS 16 -> 24 source fps and RIFE-interpolated fps
FACE_RESTORE True GFPGAN every-2nd-frame face restoration
SPATIAL_UPSCALE True Real-ESRGAN x4plus_anime_6B
COLOR_GRADE True anime LUT grading
WAN_ANIME_LORA_REPO Kijai/wan22-anime-style Wan2.2 anime style LoRA repo (skip on failure)

Story Script Format (CLI)

SCENE: Character introduction
CHAR: Sakura
A young woman with pink hair walks through a cherry blossom forest

SCENE: Bridge
CHAR: Sakura
She stops at a wooden bridge and looks at the river below

SCENE: Wind
CHAR: Sakura
The wind gently moves her hair as petals fall around her

Directives: SCENE: (or SHOT:) starts a new shot, CHAR: lists character names, POSE: references a pose video, free text is the prompt.

Training a Character LoRA

Image count Use case
10-15 prototyping; identity may drift on extreme angles
20-30 studio quality; cover front, 3/4, side, expressions, lighting
30+ diminishing returns

Best practices: official screencaps over fan art, mix front + 3/4 + side views, mix expressions, include full-body and close-up shots, use 512px+ on the shortest side.

API Endpoints

Method Endpoint Description
POST /character/create upload character sheet, train LoRA
GET /character/list list all characters
GET /character/{id} get character details
DELETE /character/{id} delete a character
POST /generate/shot generate single shot
POST /generate/sequence generate multi-shot sequence
POST /generate/anime full text -> anime video
GET /job/{job_id} check generation job status
GET /ui/ Web Studio (when frontend/dist/ exists)

Environment Variables

Variable Default Description
AI_CACHE_ROOT ./warehouse root directory for all assets
GEMINI_API_KEY — Gemini Flash API key (free tier sufficient)
ANTHROPIC_API_KEY — Claude Sonnet API key (~$0.003 per story)
REDIS_URL redis://localhost:6379/0 Celery job queue URL
PIXVERSE_API_KEY — PixVerse fallback API key (optional)
API_HOST / API_PORT 0.0.0.0 / 8080 FastAPI bind

Expected Quality on A6000

Metric v1 v1.5 v2 (current)
Identity consistency ~70% ~85% ~95%+
Face stability across shots low medium near-perfect (face pasted from keyframe)
Motion smoothness ok good better (Wan2.2 MoE temporal attention)
Anime aesthetic good good stronger (Wan2.2 anime LoRA)
Visual quality 6-7/10 7.5-8/10 8.7-9.2/10

To break 9.5/10, the next paradigm shift is HunyuanVideo full fp16 on H100 (80GB), or temporal-conditioning models like Sora / Veo 2 — both outside the A6000 envelope.

How It Works

  1. Story Decomposition — Gemini 2.5 Flash plans a structured shot list (SCENE/CHAR/ACTION/CAMERA/MOOD per shot, all sharing one environment). Claude Sonnet 4.6 refines each shot into cinematic anime language with required body movement. Falls back to rule-based on missing keys.
  2. Keyframe Generation — SDXL + animagine-xl-3.1 + character LoRA generate keyframe 0 with text2img. Keyframes 1+ use img2img chaining (StableDiffusionXLImg2ImgPipeline) with adaptive strength decay (0.40 -> 0.25). IP-Adapter SDXL conditions every shot on the identity anchor (refreshed every 3 shots). A pixel-drift quality gate regenerates outliers.
  3. Driving Clip (Phase 3a) — Wan2.2-I2V-A14B (Mixture-of-Experts) plus an optional Wan 2.2 anime LoRA generates a short clip per shot. Center-crop resize avoids face-proportion distortion. Face quality at this stage is irrelevant - it gets overwritten in Phase 3b.
  4. Face Lock (Phase 3b) — Wan2.2-Animate-14B decouples skeleton (body motion) from facial expression. The driving clip provides motion; the SDXL keyframe is the face reference. Output has the keyframe's face at every frame with the driving clip's motion. Falls back to Track A (driving clips become final) if Animate is unavailable.
  5. Face Restoration — Two-pass temporal smoothing wraps a face-region-only GFPGAN pass applied to every 2nd frame (prevents identity drift from over-restoration; preserves anime texture).
  6. Temporal + Spatial Upscale — RIFE interpolates 16fps -> 24fps; Real-ESRGAN x4plus_anime_6B sharpens each frame.
  7. Color Grading — anime LUT grading with palette presets (warm, cool, vibrant, muted).
  8. Assembly — cross-dissolve between adjacent clips, final mp4 written via OpenCV.

Contributing

Contributions welcome.

  1. Fork the repository
  2. Create a feature branch (git checkout -b feature/your-feature)
  3. Commit your changes
  4. Push to the branch
  5. Open a pull request

About

Generate consistent anime videos from a text script using AI — no animators needed

huggingface.co/AnimeLoom

Topics

python open-source ai anime ai-agents anime-generator anime-generation

Resources

Readme

License

Apache-2.0 license
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages