Runpod VLM (Vision-Language Models) — Chat · Caption · VQA · Grounding

Serverless GPU vision-language model worker. Send an OpenAI-style messages array with image content blocks and get a chat-style answer back — caption, VQA, multi-turn dialogue with images, or bounding-box grounding for models that support it.

One worker holds one loaded VLM. VLM weights are large and the model owns non-trivial CUDA state, so swaps at runtime are refused — pick your backbone via the VLM_MODEL env variable and redeploy to change it.

Highlights

• Tasks: chat (OpenAI-style with image blocks), caption, caption_long, vqa, grounding
• Curated model allow-list spanning LLaVA, Qwen2-VL, MiniCPM-V, Phi-3 Vision, and InternVL
• OpenAI-compatible message shape — text + image content blocks, multi-turn, multi-image
• Image resolver — URLs, raw base64, data URIs, OpenAI nested {"image_url":{"url":"..."}}
• Bounding-box grounding — for Qwen2-VL the worker parses a bbox from the answer; for others it returns free-text and a bbox_warning
• Quantization — 4bit / 8bit via bitsandbytes for tight VRAM budgets
• Batched requests — pass requests: [...] to fire multiple in one call with per-item error capture
• Per-family chat-template adapters (image-placeholder tokens injected the way each backbone expects)

Curated model allow-list

Model	Family	~VRAM (fp16)	Grounding	License	Strengths
Qwen/Qwen2-VL-2B-Instruct (default)	qwen2-vl	~8 GB	✓	Apache-2.0	Fast, multilingual, bbox-aware, good portability
Qwen/Qwen2-VL-7B-Instruct	qwen2-vl	~18 GB	✓	Tongyi Qianwen	Strong VQA, OCR-aware, dynamic resolution, bbox
llava-hf/llava-1.5-7b-hf	llava	~16 GB	—	LLaMA 2 Community + LLaVA	Reliable captioning / VQA baseline
llava-hf/llava-1.5-13b-hf	llava	~28 GB	—	LLaMA 2 Community + LLaVA	Larger LLaVA, more nuanced reasoning
llava-hf/llava-v1.6-mistral-7b-hf	llava-next	~18 GB	—	Apache-2.0 (Mistral) + LLaVA	High-resolution patches, improved reasoning
openbmb/MiniCPM-V-2_6	minicpm-v	~18 GB	—	Apache-2.0 / OpenBMB	Strong OCR + chart understanding
microsoft/Phi-3-vision-128k-instruct	phi3-vision	~12 GB	—	MIT	128k context, document-friendly
OpenGVLab/InternVL2-8B	internvl	~20 GB	—	MIT / InternVL	Strong general VL benchmarks

Set VLM_ALLOW_ANY=true in env to bypass the allow-list and load any HuggingFace VLM repo. Per-model licensing applies — read the model card before commercial use.

Environment variables

Var	Default	What it does
VLM_MODEL	Qwen/Qwen2-VL-2B-Instruct	Which VLM to load on worker boot
VLM_QUANTIZE	none	none / 4bit / 8bit (bitsandbytes)
VLM_DTYPE	auto	auto / fp16 / bf16 / fp32 when not quantized
VLM_DEVICE_MAP	auto	device_map passed to from_pretrained
VLM_TRUST_REMOTE_CODE	true	Trust HF repo's custom code (needed for some VLMs)
VLM_ALLOW_ANY	false	When true, bypass the curated allow-list
HF_HOME	/root/.cache/huggingface	HuggingFace cache directory

Input schema

Single request (top-level input)

{
  "task": "chat",                                  // "chat" | "caption" | "caption_long" | "vqa" | "grounding"
  "model": "Qwen/Qwen2-VL-2B-Instruct",            // optional; must match the loaded model

  "messages": [...],                               // for "chat"
  "image_url": "https://...",                      // for caption/caption_long/vqa/grounding
  "image_b64": "iVBORw0KGgo...",                   // alternative single-image input
  "images": [...],                                 // alternative plural input
  "question": "...",                               // required for "vqa"
  "target": "the bus",                             // required for "grounding"

  "max_new_tokens": 256,
  "temperature": 0.0,
  "top_p": 1.0,
  "do_sample": false,
  "quantize": "none",                              // "none" | "4bit" | "8bit" (load-time only)
  "batch_images": false                            // include "batch_images": true in the output echo
}

Chat messages — OpenAI content blocks

{
  "task": "chat",
  "messages": [
    {"role": "system", "content": "You are a careful visual analyst."},
    {
      "role": "user",
      "content": [
        {"type": "text",      "text": "What's in this picture?"},
        {"type": "image_url", "image_url": {"url": "https://..."}}
      ]
    }
  ]
}

Image blocks accept any of:

• {"type": "image_url", "image_url": {"url": "..."}} (OpenAI shape)
• {"type": "image_url", "image_url": "..."} (string url)
• {"type": "image", "image": "..."} (any spec we resolve)
• {"type": "image", "image": {"url": "..."}}
• {"type": "image_url", "image_url": "data:image/png;base64,..."}

Batched (top-level input.requests)

{
  "requests": [
    {"task": "caption", "image_url": "..."},
    {"task": "vqa", "image_url": "...", "question": "How many people?"}
  ]
}

All requests in a batch must target the same loaded model (the worker only holds one).

Output shape

Single request:

{
  "response": "A red double-decker bus parked at the curb.",
  "model": "Qwen/Qwen2-VL-2B-Instruct",
  "task": "caption",
  "image_count": 1,
  "finish_reason": "stop"
}

Grounding adds bbox (or null with a warning):

{
  "response": "The bus is at [120, 50, 380, 290] in the image.",
  "bbox": [120, 50, 380, 290],
  "supports_grounding": true,
  "task": "grounding",
  "model": "Qwen/Qwen2-VL-2B-Instruct",
  "image_count": 1,
  "finish_reason": "stop"
}

Batched response wraps a results list:

{
  "model": "Qwen/Qwen2-VL-2B-Instruct",
  "count": 2,
  "id": "vlm-1716000000000",
  "results": [
    { "...single-request shape..." },
    { "...single-request shape..." }
  ]
}

Example requests

1) Caption a single image

{
  "task": "caption",
  "image_url": "https://ultralytics.com/images/bus.jpg",
  "max_new_tokens": 64
}

2) Long, detailed caption

{
  "task": "caption_long",
  "image_url": "https://ultralytics.com/images/zidane.jpg",
  "max_new_tokens": 192
}

3) VQA — answer a question about an image

{
  "task": "vqa",
  "image_url": "https://ultralytics.com/images/bus.jpg",
  "question": "What color is the bus?",
  "max_new_tokens": 32
}

4) Chat — multi-turn with one image

{
  "task": "chat",
  "messages": [
    {"role": "system", "content": "You are a careful visual analyst."},
    {
      "role": "user",
      "content": [
        {"type": "text", "text": "What is the main subject of this photo?"},
        {"type": "image_url", "image_url": {"url": "https://ultralytics.com/images/bus.jpg"}}
      ]
    },
    {"role": "assistant", "content": "A red double-decker bus."},
    {"role": "user", "content": "Are there any people visible?"}
  ],
  "max_new_tokens": 96
}

5) Chat — multi-image comparison

{
  "task": "chat",
  "messages": [
    {
      "role": "user",
      "content": [
        {"type": "text", "text": "Compare these two images."},
        {"type": "image_url", "image_url": {"url": "https://ultralytics.com/images/bus.jpg"}},
        {"type": "image_url", "image_url": {"url": "https://ultralytics.com/images/zidane.jpg"}}
      ]
    }
  ],
  "max_new_tokens": 192
}

6) Grounding — locate an object (Qwen2-VL recommended)

{
  "task": "grounding",
  "image_url": "https://ultralytics.com/images/bus.jpg",
  "target": "bus",
  "max_new_tokens": 96
}

7) Batched — mix tasks in one call

{
  "requests": [
    {"task": "caption", "image_url": "https://ultralytics.com/images/bus.jpg"},
    {
      "task": "vqa",
      "image_url": "https://ultralytics.com/images/zidane.jpg",
      "question": "How many people are in this picture?"
    },
    {
      "task": "grounding",
      "image_url": "https://ultralytics.com/images/bus.jpg",
      "target": "front door of the bus"
    }
  ]
}

8) Quantized — run a bigger model on a smaller GPU

{
  "task": "caption_long",
  "image_url": "https://ultralytics.com/images/zidane.jpg",
  "quantize": "4bit",
  "max_new_tokens": 200
}

quantize is honored only on the first call (which triggers the model load). After that the worker is bound to whatever precision was used.

CLIP vs VLM — when to use which

This worker is the generative counterpart to runpod-clip:

	runpod-clip	runpod-vlm (this)
Output type	Vector embeddings + similarity scores	Free-form text (or text + bbox)
Models	OpenCLIP / SigLIP encoders	LLaVA / Qwen2-VL / MiniCPM-V / Phi-3 Vision
Best for	Search, ranking, zero-shot classification, dedup	Captioning, VQA, dialogue, grounding
GPU cost	Light (encoder only)	Heavy (encoder + LLM decoder)
Latency	Sub-second per batch	Seconds (depends on max_new_tokens)
Workflow shape	"How similar is this image to that text?"	"Tell me about this image / answer my question"

Use CLIP for retrieval, classification, and ranking. Use VLM when you need a natural-language answer or a per-image description.

Local testing

pip install -r requirements.txt   # heavy; tests don't actually need most of this
python3 test_handler.py

test_handler.py stubs out transformers, torch, qwen_vl_utils, and bitsandbytes via sys.modules injection — it runs without a GPU and without the real model. 18 tests covering ~75 assertions exercise the image resolver, message-block parsing, per-family chat templates, all 5 tasks, the bbox parser, batching, per-request error capture, the quantization plumbing, and the model-swap rejection.

Deployment

1. Build: docker build -t your-org/runpod-vlm:latest .
2. Push or link the repo on the RunPod Hub at https://console.runpod.io/hub.
3. Set env vars — at minimum VLM_MODEL. Pick VLM_QUANTIZE=4bit if VRAM is tight.
4. Pick a GPU matching the model's VRAM column above. Cold start = model download + load → first-request latency.

Performance notes

• The first request after worker boot pays the model load cost (10–90 s depending on size and quantization). Subsequent invocations reuse the loaded model.
• 4bit quantization roughly halves VRAM usage at the cost of ~10–20% generation latency and a small accuracy hit. 8bit is in between.
• For interactive use prefer the 2B/7B class; LLaVA-13B and Phi-3 Vision pay off when accuracy matters more than latency.
• temperature=0.0 + do_sample=false (the defaults) give greedy decoding — deterministic answers, no variance call-to-call. Crank up temperature for more varied caption styles.
• Pack multiple requests into a single requests: [...] call when you can — saves on dispatch overhead and HF tokenizer warmup.

License notes per model

• LLaVA 1.5 (7B/13B): weights inherit LLaMA 2 Community License — commercial use requires Meta acceptance and has DAU restrictions. Code is Apache-2.0.
• LLaVA-Next (1.6) Mistral 7B: Apache-2.0 (Mistral base) + LLaVA code license. Generally commercial-friendly.
• Qwen2-VL 7B: Tongyi Qianwen license — commercial use requires registration above a DAU threshold.
• Qwen2-VL 2B: Apache-2.0 — fully commercial-friendly.
• MiniCPM-V 2.6: weights are Apache-2.0; commercial use requires a free registration with OpenBMB for some downstream uses.
• Phi-3 Vision: MIT — commercial-friendly.
• InternVL2 8B: MIT (weights) — commercial-friendly.

Always re-check the upstream HuggingFace model card; licenses evolve.

Notes

• The worker holds one loaded model per process. VLM_MODEL env determines which. Per-request model is validated but cannot trigger a swap (returns an error if it doesn't match).
• Grounding output is best-effort — the bbox parser supports [x1,y1,x2,y2], {"bbox": [...]}, Qwen-style (x1,y1),(x2,y2), and plain comma-separated quartets. If parsing fails the worker returns "bbox": null plus a "bbox_warning" field.
• qwen_vl_utils is imported lazily for Qwen2-VL; if it's missing the worker still works, just without that helper's image-resizing shortcuts.