feat(extensions): add vLLM backend support

[shlok-mehndiratta]

Description

This PR introduces vLLM as an opt-in Extension Service, directly addressing the recent call for alternative, high-throughput inference backends.

While the core llama-server is excellent for broad consumer hardware compatibility, advanced users with multi-GPU arrays or datacenter hardware are heavily bottlenecked without PagedAttention and continuous batching. This extension provides a first-class, drop-in vLLM OpenAI-compatible server that users can route their ODS traffic to by simply pointing LLM_API_URL to http://vllm:8000/v1.

Implementation Details

Built on top of the latest ods directory re-architecture, this PR strictly follows Pattern 2 (Empty Base with Full GPU Overlay) from EXTENSIONS.md. It safely isolates the addition without touching any High-Risk core routing logic.

• GPU Backends: Provides full overlays for both nvidia (using vllm/vllm-openai:latest) and amd (using rocm/vllm:latest, mounting /dev/kfd and /dev/dri).
• Environment Configuration: Exposes VLLM_MODEL, VLLM_TP_SIZE, and HUGGING_FACE_HUB_TOKEN via .env variables so users can dynamically spin up gated models and scale tensor parallelism.
• Cache Mounting: Mounts ./data/models/huggingface to /root/.cache/huggingface:z for proper persistence and strict SELinux compliance.
• NCCL/RCCL Crash Prevention: Explicitly allocates shm_size: '8g' in both compose overlays. Without this, users running VLLM_TP_SIZE > 1 immediately hit bus error crashes due to Docker's default 64MB /dev/shm limit restricting NCCL/RCCL shared memory allocation.
• Port Safety: Runs internally on 8000 but binds externally to 8008 (VLLM_PORT) to prevent collision with whisper (9000) or llama-server (8080).

Validation

• Passed python3 scripts/audit-extensions.py --project-dir . vllm with 0 errors/warnings.
• Manifest successfully validated against the new ods.services.v1 schema.
• git diff --check passes cleanly (no trailing whitespace).
• Tested against the latest upstream/main merge to guarantee zero integration conflicts.

[Lightheartdevs]

Thanks for taking on the vLLM extension path here; the GPU-only overlay shape is a good fit for how ODS models optional heavy inference services.

Advisory status: not ready yet

Quickest path to merge:

• ods/extensions/services/vllm/compose.nvidia.yaml:7 and ods/extensions/services/vllm/compose.amd.yaml:7: bind the host port through the repo's normal localhost-first contract, e.g. ${BIND_ADDRESS:-127.0.0.1}:${VLLM_PORT:-8008}:8000. As written, ${VLLM_PORT:-8008}:8000 publishes vLLM on all interfaces by default, which bypasses the ODS network-exposure posture and could expose an OpenAI-compatible model endpoint on the LAN unintentionally.
• ods/extensions/services/vllm/compose.nvidia.yaml:3, ods/extensions/services/vllm/compose.amd.yaml:3, and ods/config/dependency-lock.json: replace the :latest images with retained version tags or @sha256 digest pins and add matching dependency-lock entries. The repo has an explicit runtime image pin contract, and this head currently fails it for both vllm/vllm-openai:latest and rocm/vllm:latest.

What I checked:

• PR metadata, changed files, full diff, and existing comments/reviews; GitHub reports no checks for this branch yet.
• python3 scripts/audit-extensions.py --project-dir . vllm passes.
• NVIDIA and AMD compose config checks pass with the same placeholder secrets used by CI.
• python3 tests/test-dependency-pins.py fails on the two new unpinned :latest image refs above.

Thanks again for pushing this forward.

[Lightheartdevs]

Thanks for tightening up the vLLM extension follow-up; this head addresses the two concrete blockers from the earlier advisory pass.

Advisory status: looks merge-ready

What looks good:

• ods/extensions/services/vllm/compose.nvidia.yaml and ods/extensions/services/vllm/compose.amd.yaml now bind through ${BIND_ADDRESS:-127.0.0.1}:${VLLM_PORT:-8008}:8000, which keeps the optional OpenAI-compatible endpoint aligned with the repo's localhost-first exposure posture.
• The vLLM images are now pinned and recorded in ods/config/dependency-lock.json, so the dependency-pin contract no longer rejects the new runtime images.
• The change stays scoped to a Pattern-2 optional extension plus dependency-lock metadata; it does not alter core llama-server routing or the installer lifecycle.

What I checked:

• PR metadata, current head SHA, changed files, diff, and the prior advisory comment; the previous marker was for an older head only.
• git diff --check origin/main...HEAD passes.
• python3 scripts/audit-extensions.py --project-dir . vllm passes.
• python3 tests/test-dependency-pins.py passes.
• Docker compose config checks pass for base+nvidia+vLLM and base+amd+vLLM with CI-style placeholder secrets.
• gh pr checks currently reports no checks for this branch, so this is based on focused local validation rather than a GitHub check run.

Thanks again for pushing this forward.