Popular repositories Loading

1. llm-d-infra llm-d-infra Public

   llm-d helm charts and deployment examples

   Go Template 50 55

2. llm-d-modelservice llm-d-modelservice Public

   helm charts for deploying models with llm-d

   Go Template 29 52

3. llm-d-fast-model-actuation llm-d-fast-model-actuation Public

   Kubernetes controllers for fast model actuation using vLLM sleep/wake and launcher-based model swapping

   Go 9 11

4. batch-gateway batch-gateway Public

   The batch gateway is an llm-d implementation of the OpenAI batch inference API

   Go 7 10

5. secure-inference secure-inference Public

   Go 3 2

6. ig-wva ig-wva Public

   Workload Variant Autoscaler is a service to compute the cost-optimal provisioning of heterogeneous accelerators for inference workloads with varying request latency objectives

   Jupyter Notebook 2 1