-
Notifications
You must be signed in to change notification settings - Fork 58
Description
Problem
When creating a pod with runpodctl pod create, there's no way to specify the minimum CUDA version required by the container image. This leads to pods being assigned to machines with incompatible (older) NVIDIA drivers, causing container startup failures:
nvidia-container-cli: requirement error: unsatisfied condition: cuda>=12.6,
please update your driver to a newer version, or use an earlier cuda container
The only workaround is to manually pick GPU types known to have recent drivers (e.g., RTX 5090), which defeats the purpose of flexible GPU selection.
Proposed Solution
Add a --min-cuda-version flag (or similar) to runpodctl pod create:
runpodctl pod create \
--image runpod/worker-comfyui:5.7.1-base \
--gpu-id "NVIDIA GeForce RTX 4090" \
--min-cuda-version 12.6 \
...This would filter out machines whose driver doesn't support the requested CUDA version, preventing the pod from being scheduled on incompatible hardware.
Context
The RunPod GraphQL API already has some filtering capabilities. Exposing CUDA version filtering through runpodctl would make it much easier to work with newer CUDA-based images without having to guess which GPU types or datacenters have updated drivers.
This came up while building automated Dockerfile verification scripts that spin up pods programmatically — having to hardcode specific GPU types as a driver workaround is fragile and costly.