[Enhancement] Resilient Multi-Version Fallback & Build Channels for CUDA Stack

[ptrecenti]

Description

Recent build breakages in cudastack:standard (e.g., PR #1653) highlight a systemic fragility: the project's declarative, hardcoded versioning frequently outpaces NVIDIA's public distribution CDN. When a marketing announcement precedes the actual repository update, hardcoded wget or apt-get calls fail, aborting multi-hour container builds.

To maintain a "cutting-edge" profile while ensuring production-grade resilience, I propose a refactor of the package acquisition architecture.

Proposed Architectural Improvements

1. Implement Fallback Resolution Pattern

Refactor shell installers (e.g., install_cusparselt.sh, install_nvshmem.sh) to accept a prioritized list of versions. The installer should iterate through the list and download the first version that returns a 200 OK.

Example Logic:

# In config.py: 'CUSPARSELT_VERSIONS': '0.9.0,0.8.1'
IFS=',' read -ra VERSIONS <<< "$CUSPARSELT_VERSIONS"
for VER in "${VERSIONS[@]}"; do
    if wget --spider --quiet "$URL_FOR_$VER"; then
        wget -O "$ARCHIVE" "$URL_FOR_$VER"
        DOWNLOAD_SUCCESS=1; break
    fi
done

This allows maintainers to safely push "speculative" versions to the top of the list without breaking current builds for users.

2. Formalize Build Channels (stable vs edge)

Introduce explicit tags in config.py to isolate experimental versioning from verified releases.

• cudastack:stable: Uses only empirically verified versions available in public APT/Redist channels.
• cudastack:edge: Uses the newest announced versions (may 404 during the first 48-72 hours of a release).

3. Dynamic Network Index Parsing

Instead of hardcoding patch suffixes (e.g., mapping 0.8.0 -> .4 vs 0.8.1 -> .1), the build scripts should query the NVIDIA redistribution directory indexes via curl to dynamically resolve the highest available patch version for the requested major/minor.

4. Build on Centralized Repo Management (Ref: PR #1657)

Integrate these fallbacks into the proposed cuda-repo infrastructure. A centralized pre-flight check should validate repository connectivity and package availability before the main cudastack build begins.

Context

This proposal addresses the recurring "revert vs progress" friction seen in PR #1653, ensuring that the repository stays ahead of the curve without sacrificing the reliability expected by the Jetson developer community.

CC: @dusty-nv @johnnynunez @ptrecenti

[johnnynunez]

we don't install global keyring because with that you cannot control all exactly packages that you want.
Anyways all is moving to cuda 13.2 https://developer.nvidia.com/blog/cuda-13-2-introduces-enhanced-cuda-tile-support-and-new-python-features/ , frameworks will distribute for jetson agx orin natively and ngc containers will be compatible with jetson orin too

[ptrecenti]

@johnnynunez, I completely agree on avoiding global keyrings. Strict, deterministic .deb isolation is non-negotiable for NGC containers to prevent APT from breaking the AI stack.
However, the current hardcoded wget URLs are dropping 404s when NVIDIA's CDN updates a minor patch, breaking multi-hour builds (e.g., when provisioning an Orin 64GB for heavy LLM inference).

The proposal isn't to use APT's dependency solver. It's simply to add a basic array fallback to the existing bash scripts (e.g., trying .2 if .1 404s) to survive CDN shifts until CUDA 13.2 fully unifies the ecosystem.

Would you be open to a PR that just adds this lightweight wget fallback loop to the current installer scripts?