[v4] Improve download progress tracking (model cache registry and define which files will be loaded for pipelines)

[nico-martin]

Improved Download Progress Tracking

Problem

Transformers.js couldn't reliably track total download progress because:

• File lists weren't known before downloads started
• File sizes were inconsistent (compressed vs uncompressed)
• No cache awareness before initiating downloads

Solution

New Exported Functions

• get_files(): Determines required files before downloading
• get_model_files() / get_tokenizer_files() / get_processor_files(): Helper functions to identify files for each component
• get_file_metadata(): Fetches file metadata using Range requests without downloading full content
  • Returns fromCache boolean to identify cached files
  • Ensures consistent uncompressed file sizes
• is_cached(): Checks if all files from a model are already in cache

Enhanced Progress Tracking

• readResponse() with expectedSize: Falls back to metadata when content-length header is missing
• total_progress callback: Provides aggregate progress across all files

Review

One thing I am not super confident is the get_model_files function. I tried to test it with different model architectures, but maybe I missed some that load files that are not in that function. @xenova, could you smoke-test some models and write mie the models that fail?

Easiest way to do that is:

import {
  get_files,
  pipeline,
} from "@huggingface/transformers";

const expectedFiles = await get_files(
  "onnx-community/gemma-3-270m-it-ONNX",
  {
    dtype: "fp32",
    device: "webgpu",
  }
);
const loadedFiles = new Set();
const pipe = await pipeline(
  "text-generation",
  "onnx-community/gemma-3-270m-it-ONNX",
  {
    dtype: "fp32",
    device: "webgpu",
    progress_callback: (e) => {
      if (e.file) loadedFiles.add(e.file);
    },
  }
);

console.log(
  "SAME FILES:",
  expectedFiles.sort().join(",") === Array.from(loadedFiles).sort().join(",")
);

---

Closes #1345

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[xenova]

Very exciting PR! 🙌 Just a quick review from scanning the PR briefly.

[xenova]

Solid progress! Thanks 🔥

[xenova]

Huge PR! 🔥 Thanks so much @nico-martin.