audio_model.onnx: Protobuf parsing failed with onnxruntime-node 1.21.0 – causes OOM crash (exit 137)

[Bagoboga]

Bug: audio_model.onnx fails with "Protobuf parsing failed" on onnxruntime-node 1.21.0

Environment

• OS: Windows 11 with WSL2 + Docker Desktop
• GPU: NVIDIA GeForce RTX 4070 (8GB)
• CUDA: 12.8.0 / Driver: 591.74
• Image: edit-mind-background-jobs:latest-gpu (v0.14.3)
• onnxruntime-node: 1.21.0

Description
Every time background-jobs processes audio embeddings, it fails with:

Failed to process audio embedding: Error: Load model from /ml-models/embedding-models/Xenova/clap-htsat-unfused/onnx/audio_model.onnx failed: Protobuf parsing failed.

Verified

• The file is NOT corrupt: 112.1 MB, valid ONNX header (08 prefix)
• Deleting and re-downloading the model does not fix the issue
• The error occurs on every scene, causing massive log spam and eventual OOM crash (exit 137)

File info

Filstorlek: 117528416 bytes
Första bytes: 08061207
onnxruntime version: { common: '1.21.0', node: '1.21.0' }

Impact

• Audio and visual embeddings fail completely
• background-jobs crashes with OOM (exit 137) due to the model being loaded once per scene in parallel
• Videos get stuck at "Embedding Visuals 100%" indefinitely

Question
Is there an environment variable or config option to disable audio embeddings as a workaround?

[IliasHad]

@Bagoboga Thank you so much for the detailed issue. I appreciate that.

Is there an environment variable or config option to disable audio embeddings as a workaround?

Unfortunately, there's no option to disable audio embedding.

I just wanna double check if you have the ml-models volume mounted to the background-jobs service as mentioned here at

edit-mind/docker-compose.yml

Line 25 in 82a4a38

- ml-models:/app/ml-models:rw

Can you send the file names and sizes that you have under the/ml-models/embedding-models/Xenova folder?

[Bagoboga]

Hi, I have investigated the directory structure as requested.

Running: docker exec edit-mind-background-jobs sh -c "ls -lhR /app/ml-models/embedding-models/Xenova"
Returns: ls: cannot access '/app/ml-models/embedding-models/Xenova': No such file or directory

And Running: docker exec edit-mind-background-jobs sh -c "ls -lhR /ml-models/embedding-models/Xenova"
Returns: /ml-models/embedding-models/Xenova:
total 12K
drwxr-xr-x 3 root root 4.0K Mar 2 10:52 all-mpnet-base-v2
drwxr-xr-x 3 root root 4.0K Mar 2 10:52 clap-htsat-unfused
drwxr-xr-x 3 root root 4.0K Mar 2 10:52 clip-vit-base-patch32

/ml-models/embedding-models/Xenova/all-mpnet-base-v2:
total 708K
-rw-r--r-- 1 root root 591 Mar 2 10:52 config.json
drwxr-xr-x 2 root root 4.0K Mar 2 10:52 onnx
-rw-r--r-- 1 root root 695K Mar 2 10:52 tokenizer.json
-rw-r--r-- 1 root root 357 Mar 2 10:52 tokenizer_config.json

/ml-models/embedding-models/Xenova/all-mpnet-base-v2/onnx:
total 209M
-rw-r--r-- 1 root root 209M Mar 2 10:53 model_fp16.onnx

/ml-models/embedding-models/Xenova/clap-htsat-unfused:
total 12K
-rw-r--r-- 1 root root 699 Mar 2 10:52 config.json
drwxr-xr-x 2 root root 4.0K Mar 2 10:52 onnx
-rw-r--r-- 1 root root 541 Mar 2 10:52 preprocessor_config.json

/ml-models/embedding-models/Xenova/clap-htsat-unfused/onnx:
total 113M
-rw-r--r-- 1 root root 113M Mar 2 10:53 audio_model.onnx

/ml-models/embedding-models/Xenova/clip-vit-base-patch32:
total 16K
-rw-r--r-- 1 root root 4.5K Mar 2 10:52 config.json
drwxr-xr-x 2 root root 4.0K Mar 2 10:52 onnx
-rw-r--r-- 1 root root 520 Mar 2 10:52 preprocessor_config.json

/ml-models/embedding-models/Xenova/clip-vit-base-patch32/onnx:
total 168M
-rw-r--r-- 1 root root 168M Mar 2 10:53 vision_model_fp16.onnx

[IliasHad]

Awesome, thank you for your response.

Returns: ls: cannot access '/app/ml-models/embedding-models/Xenova': No such file or directory

This is to be expected because we have the models inside /ml-models.

I'll be focusing on this issue and will get back to you if I need more details or have an update. Thank you for your contribution :)

[IliasHad]

@Bagoboga, I couldn't replicate this issue on my machine. I tried virtualization on my Apple machine with an ARM cpu but I couldn't replicate this issue because of a lack of official support for Docker over Windows (ARM) (https://docs.docker.com/desktop/setup/install/windows-install/).

I added two new environment variables that you can set over the background-jobs service to disable audio or visual embedding processing stage. You need to upgrade to 0.14.4 by using this command docker compose pull && docker compose up

Let me know if this solution works for you. Thank you for your contribution 🥇

  background-jobs:
    restart: unless-stopped
    image: ghcr.io/iliashad/edit-mind-background-jobs:latest
    container_name: edit-mind-background-jobs
    env_file:
      - .env.system
      - .env
    ports:
      - "${BACKGROUND_JOBS_PORT}:${BACKGROUND_JOBS_PORT}"
    environment:
      DISABLE_AUDIO_EMBEDDINGS: true
      DISABLE_VISUAL_EMBEDDINGS: false
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
      chroma:
        condition: service_started
      ml:
          condition: service_started
    networks:
      - app-network
    volumes:
      - ${HOST_MEDIA_PATH:-./media}:/media/videos:ro
      - .data:/app/data
      - ml-models:/app/ml-models:rw

Note: I'll keep this issue open until we have full support for the audio embedding for WSL 2 with Windows 11.