Can't load Whisper Model

[renatobrusarosco]

Hi there,

I don't have a lot of knowledge about programming, and I'm struggling to deal with your tool. I tried to use the tool in Jupyter Anaconda. I installed all the required libraries, but when I attempted to run the final code, I encountered the same error message multiple times:

vbnet
Copy code
Python >= 3.10
Using cache found in C:\Users\renat/.cache\torch\hub\snakers4_silero-vad_master
Using Demucs
Using standard Whisper
LOADING: medium GPU:0 BS: 2
100%|█████████████████████████████████████| 1.42G/1.42G [00:25<00:00, 59.9MiB/s]
Can't load Whisper model: STD/medium

Below you can find more information:

RuntimeError Traceback (most recent call last)
File ~\Downloads\Jupyter\WhisperHallu\transcribeHallu.py:110, in loadModel(gpu, modelSize)
109 print("LOADING: "+modelSize+" GPU:"+gpu+" BS: "+str(beam_size))
--> 110 model = whisper.load_model(modelSize,device=torch.device("cuda:"+gpu)) #May be "cpu"
111 elif whisperFound == "SM4T":

File ~\AppData\Roaming\Python\Python311\site-packages\whisper_init_.py:146, in load_model(name, device, download_root, in_memory)
143 with (
144 io.BytesIO(checkpoint_file) if in_memory else open(checkpoint_file, "rb")
145 ) as fp:
--> 146 checkpoint = torch.load(fp, map_location=device)
147 del checkpoint_file

File ~\AppData\Roaming\Python\Python311\site-packages\torch\serialization.py:1014, in load(f, map_location, pickle_module, weights_only, mmap, **pickle_load_args)
1013 raise pickle.UnpicklingError(UNSAFE_MESSAGE + str(e)) from None
-> 1014 return _load(opened_zipfile,
1015 map_location,
1016 pickle_module,
1017 overall_storage=overall_storage,
1018 **pickle_load_args)
1019 if mmap:

File ~\AppData\Roaming\Python\Python311\site-packages\torch\serialization.py:1422, in _load(zip_file, map_location, pickle_module, pickle_file, overall_storage, **pickle_load_args)
1421 unpickler.persistent_load = persistent_load
-> 1422 result = unpickler.load()
1424 torch._utils._validate_loaded_sparse_tensors()

File ~\AppData\Roaming\Python\Python311\site-packages\torch\serialization.py:1392, in _load..persistent_load(saved_id)
1391 nbytes = numel * torch._utils._element_size(dtype)
-> 1392 typed_storage = load_tensor(dtype, nbytes, key, _maybe_decode_ascii(location))
1394 return typed_storage

File ~\AppData\Roaming\Python\Python311\site-packages\torch\serialization.py:1366, in _load..load_tensor(dtype, numel, key, location)
1363 # TODO: Once we decide to break serialization FC, we can
1364 # stop wrapping with TypedStorage
1365 typed_storage = torch.storage.TypedStorage(
-> 1366 wrap_storage=restore_location(storage, location),
1367 dtype=dtype,
1368 _internal=True)
1370 if typed_storage._data_ptr() != 0:

File ~\AppData\Roaming\Python\Python311\site-packages\torch\serialization.py:1299, in _get_restore_location..restore_location(storage, location)
1298 def restore_location(storage, location):
-> 1299 return default_restore_location(storage, str(map_location))

File ~\AppData\Roaming\Python\Python311\site-packages\torch\serialization.py:381, in default_restore_location(storage, location)
380 for _, _, fn in _package_registry:
--> 381 result = fn(storage, location)
382 if result is not None:

File ~\AppData\Roaming\Python\Python311\site-packages\torch\serialization.py:274, in _cuda_deserialize(obj, location)
273 if location.startswith('cuda'):
--> 274 device = validate_cuda_device(location)
275 if getattr(obj, "_torch_load_uninitialized", False):

File ~\AppData\Roaming\Python\Python311\site-packages\torch\serialization.py:258, in validate_cuda_device(location)
257 if not torch.cuda.is_available():
--> 258 raise RuntimeError('Attempting to deserialize object on a CUDA '
259 'device but torch.cuda.is_available() is False. '
260 'If you are running on a CPU-only machine, '
261 'please use torch.load with map_location=torch.device('cpu') '
262 'to map your storages to the CPU.')
263 device_count = torch.cuda.device_count()

RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.

During handling of the above exception, another exception occurred:

SystemExit Traceback (most recent call last)
[... skipping hidden 1 frame]

Cell In[3], line 26
15 #Example
16 #lng="uk"
17 #prompt= "Whisper, Ok. "
(...)
23 # +"Ok, Whisper. "
24 #path="/path/to/your/uk/sound/file"
---> 26 loadModel("0")
27 result = transcribePrompt(path=path, lng=lng, prompt=prompt)

File ~\Downloads\Jupyter\WhisperHallu\transcribeHallu.py:117, in loadModel(gpu, modelSize)
116 print("Can't load Whisper model: "+whisperFound+"/"+modelSize)
--> 117 sys.exit(-1)

SystemExit: -1

During handling of the above exception, another exception occurred:

AttributeError Traceback (most recent call last)
[... skipping hidden 1 frame]

File ~\anaconda3\Lib\site-packages\IPython\core\interactiveshell.py:2097, in InteractiveShell.showtraceback(self, exc_tuple, filename, tb_offset, exception_only, running_compiled_code)
2094 if exception_only:
2095 stb = ['An exception has occurred, use %tb to see '
2096 'the full traceback.\n']
-> 2097 stb.extend(self.InteractiveTB.get_exception_only(etype,
2098 value))
2099 else:
2101 def contains_exceptiongroup(val):

File ~\anaconda3\Lib\site-packages\IPython\core\ultratb.py:710, in ListTB.get_exception_only(self, etype, value)
702 def get_exception_only(self, etype, value):
703 """Only print the exception type and message, without a traceback.
704
705 Parameters
(...)
708 value : exception value
709 """
--> 710 return ListTB.structured_traceback(self, etype, value)

File ~\anaconda3\Lib\site-packages\IPython\core\ultratb.py:568, in ListTB.structured_traceback(self, etype, evalue, etb, tb_offset, context)
565 chained_exc_ids.add(id(exception[1]))
566 chained_exceptions_tb_offset = 0
567 out_list = (
--> 568 self.structured_traceback(
569 etype,
570 evalue,
571 (etb, chained_exc_ids), # type: ignore
572 chained_exceptions_tb_offset,
573 context,
574 )
575 + chained_exception_message
576 + out_list)
578 return out_list

File ~\anaconda3\Lib\site-packages\IPython\core\ultratb.py:1435, in AutoFormattedTB.structured_traceback(self, etype, evalue, etb, tb_offset, number_of_lines_of_context)
1433 else:
1434 self.tb = etb
-> 1435 return FormattedTB.structured_traceback(
1436 self, etype, evalue, etb, tb_offset, number_of_lines_of_context
1437 )

File ~\anaconda3\Lib\site-packages\IPython\core\ultratb.py:1326, in FormattedTB.structured_traceback(self, etype, value, tb, tb_offset, number_of_lines_of_context)
1323 mode = self.mode
1324 if mode in self.verbose_modes:
1325 # Verbose modes need a full traceback
-> 1326 return VerboseTB.structured_traceback(
1327 self, etype, value, tb, tb_offset, number_of_lines_of_context
1328 )
1329 elif mode == 'Minimal':
1330 return ListTB.get_exception_only(self, etype, value)

File ~\anaconda3\Lib\site-packages\IPython\core\ultratb.py:1173, in VerboseTB.structured_traceback(self, etype, evalue, etb, tb_offset, number_of_lines_of_context)
1164 def structured_traceback(
1165 self,
1166 etype: type,
(...)
1170 number_of_lines_of_context: int = 5,
1171 ):
1172 """Return a nice text document describing the traceback."""
-> 1173 formatted_exception = self.format_exception_as_a_whole(etype, evalue, etb, number_of_lines_of_context,
1174 tb_offset)
1176 colors = self.Colors # just a shorthand + quicker name lookup
1177 colorsnormal = colors.Normal # used a lot

File ~\anaconda3\Lib\site-packages\IPython\core\ultratb.py:1063, in VerboseTB.format_exception_as_a_whole(self, etype, evalue, etb, number_of_lines_of_context, tb_offset)
1060 assert isinstance(tb_offset, int)
1061 head = self.prepare_header(str(etype), self.long_header)
1062 records = (
-> 1063 self.get_records(etb, number_of_lines_of_context, tb_offset) if etb else []
1064 )
1066 frames = []
1067 skipped = 0

File ~\anaconda3\Lib\site-packages\IPython\core\ultratb.py:1131, in VerboseTB.get_records(self, etb, number_of_lines_of_context, tb_offset)
1129 while cf is not None:
1130 try:
-> 1131 mod = inspect.getmodule(cf.tb_frame)
1132 if mod is not None:
1133 mod_name = mod.name

AttributeError: 'tuple' object has no attribute 'tb_frame'

In my research, I discovered that the issue is related to something with the GPU. Here are my PC specifications:
AMD Ryzen 7 6800H with Radeon Graphics 3.20 GHz
16.0 GB RAM
NVIDIA GeForce RTX 3070 Ti
Thank you in advance.

[EtienneAb3d]

@renatobrusarosco

Seems your problem is here:

RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.

You need a graphic card with CUDA installed.

Try to make Whisper working alone first, then, WhisperHallu should also work.
https://github.com/openai/whisper

[renatobrusarosco]

I have a NVIDIA GeForce RTX 3070 Ti, after researching a bit, I found out that I needed to install CUDA in a specific way, and this problem was solved. However, another issue has arisen. When I run the final code, I always receive this message (I have the latest version of ffmpeg installed). Could you help me? Thank you.

RuntimeError Traceback (most recent call last)
Cell In[19], line 27
15 # Example
16 # lng = "uk"
17 # prompt = "Whisper, Ok. "
(...)
23 # "Ok, Whisper. "
24 # path = "/path/to/your/uk/sound/file"
26 loadModel("0")
---> 27 result = transcribePrompt(path=path, lng=lng, prompt=prompt)

File ~\Downloads\Jupyter\WhisperHalluEnv\WhisperHallu\transcribeHallu.py:201, in transcribePrompt(path, lng, prompt, lngInput, isMusic, addSRT, truncDuration, maxDuration)
199 print("PROMPT="+prompt,flush=True)
200 opts = dict(language=lng,initial_prompt=prompt)
--> 201 return transcribeOpts(path, opts,lngInput,isMusic=isMusic,addSRT=addSRT,truncDuration=truncDuration,maxDuration=maxDuration)

File ~\Downloads\Jupyter\WhisperHalluEnv\WhisperHallu\transcribeHallu.py:265, in transcribeOpts(path, opts, lngInput, isMusic, onlySRT, addSRT, truncDuration, maxDuration)
260 pathDemucs=pathIn+".vocals.wav" #demucsDir+"/htdemucs/"+os.path.splitext(os.path.basename(pathIn))[0]+"/vocals.wav"
261 #Demucs seems complex, using CLI cmd for now
262 #aCmd = "python -m demucs --two-stems=vocals -d "+device+":"+cudaIdx+" --out "+demucsDir+" "+pathIn
263 #print("CMD: "+aCmd)
264 #os.system(aCmd)
--> 265 demucs_audio(pathIn=pathIn,model=modelDemucs,device="cuda:"+cudaIdx,pathVocals=pathDemucs,pathOther=pathIn+".other.wav")
266 print("T=",(time.time()-startTime))
267 print("PATH="+pathDemucs,flush=True)

File ~\Downloads\Jupyter\WhisperHalluEnv\WhisperHallu\demucsWrapper.py:43, in demucs_audio(pathIn, model, device, pathVocals, pathOther)
41 source_idx=model.sources.index(name)
42 source=result[0, source_idx].mean(0)
---> 43 torchaudio.save(pathIn+"."+name+".wav", source[None], model.samplerate)

File ~\AppData\Roaming\Python\Python311\site-packages\torchaudio_backend\utils.py:311, in get_save_func..save(uri, src, sample_rate, channels_first, format, encoding, bits_per_sample, buffer_size, backend, compression)
223 def save(
224 uri: Union[BinaryIO, str, os.PathLike],
225 src: torch.Tensor,
(...)
233 compression: Optional[Union[CodecConfig, float, int]] = None,
234 ):
235 """Save audio data to file.
236
237 Note:
(...)
309
310 """
--> 311 backend = dispatcher(uri, format, backend)
312 return backend.save(
313 uri, src, sample_rate, channels_first, format, encoding, bits_per_sample, buffer_size, compression
314 )

File ~\AppData\Roaming\Python\Python311\site-packages\torchaudio_backend\utils.py:221, in get_save_func..dispatcher(uri, format, backend_name)
219 if backend.can_encode(uri, format):
220 return backend
--> 221 raise RuntimeError(f"Couldn't find appropriate backend to handle uri {uri} and format {format}.")

RuntimeError: Couldn't find appropriate backend to handle uri data/KatyPerry-Firework.mp3.WAV.wav.drums.wav and format None.

[EtienneAb3d]

@renatobrusarosco

First, check that this file exists and is not empty at the moment the error occurs:
data/KatyPerry-Firework.mp3.WAV.wav.drums.wav

Each processing step is adding a suffix to the original file path, and is creating a specific LOG file.
Check each of these LOG files to see if something is more clear about the problem at one step or an other.

[gmmarc]

For people facing this issue without GPU, here's how you can change it to CPU.

In https://github.com/EtienneAb3d/WhisperHallu/blob/main/transcribeHallu.py#L110, set device to cpu:

model = whisper.load_model(modelSize,device=torch.device("cpu"))

Same thing in https://github.com/EtienneAb3d/WhisperHallu/blob/main/transcribeHallu.py#L265:

demucs_audio(pathIn=pathIn,model=modelDemucs,device="cpu",pathVocals=pathDemucs,pathOther=pathIn+".other.wav")

You will need PyTorch compiled for CPU, I did:

pip uninstall torch
pip install --pre torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/nightly/cpu

It will take some time, but works.

Tested on macOS Sonoma 14.1 M2