Configure realtime audio buffer frame / polling rate?

[LumineBot]

Hello, first of all, thank you author for your amazing work for this library (and also whisper.net too!)

I have a question regarding the realtime transcription. So far I can use echo sharp to trainscribe audio in real time by catching RealtimeSegmentRecognizing and RealtimeSegmentRecognized transcription types.

But there's slight problem: I can observe that RealtimeSegmentRecognizing is less accurate, which sounds reasonable and obvious to me since the whole recognizing process is not completed, but my assumption is that it is because the audio buffer is too small / short. I feel like it tries to recognize somewhere around 50ms audio buffer where not even a quarter of word spoken.

Of course this is only based on rough observation, but the result is that displaying text of the latest of these events does not look that pretty. I'm also using SileroVadDetector but I'm not familiar with the inner working of the detection.

I feel like this could be better if I can somehow throttle the processing rate, or in other words make audio bit bigger. There would be more delay overhead but at least it will came out nicer. I could put Task.Delay between my TranscribeAsync call but I don't think it is the right way to do it.

Any advice would be highly appreciated. Thank you in advance!

[sandrohanea]

Hello @LumineBot ,

That is a really good observation.

If the processing time is really fast, it can lead to that type of issue it seems. Usually, it is good as the processing time takes a little longer, so enough frames can accumulate in that time, but if you're using some fast hardware, it will run the Recognizing way too often.

We already have MinTranscriptDuration parameter that can be used to set minimum processing time, but probably that won't be enough.

The reason is that on a fast environment, it will delay the first run of the transcribing events (to 200 ms) but after that it will run, if that completes fast, it will run again with few added samples (e.g. 250 ms, 300ms)

There is another ProcessingInterval parameter, but that is used currently only if the same amount of samples were read, so that we wait for newer samples with that parameter, but this won't be used even if one single sample will be written, and it will be processed again.

I think the solution will be to fix the RealTimeProcessor and if the difference between the current sample length (e.g. 250ms) and the last processing duration (e.g. 200ms) is less than the ProcessingInterval (defaults to 100ms) => To not process "RecognizingEvents".

However, we should still process "RecognizedEvents" if the VAD component detects that exact at 270ms will detect the end of segment.

Will apply the fixes and hopefully release a new version soon so that these parameters will work better.

Thanks for taking the time to fill the issue with such great details!

[LumineBot]

Hi @sandrohanea,

Thank you so much for a thorough explanation!

I'm using RTX 4080 on Windows, and the processing time is fast indeed. It made me curious how fast the transcription processing time is for older hardware / apple m1 and above since I don't have other PC.

Out of curiosity, would not processing these RecognizingEvents affect the performance of slower environments? I reckon it should not, and even if it did, we can tune the intervals/length.

Thanks for taking the time to address and raise fixes for this!

[ivoryguard]

Hi @sandrohanea, thank you for very much about this wonderful proejct!

In my case, I found that adjusting the EchoSharpRealtimeOptions as follows improved recognition quality:

IRealtimeSpeechTranscriptorFactory GetEchoSharpTranscriptorFactory(ISpeechTranscriptorFactory speechTranscriptorFactory, IVadDetectorFactory vadDetectorFactory)
{
    return new EchoSharpRealtimeTranscriptorFactory(speechTranscriptorFactory, vadDetectorFactory, echoSharpOptions: new EchoSharpRealtimeOptions()
    {
        ConcatenateSegmentsToPrompt = false, // Flag to concatenate segments to prompt when new segment is recognized (for the whole session)
        ProcessingInterval = TimeSpan.FromMilliseconds(200),
        MinTranscriptDuration = TimeSpan.FromMilliseconds(400),
    });
}

If MinTranscriptDuration is set to the default value of 200ms, the speech recognition quality drops significantly for non-English languages.
When the MinTranscriptDuration value is increased, recognition quality for non-English languages is greatly improved.

Actually, I’m not exactly sure how all the properties of EchoSharpRealtimeOptions-especially those related to padding-affect the results.

RecognizingEvents are a very important feature in real-time audio transcription.
If someone speaks a long sentence without pausing, the displayed text may not appear on the screen until a long time has passed.

However, the interval at which Partial Transcriptions trigger RecognizingEvents seems to have a significant impact on transcription speed.

There still seems to be room for improvement in terms of audio transcription speed.

When I created a sample project with Whisper.net and transcribed a wav file, I found that even with an integrated GPU supporting Vulkan, the speed was quite good (about 3-4x real-time).
For example, if the audio is 4 minutes long and the transcription (excluding model load time) takes about 1 minute, it can be said that the process is running at about 4x real-time speed.

In real-time audio transcription, the input audio buffer inevitably accumulates before processing.
Unlike wav file transcription, this means there will inevitably be some duplicate transcriptions.

If 3-4x real-time speed can be achieved with wav file transcription, I believe that, even accounting for duplicate processing, real-time audio transcription results should be possible. So, I modified the EchoSharp sample source and tested it for this purpose.

However, contrary to my expectations, it ran very slowly.

When CUDA acceleration is used on an Nvidia GPU, speeds of more than 20x real-time are achieved, so it worked quickly and well.
But on my laptop’s Vulkan Runtime, it was very slow.

With the Small model, the speed on my laptop’s Vulkan Runtime was relatively usable.
However, for non-English languages, the Small model’s error rate is too high to be practical, so in reality, large-v3-turbo must be used.

I thought that increasing the audio buffer size processed at once by changing MinTranscriptDuration in WhisperProcessor would improve recognition speed, but in practice, there was no significant difference.

With updates to Whisper.net (specifically Whisper.cpp), speed has been improving, and I believe that with further optimization of EchoSharp’s logic, sufficient speed should theoretically be achievable even on the integrated GPU of a modern laptop.