feat(plugins): added new POC transcription node using Kroko ASR - #178
Merged
Conversation
Collaborator
|
Thank you for this contribution! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR provides a new processor node to Juturna, implementing transcriptions via Kroko ASR. It's meant mostly as a proof-of-concept, as I used this code to demonstrate how easy it would be to add a new node to Juturna in a Dangerous Demos session during Kamailio World.
PR type
Select all the labels that apply to this PR.
Key modifications and changes
As anticipated in the intro, this PR provides a new processor node to Juturna, implementing transcriptions via Kroko ASR. It's meant mostly as a proof-of-concept, as the code will definitely need some refactoring before it can actually be relied upon for more than a simple demo.
It leverages the WebSocket API that Kroko ASR exposes as a way to send audio samples and receive transcription data back. The implementation is currently naive, as WebSocket interaction is only performed contextually to
update()calls: any time samples come in viaupdate, they are sent to Kroko via WebSocket, after which the code tries to perform one or more WebSocket reads, until there's no more data to receive. This is supposed to ensure we get all transcription data that Kroko processed in the meanwhile, but is probably a suboptimal way to handle this, as:updatecall, andA queue and/or a separate thread for handling incoming data will probably be a better way to handle that.
Considering that the language. model and other settings are configured when launching the separate Kroko ASR binaries, these settings are not exposed in this node, as there would be no way to enforce them: this may be possible using the SDK APIs, but I haven't investigated this yet. The only argument that can currently be passed to the node is the WebSocket address to contact (
ws://localhost:6006by default). The node expects 16khz mono float samples, as that's what Kroko ASR requires.This was tested using the
source/audio_rtp_avnode feedingfltsamples, and using precompiled Kroko ASR binaries that Banafo graciously provided us with, as I had problems building the project from the repo myself. While Kroko ASR requires licensing to use pro models, there are community models that can be used for testing, and community licenses that allow for free limited testing, so that's what I used in my tests of the node.