Skip to content

Feature: Add SenseVoice/FunASR as alternative STT backend — non-autoregressive, 5x faster #400

Description

@LauraGPT

Hi! VibeVoice is impressive — frontier voice AI with full pipeline.

I'd like to suggest SenseVoice from FunASR as an alternative STT option. It offers a fundamentally different architecture that avoids some issues seen with autoregressive ASR (like the "infinite loop" in #373):

Key differences

Autoregressive ASR SenseVoice (Non-autoregressive)
Decoding Sequential token generation Single forward pass
Failure mode Can loop/degenerate Deterministic output
Speed Grows with input length Constant time per segment
Params Varies 234M
RTF (GPU) ~0.1 ~0.006 (170x realtime)

The non-autoregressive architecture fundamentally cannot produce "infinite loop" outputs since it decodes the entire sequence in one step.

Features

  • 50+ languages with auto-detection
  • Emotion detection — happy/sad/angry/neutral
  • Audio event classification — laughter, music, applause
  • Built-in VAD (FSMN-VAD, 5.2M params) + punctuation restoration

Integration

from funasr import AutoModel

model = AutoModel(model="iic/SenseVoiceSmall", vad_model="fsmn-vad")
result = model.generate(input=audio_chunk)
# Returns: text + language + emotion + events

OpenAI-compatible server:

pip install funasr && funasr-server --device cuda
# POST /v1/audio/transcriptions

Links

This could serve as a lightweight, fast alternative for users who don't need the full VibeVoice-ASR model but want reliable, fast transcription.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions