|
| 1 | +# Contributing to SenseVoice |
| 2 | + |
| 3 | +Thank you for your interest in contributing to SenseVoice! This guide will help you get started. |
| 4 | + |
| 5 | +## Getting Started |
| 6 | + |
| 7 | +### Prerequisites |
| 8 | + |
| 9 | +- Python 3.8 or higher |
| 10 | +- Git |
| 11 | +- (Optional) CUDA-compatible GPU for faster inference |
| 12 | + |
| 13 | +### Setting Up the Development Environment |
| 14 | + |
| 15 | +1. **Fork and clone the repository:** |
| 16 | + |
| 17 | +```bash |
| 18 | +git clone https://github.com/<your-username>/SenseVoice.git |
| 19 | +cd SenseVoice |
| 20 | +``` |
| 21 | + |
| 22 | +2. **Create a virtual environment:** |
| 23 | + |
| 24 | +```bash |
| 25 | +python3 -m venv venv |
| 26 | +source venv/bin/activate # On Windows: venv\Scripts\activate |
| 27 | +``` |
| 28 | + |
| 29 | +3. **Install dependencies:** |
| 30 | + |
| 31 | +```bash |
| 32 | +pip install -r requirements.txt |
| 33 | +``` |
| 34 | + |
| 35 | +4. **Verify the installation:** |
| 36 | + |
| 37 | +```bash |
| 38 | +python -c "from funasr import AutoModel; print('Installation successful')" |
| 39 | +``` |
| 40 | + |
| 41 | +### Running on CPU |
| 42 | + |
| 43 | +If you don't have a GPU, you can run SenseVoice on CPU by setting the device to `"cpu"`: |
| 44 | + |
| 45 | +```python |
| 46 | +model = AutoModel( |
| 47 | + model="iic/SenseVoiceSmall", |
| 48 | + trust_remote_code=True, |
| 49 | + device="cpu", |
| 50 | +) |
| 51 | +``` |
| 52 | + |
| 53 | +For the FastAPI server: |
| 54 | + |
| 55 | +```bash |
| 56 | +export SENSEVOICE_DEVICE=cpu |
| 57 | +fastapi run --port 50000 |
| 58 | +``` |
| 59 | + |
| 60 | +## How to Contribute |
| 61 | + |
| 62 | +### Reporting Bugs |
| 63 | + |
| 64 | +1. Search [existing issues](https://github.com/FunAudioLLM/SenseVoice/issues) to avoid duplicates. |
| 65 | +2. Use the [Bug Report template](https://github.com/FunAudioLLM/SenseVoice/issues/new?template=bug_report.md). |
| 66 | +3. Include your environment details (OS, Python version, PyTorch version, GPU, CUDA version). |
| 67 | +4. Provide a minimal code sample to reproduce the issue. |
| 68 | + |
| 69 | +### Submitting Pull Requests |
| 70 | + |
| 71 | +1. **Fork the repository** and create a new branch from `main`: |
| 72 | + |
| 73 | +```bash |
| 74 | +git checkout -b your-branch-name |
| 75 | +``` |
| 76 | + |
| 77 | +2. **Make your changes.** Keep commits focused and atomic. |
| 78 | + |
| 79 | +3. **Test your changes** to make sure nothing is broken. |
| 80 | + |
| 81 | +4. **Push to your fork** and open a Pull Request against `FunAudioLLM/SenseVoice:main`. |
| 82 | + |
| 83 | +5. **Describe your changes** clearly in the PR description. Explain *what* changed and *why*. |
| 84 | + |
| 85 | +### Types of Contributions We Welcome |
| 86 | + |
| 87 | +- **Bug fixes** — Check [open issues labeled `bug`](https://github.com/FunAudioLLM/SenseVoice/issues?q=is%3Aissue+is%3Aopen+label%3Abug) for known problems. |
| 88 | +- **Documentation improvements** — Typo fixes, clarifications, additional examples, translations. |
| 89 | +- **New examples** — Demo scripts showing different use cases (emotion detection, event detection, multilingual transcription). |
| 90 | +- **Performance improvements** — Optimizations for inference speed or memory usage. |
| 91 | +- **Test coverage** — Unit tests and integration tests are very welcome. |
| 92 | + |
| 93 | +### Code Style |
| 94 | + |
| 95 | +- Follow existing code patterns and conventions in the repository. |
| 96 | +- Use type hints where applicable. |
| 97 | +- Add docstrings to new functions and classes. |
| 98 | +- Keep lines under 120 characters. |
| 99 | + |
| 100 | +## Project Structure |
| 101 | + |
| 102 | +``` |
| 103 | +SenseVoice/ |
| 104 | +├── model.py # Core SenseVoiceSmall model (encoder, CTC decoder, emotion/event embeddings) |
| 105 | +├── api.py # FastAPI server for inference |
| 106 | +├── webui.py # Gradio web interface |
| 107 | +├── demo1.py # Inference example using FunASR AutoModel |
| 108 | +├── demo2.py # Direct model inference with timestamp support |
| 109 | +├── export.py # ONNX model export |
| 110 | +├── export_meta.py # Export utilities for model rebuilding |
| 111 | +├── finetune.sh # Fine-tuning script with DeepSpeed support |
| 112 | +├── requirements.txt # Python dependencies |
| 113 | +├── Dockerfile # Docker build configuration |
| 114 | +├── data/ # Example training and validation data |
| 115 | +├── utils/ # Utilities (frontend, ONNX inference, CTC alignment, export) |
| 116 | +└── image/ # Documentation images |
| 117 | +``` |
| 118 | + |
| 119 | +## Understanding the Model |
| 120 | + |
| 121 | +SenseVoice is a non-autoregressive encoder-only model that outputs: |
| 122 | + |
| 123 | +- **Speech transcription** (ASR) across 50+ languages |
| 124 | +- **Emotion labels**: `HAPPY`, `SAD`, `ANGRY`, `NEUTRAL`, `FEARFUL`, `DISGUSTED`, `SURPRISED` |
| 125 | +- **Audio event labels**: `BGM`, `Speech`, `Applause`, `Laughter`, `Cry`, `Sneeze`, `Breath`, `Cough` |
| 126 | +- **Language identification** for Mandarin, English, Cantonese, Japanese, and Korean |
| 127 | + |
| 128 | +The model uses a SANM (Self-Attention with Normalized Memory) encoder architecture with CTC decoding. Emotion and event labels are predicted from the first 4 encoder output tokens, while the remaining tokens produce the transcription. |
| 129 | + |
| 130 | +## Docker |
| 131 | + |
| 132 | +You can also run SenseVoice using Docker: |
| 133 | + |
| 134 | +```bash |
| 135 | +# Build |
| 136 | +docker build -t sensevoice . |
| 137 | + |
| 138 | +# Run with GPU |
| 139 | +docker run --gpus all -p 50000:50000 sensevoice |
| 140 | + |
| 141 | +# Run on CPU |
| 142 | +docker run -e SENSEVOICE_DEVICE=cpu -p 50000:50000 sensevoice |
| 143 | +``` |
| 144 | + |
| 145 | +## Questions? |
| 146 | + |
| 147 | +- Open an issue using the [Questions template](https://github.com/FunAudioLLM/SenseVoice/issues/new?template=ask_questions.md). |
| 148 | +- Join the community via the DingTalk group (see [README](./README.md#community)). |
| 149 | + |
| 150 | +## License |
| 151 | + |
| 152 | +By contributing to SenseVoice, you agree that your contributions will be licensed under the same license as the project. See [FunASR License](https://github.com/modelscope/FunASR?tab=readme-ov-file#license) for details. |
0 commit comments