M5‑TTS is an open‑source lightweight multilingual & multi‑dialect Text‑to‑Speech (TTS) model developed by ASLP@NPU (Audio, Speech and Language Processing Lab), Northwestern Polytechnical University, a well‑known speech research team in China. With small‑size model weights, it supports local CPU‑based inference. It covers multilingual synthesis, Chinese multi‑dialect generation and zero‑shot voice cloning. Model checkpoints and inference source code are fully open‑sourced for academic research and developer customization.
✅ Supported Languages & Dialects
Foreign Languages
- English
- German
- French
- Japanese
- Korean
Chinese and Chinese Dialects
- Mandarin
- Cantonese
- Sichuanese
Note: These languages and dialects are explicitly covered in the training corpus and official evaluation. Only Cantonese and Sichuanese are formally benchmarked in the paper. Other Chinese dialects are not officially tested and quality is not guaranteed. Cross‑lingual speaker identity preservation is supported: after cloning a voice from a reference audio, you can use the same voice to generate speech in any of the supported languages or dialects above.
Core Functions & Key Features
- Light‑weight design with CPU streaming inference: The language‑model part has only around 50M parameters. It can run streaming audio generation on regular CPUs without high‑end GPUs. It achieves low first‑chunk latency, suitable for local‑hosted and edge‑device deployment, no cloud API dependency required.
- Multilingual and Chinese multi‑dialect support: It supports the foreign languages listed above plus Mandarin, Cantonese and Sichuanese. It preserves the target speaker’s voice identity across different languages and dialects within one single model.
- Zero‑shot voice cloning: Given a short reference audio clip, the model can replicate the speaker’s timbre without large‑scale training data for that speaker.
- High text fidelity: It reduces mispronunciation and text omission in synthesized speech, delivering relatively low character‑error rates when reading input text.
Application Scenarios
- Off‑line local voice assistants & edge‑device voice broadcasting
- Audiobook & audio content creation for multiple languages and dialects
- Speech output module for AI conversational agents
- TTS‑related academic research, model fine‑tuning and algorithm benchmarking
Underlying Technologies
- Architecture: Temporal backbone network combined with Multi‑Token Prediction module, modeling speech tokens over 16 codebooks.
- Post‑training: GRPO group‑wise reward optimization with CER (Character Error Rate) guided local feedback, improving speech naturalness and text correctness.
- Pipeline: Text is mapped to discrete speech tokens, then decoded to raw audio waveform via an external speech codec, implementing end‑to‑end TTS generation.