Artificial Intelligence (AI) advancements have significantly evolved voice interaction technology with the primary goal to make the interaction between humans and machines more intuitive and human-like. Recent developments have led to the attainment of high-precision speech recognition, emotion detection, and natural speech generation. Despite these advancements, voice interaction needs to improve latency, multilingual support, and contextually appropriate speech to become more people-friendly and mainstream.
Voice interaction is an ongoing field of research with existing models like speech recognition tools and traditional models for emotion detection. However, these models often fall short when it comes to delivering low-latency, high-precision and emotionally expressive interactions across multiple languages, thus, the need for robust and comprehensive solutions to handle these challenges more efficiently are growing increasingly evident.
Alibaba Group, in the quest to push the boundaries of voice interaction technology, introduced FunAudioLLM, comprising two core models, SenseVoice and CosyVoice. SenseVoice is responsible for multilingual speech recognition, emotion recognition, and audio event detection and supports over 50 languages. CosyVoice, on the other hand, focuses on natural speech generation controlling factors like language, timbre, speaking style, and speaker identity.
FunAudioLLM is designed on advanced architectures for SenseVoice and CosyVoice models. SenseVoice-Small uses a non-autoregressive model for fast speech recognition in five languages and is more than fifteen times faster than its counterparts. SenseVoice-Large supports speech recognition in over 50 languages, providing high precision on complex tasks like emotion recognition and audio event detection. CosyVoice leverages supervised semantic speech tokens for natural and emotionally expressive voice generation. It supports zero-shot learning and cross-lingual voice cloning.
In terms of performance, FunAudioLLM shows significant improvements over existing models. SenseVoice delivers faster and more accurate speech recognition than the competition. In specific, SenseVoice-Small offers a recognition latency of less than 80ms, outperforming its counterparts. SenseVoice-Large ensures high-precision automatic speech recognition with word error reduction by more than 20% in multiple languages compared to the existing technology. CosyVoice is excellent in generating multilingual voices tailored to individual speakers and it manages to keep error rates low with a word error rate of less than 2%.
In conclusion, Alibaba Group demonstrated that FunAudioLLM has practical applications like speech-to-speech translation, enabling users to speak in foreign languages using their voice, emotional voice chat, interactive podcasts and expressive audiobook narration. The integration of SenseVoice and CosyVoice has allowed these capabilities and showcases the potential of FunAudioLLM in advancing voice interaction technology.
