The evolution of voice AI systems has confronted numerous challenges, with traditional pipeline architectures embodying notable constraints such as high latency, the erosion of vocal nuances, and mechanical, reactive interactions. While these structures sufficiently accommodate basic voice-driven interfaces, the objective of achieving truly autonomous interactions akin to natural human communication has remained largely unmet. Voila, a pioneering family of large audio-language foundation models, is introduced to transcend these limitations. By leveraging an innovative end-to-end model design, Voila targets the dual objectives of facilitating real-time, autonomous, and flexible voice interactions while preserving rich vocal details. Voila's novel hierarchical Transformer architecture integrates streaming audio encoding with multitier audio generators, achieving a high-fidelity audio processing experience that minimizes latency to 195 ms—surpassing the average human response time. Additionally, Voila intelligently melds the voice and language modeling capabilities inherent in LLMs, empowering users with customizable voice and persona-driven interactions. This framework not only retains linguistic proficiency and vast knowledge but also incorporates millions of pre-built and customizable voices to enhance user engagement.
Moreover, Voila is established as a unified model adept in various audio tasks beyond spoken dialogue, including Automatic Speech Recognition (ASR), Text-to-Speech (TTS), and adaptations such as speech translation. Capable of supporting six languages and enriched by extensive multilingual training data, Voila embodies a significant step forward in transforming machine interactions from passive exchanges into seamless, proactive dialogues. The deployment of Voila's web demo promises broad user accessibility and invites further exploration into redefining the dynamics of human-AI communication.
Voila designs a hierarchical Transformer architecture, including streaming audio encoding and tokenization, and multi-scale Transformers consisting of an LLM backbone and a hierarchical audio generator. The models are trained in an end-to-end way with extensive audio-text data. The key advancements are:
We release the model in Hugging face and encourage the readers to try our web demo.
Experience Voila's capabilities through these interactive demonstrations
Show smooth voice transitions
Switch between multiple voices during the dialogue.
The video shows rich emotions in the conversation (timbre, intonation, speaking speed, modal particles).
This video brings humorous and light-hearted dialogue.
@article{voila2025,
author = {Yemin Shi, Yu Shu, Siwei Dong, Guangyi Liu, Jaward Sesay, Jingwen Li, Zhiting Hu},
title = {Voila: Voice-Language Foundation Models for Real-Time Autonomous Interaction and Voice Roleplay},
eprint={1111.11111},
archivePrefix={arXiv},
primaryClass={cs.CL},
year = {2025}
}