Tencent AI Open Sources Covo-Audio: A 7B Speech Language Mannequin and Inference Pipeline for Actual-Time Audio Conversations and Reasoning

Tencent AI Lab has launched Covo-Audio, a 7B-parameter end-to-end Massive Audio Language Mannequin (LALM). The mannequin is designed to unify speech processing and language intelligence by instantly processing steady audio inputs and producing audio outputs inside a single structure.

System Structure

The Covo-Audio framework consists of 4 major elements designed for seamless cross-modal interplay:

• Audio Encoder: The mannequin makes use of Whisper-large-v3 as its major encoder because of its robustness towards background noise and assorted accents. This part operates at a body fee of 50 Hz.
• Audio Adapter: To bridge the encoder and the LLM, a specialised adapter employs three downsampling modules, integrating linear and convolution layers to cut back the body fee from 50 Hz to six.25 Hz.
• LLM Spine: The system is constructed upon Qwen2.5-7B-Base, which has been tailored to course of interleaved sequences of steady acoustic options and textual tokens.
• Speech Tokenizer and Decoder: The tokenizer, based mostly on WavLM-large, makes use of a codebook dimension of 16,384 to provide discrete audio tokens at 25 Hz. The decoder employs a Move-Matching (FM) based mostly framework and a BigVGAN vocoder to reconstruct high-fidelity 24K waveforms.

Hierarchical Tri-modal Interleaving

A core contribution of this work is the Hierarchical Tri-modal Speech-Textual content Interleaving technique. In contrast to conventional strategies that function solely on the phrase or character degree, this framework aligns steady acoustic options $(a_c)$, discrete speech tokens $(a_d)$, and pure language textual content $(t)$.

The mannequin makes use of two major patterns:

1. Sequential Interleaving $(a_c rightarrow t rightarrow a_d)$: Steady options, textual content, and discrete tokens are organized in a progressive chain.
2. Parallel Integration $(a_c rightarrow t | a_d)$: Steady options are aligned with a coupled text-discrete unit.

The hierarchical side ensures structural coherence through the use of phrase-level interleaving for fine-grained alignment and sentence-level interleaving to protect world semantic integrity in long-form utterances^{. The coaching course of concerned a two-stage pre-training pipeline processing a complete of 2T tokens.}

Intelligence-Speaker Decoupling

To mitigate the excessive price of establishing large-scale dialogue knowledge for particular audio system, the analysis crew proposed an Intelligence Speaker Decoupling technique. This system separates dialogue intelligence from voice rendering, permitting for versatile voice customization utilizing minimal text-to-speech (TTS) knowledge.

The strategy reformats high-quality TTS recordings into pseudo-conversations with masked textual content loss^{. By excluding the textual content response portion from the loss calculation, the mannequin preserves its reasoning skills whereas inheriting the naturalness of the TTS speaker. This permits personalised interplay with out the necessity for in depth, speaker-specific dialogue datasets.}

Full-Duplex Voice Interplay

Covo-Audio developed into Covo-Audio-Chat-FD, a variant able to simultaneous dual-stream communication^{. The audio encoder is reformatted right into a chunk-streaming method, and the consumer and mannequin streams are chunk-interleaved in a 1:4 ratio. Every chunk represents 0.16s of audio.}

The system manages conversational states via particular architectural tokens:

• THINK Token: Signifies a listening-only state whereas the mannequin waits to reply.
• SHIFT Token: Signifies the transition to the mannequin’s talking flip.
• BREAK Token: Detects interruption indicators (barge-ins), triggering the mannequin to terminate talking instantly and swap again to listening.

For multi-turn situations, the mannequin implements a recursive context-filling technique, the place steady audio options from consumer enter and generated tokens from earlier turns are prefixed as historic context^.

Audio Reasoning and Reinforcement Studying

To boost advanced reasoning, the mannequin incorporates Chain-of-Thought (CoT) reasoning and Group Relative Coverage Optimization (GRPO). The mannequin is optimized utilizing a verifiable composite reward perform:

$$R_{complete} = R_{accuracy} + R_{format} + R_{consistency} + R_{considering}$$

This construction permits the mannequin to optimize for correctness $(R_{accuracy})$, structured output adherence $(R_{format})$, logical coherence $(R_{consistency})$, and reasoning depth $(R_{considering})$.

Analysis and Efficiency

Covo-Audio (7B) reveals aggressive or superior outcomes on a number of evaluated benchmarks, with strongest claims made for fashions of comparable scale and chosen speech/audio duties. ^{On the MMAU benchmark, it achieved a mean rating of 75.30%, the best amongst evaluated 7B-scale fashions. It notably excelled in music understanding with a rating of 76.05%. On the MMSU benchmark, Covo-Audio achieved a number one 66.64% common accuracy.}

Concerning its conversational variants, Covo-Audio-Chat demonstrated robust efficiency on URO-Bench, significantly in speech reasoning and spoken dialogue duties, outperforming fashions like Qwen3-Omni on the Chinese language monitor^{. For empathetic interplay on the VStyle benchmark, it achieved state-of-the-art ends in Mandarin for anger (4.89), unhappiness (4.93), and anxiousness (5.00).}

The analysis crew notes an ‘early-response’ situation on the GaokaoEval full-duplex setting, the place unusually lengthy silent pauses between vocal fragments could cause untimely responses. This ‘early-response’ habits correlates with the mannequin’s pause-handling success metric and is recognized as a important path for future optimization.

Key Takeaways

• Unified Finish-to-Finish Structure: Covo-Audio is a 7B-parameter mannequin that natively processes steady audio inputs and generates high-fidelity audio outputs inside a single, unified structure. It eliminates the necessity for cascaded ASR-LLM-TTS pipelines, lowering error propagation and data loss.
• Hierarchical Tri-modal Interleaving: The mannequin employs a specialised technique to align steady acoustic options, discrete speech tokens, and pure language textual content. By interleaving these modalities at each phrase and sentence ranges, it preserves world semantic integrity whereas capturing fine-grained prosodic nuances.
• Intelligence-Speaker Decoupling: Tencent analysis crew introduces a way to decouple dialogue intelligence from particular voice rendering. This enables for versatile voice customization utilizing light-weight Textual content-to-Speech (TTS) knowledge, considerably reducing the price of creating personalised conversational brokers.
• Native Full-Duplex Interplay: The Covo-Audio-Chat-FD variant helps simultaneous listening and talking. It makes use of particular architectural tokens—THINK, SHIFT, and BREAK—to handle advanced real-time dynamics similar to easy turn-taking, backchanneling, and consumer barge-ins.
• Superior Parameter Effectivity: Regardless of its compact 7B scale, Covo-Audio achieves state-of-the-art or extremely aggressive efficiency throughout core benchmarks, together with MMAU, MMSU, and URO-Bench. It ceaselessly matches or exceeds the efficiency of a lot bigger methods, similar to 32B-parameter fashions, in audio and speech understanding duties.

Take a look at the Paper, Mannequin on HF and Repo. Additionally, be at liberty to comply with us on Twitter and don’t neglect to affix our 120k+ ML SubReddit and Subscribe to our Publication. Wait! are you on telegram? now you may be part of us on telegram as properly.

Michal Sutter is a knowledge science skilled with a Grasp of Science in Knowledge Science from the College of Padova. With a strong basis in statistical evaluation, machine studying, and knowledge engineering, Michal excels at reworking advanced datasets into actionable insights.