Environment friendly and Adaptable Speech Enhancement by way of Pre-trained Generative Audioencoders and Vocoders

Current advances in speech enhancement (SE) have moved past conventional masks or sign prediction strategies, turning as an alternative to pre-trained audio fashions for richer, extra transferable options. These fashions, similar to WavLM, extract significant audio embeddings that improve the efficiency of SE. Some approaches use these embeddings to foretell masks or mix them with spectral information for higher accuracy. Others discover generative methods, utilizing neural vocoders to reconstruct clear speech instantly from noisy embeddings. Whereas efficient, these strategies usually contain freezing pre-trained fashions or require intensive fine-tuning, which limits adaptability and will increase computational prices, making switch to different duties harder. 

Researchers at MiLM Plus, Xiaomi Inc., current a light-weight and versatile SE technique that makes use of pre-trained fashions. First, audio embeddings are extracted from noisy speech utilizing a frozen audioencoder. These are then cleaned by a small denoise encoder and handed to a vocoder to generate clear speech. Not like task-specific fashions, each the audioencoder and vocoder are pre-trained individually, making the system adaptable to duties like dereverberation or separation. Experiments have proven that generative fashions outperform discriminative ones by way of speech high quality and speaker constancy. Regardless of its simplicity, the system is extremely environment friendly and even surpasses a number one SE mannequin in listening checks. 

The proposed speech enhancement system is split into three primary elements. First, noisy speech is handed by means of a pre-trained audioencoder, which generates noisy audio embeddings. A denoise encoder then refines these embeddings to supply cleaner variations, that are lastly transformed again into speech by a vocoder. Whereas the denoise encoder and vocoder are educated individually, they each depend on the identical frozen, pre-trained audioencoder. Throughout coaching, the denoise encoder minimizes the distinction between noisy and clear embeddings, each of that are generated in parallel from paired speech samples, utilizing a Imply Squared Error loss. This encoder is constructed utilizing a ViT structure with normal activation and normalization layers.

For the vocoder, coaching is finished in a self-supervised method utilizing clear speech information alone. The vocoder learns to reconstruct speech waveforms from audio embeddings by predicting Fourier spectral coefficients, that are transformed again to audio by means of the inverse short-time Fourier rework. It adopts a barely modified model of the Vocos framework, tailor-made to accommodate numerous audioencoders. A Generative Adversarial Community (GAN) setup is employed, the place the generator is predicated on ConvNeXt, and the discriminators embrace each multi-period and multi-resolution varieties. The coaching additionally incorporates adversarial, reconstruction, and have matching losses. Importantly, all through the method, the audioencoder stays unchanged, utilizing weights from publicly obtainable fashions. 

The analysis demonstrated that generative audioencoders, similar to Dasheng, persistently outperformed discriminative ones. On the DNS1 dataset, Dasheng achieved a speaker similarity rating of 0.881, whereas WavLM and Whisper scored 0.486 and 0.489, respectively. When it comes to speech high quality, non-intrusive metrics like DNSMOS and NISQAv2 indicated notable enhancements, even with smaller denoise encoders. For example, ViT3 reached a DNSMOS of 4.03 and a NISQAv2 rating of 4.41. Subjective listening checks involving 17 individuals confirmed that Dasheng produced a Imply Opinion Rating (MOS) of three.87, surpassing Demucs at 3.11 and LMS at 2.98, highlighting its robust perceptual efficiency. 

In conclusion, the research presents a sensible and adaptable speech enhancement system that depends on pre-trained generative audioencoders and vocoders, avoiding the necessity for full mannequin fine-tuning. By denoising audio embeddings utilizing a light-weight encoder and reconstructing speech with a pre-trained vocoder, the system achieves each computational effectivity and robust efficiency. Evaluations present that generative audioencoders considerably outperform discriminative ones by way of speech high quality and speaker constancy. The compact denoise encoder maintains excessive perceptual high quality even with fewer parameters. Subjective listening checks additional verify that this technique delivers higher perceptual readability than an current state-of-the-art mannequin, highlighting its effectiveness and flexibility. 

Try the Paper and GitHub Web page. All credit score for this analysis goes to the researchers of this mission. Prepared to attach with 1 Million+ AI Devs/Engineers/Researchers? See how NVIDIA, LG AI Analysis, and high AI firms leverage MarkTechPost to succeed in their audience [Learn More]

Sana Hassan, a consulting intern at Marktechpost and dual-degree scholar at IIT Madras, is captivated with making use of know-how and AI to handle real-world challenges. With a eager curiosity in fixing sensible issues, he brings a recent perspective to the intersection of AI and real-life options.