Spectrum-aware neural vocoder based on self-supervised learning for speech enhancement

We consider a speech enhancement setup where neural speech embeddings, obtained from pre-trained self-supervised learning (SSL) models applied to the noisy signal, are subsequently input to a neural vocoder to synthesize the underlying clean speech. The key innovation is in enhancing these latent neural embeddings to mitigate the distortions due to noise and reverberation, resulting in a superior quality of the synthesized signal. Separating the task, thus, into distinct embedding enhancement and speech generation phases permits increased flexibility in network design. We further explore the benefit of combining the hidden states from the SSL model in a learnable manner, to produce a more robust embedding as the vocoder input. Finally, we also investigate different loss functions for training the neural vocoder. Experimental results validate the effectiveness of our proposed approach, particularly in scenarios characterized by concurrent background noise and reverberation.