Simple and Effective Unsupervised Speech Synthesis

Authors: Alexander H. Liu and Cheng-I Jeff Lai (equal contribution)

Abstract:
We introduce the first unsupervised speech synthesis system based on a simple, yet effective recipe. The framework leverages recent work in unsupervised speech recognition as well as existing neural-based speech synthesis. Using only unlabeled speech audio and unlabeled text as well as a lexicon, our method enables speech synthesis without the need for a human-labeled corpus. Experiments demonstrate the unsupervised system can synthesize speech similar to a supervised counterpart in terms of naturalness and intelligibility measured by human evaluation.

References: arxiv, TTS code

We recommend loading the demo pages with Chrome since Safari sometimes freezes during loading.

Unsupervised TTS on Single-Speaker LJspeech

Train end-to-end TTS entirely based on psuedo-labels from wav2vec-U 2.0. The TTS model is based on Transformer-TTS and the vocoder is based on Parallel WaveGAN.
We also provide detailed demo for unsupervised Tacotron2 models and more variants of Unsupervised Transformer-TTS models.
Additional modifications are needed to make the TTS training work. Please refer to the paper for details.

ID Text Original Supervised Unsupervised
LJ026-0086 Text:
the kinetic energy of sunlight was expended in lifting the atoms into position of vantage, thus endowing them with energy of position.

 Raw:
 Raw:
 Raw:
LJ039-0017 Text:
I asked him where he was going, and why he was getting dressed. He answered Nixon is coming. I want to go and have a look, end quote.

 Raw:
 Raw:
 Raw:
LJ046-0236 Text:
none of the cases in the PRS general files was available for systematic review on a geographic basis when the President planned a particular trip.

 Raw:
 Raw:
 Raw:
LJ012-0288 Text:
and was seen to be busily engaged in washing down the house with bucket and mop.

 Raw:
 Raw:
 Raw:
LJ028-0426 Text:
For fifteen years Dr. Koldewey and his assistants, with a force of two hundred native workmen, have labored there winter and summer.

 Raw:
 Raw:
 Raw:
LJ017-0209 Text:
more particularly as regards locomotion.

 Raw:
 Raw:
 Raw:
LJ018-0333 Text:
it was the principal incentive with Kate Webster, that fierce and brutal female savage who took the life of her mistress at Richmond.

 Raw:
 Raw:
 Raw:
LJ006-0224 Text:
New arrivals, especially the innocent and still guileless debutant, were tormented with rude horse-play, and assailed by the most insulting "chaff."

 Raw:
 Raw:
 Raw:
LJ023-0063 Text:
But the framers went further.

 Raw:
 Raw:
 Raw:
LJ036-0058 Text:
People on the bus began talking about it. As the bus neared Lamar Street, Oswald left the bus and disappeared into the crowd.

 Raw:
 Raw:
 Raw:
LJ042-0125 Text:
I am increasingly aware of the presence, in all thing, of Lebizen, shop party secretary, fat, fortyish, and jovial on the outside.

 Raw:
 Raw:
 Raw:
LJ043-0115 Text:
He lost his job on July 19, 1963, because his work was not satisfactory

 Raw:
 Raw:
 Raw:
LJ042-0084 Text:
referring to them as experiences that he did not intend to have himself, even though he stated that he had never held a civilian job.

 Raw:
 Raw:
 Raw:
LJ048-0274 Text:
However, he felt that any disciplinary action might have given rise

 Raw:
 Raw:
 Raw:
LJ022-0047 Text:
However, for the first time in five years the relief rolls have declined instead of increased during the winter months.

 Raw:
 Raw:
 Raw:
LJ048-0261 Text:
a. Employees are strictly enjoined to refrain from the use of intoxicating liquor

 Raw:
 Raw:
 Raw:
LJ012-0259 Text:
who had been missing since the previous Christmas Day.

 Raw:
 Raw:
 Raw:
LJ005-0067 Text:
The Prison Society reproves the misdirected efforts of ambitious architects, who by a lavish and improvident expenditure of public money

 Raw:
 Raw:
 Raw:
LJ037-0199 Text:
The mail-order form had a line for the name of a witness to atalex_test that the person ordering the gun was a U.S. citizen and had not been convicted of a felony.

 Raw:
 Raw:
 Raw:
LJ022-0047 Text:
However, for the first time in five years the relief rolls have declined instead of increased during the winter months.

 Raw:
 Raw:
 Raw:

Unsupervised TTS on Multi-Speaker LibriTTS

Similar to the single-speaker setup, but with the following modifications, due to the difficulty of (1) modeling over 2k speakers in a single model, and (2) LibriTTS is noiser than LJspeech:
  1. A separate ECAPA-TDNN for speaker modeling, and Global Style Tokens for style modeling.
  2. Vocoder is HiFi-GAN. We found its synthesis sounds crisper and louder compared to Parallel WaveGAN and MelGAN.
  3. Target speaker finetuning after multi-speaker TTS training. This further improves speaker similarity.
We also provide detailed demo for more speaker synthesis (including the bad ones) here.
Speaker modeling is not the focus of this work, and therefore some synthesis have poor speaker modeling. This could be easily improved with a better speaker encode (ref). We leave this aspect for future study.

ID Text Original Speaker 4535
Supervised		 Speaker 4535
Unsupervised		 Speaker 8699
Supervised		 Speaker 8699
Unsupervised
4446_2273_000040_000003 Text:
But he's given me some good Irish songs.

 Raw:
 Raw:
 Raw:
 Raw:
 Raw:
7176_88083_000022_000001 Text:
He saw the frantic fugitive dive over the ledge and disappear.

 Raw:
 Raw:
 Raw:
 Raw:
 Raw:
7729_102255_000020_000006 Text:
He ordered the forces to disband; prisoners were liberated, and with the opportune aid of a furious rain storm the Border Ruffian army gradually melted away.

 Raw:
 Raw:
 Raw:
 Raw:
 Raw:
8555_284447_000033_000002 Text:
But I'm the Ruler here, and I'll do as I please.

 Raw:
 Raw:
 Raw:
 Raw:
 Raw:
2961_961_000009_000002 Text:
Now he who is the best could only create the fairest; and reflecting that of visible things the intelligent is superior to the unintelligent, he put intelligence in soul and soul in body, and framed the universe to be the best and fairest work in the order of nature, and the world became a living soul through the providence of God.

 Raw:
 Raw:
 Raw:
 Raw:
 Raw:
8555_284449_000017_000000 Text:
"Pinkies," said she, "your Queen has conquered the Boolooroo and is now the Queen of the Blues.

 Raw:
 Raw:
 Raw:
 Raw:
 Raw:
7729_102255_000035_000001 Text:
He continued his pretended search and, to give color to his errand, made two arrests.

 Raw:
 Raw:
 Raw:
 Raw:
 Raw:
1089_134686_000009_000008 Text:
What did it avail to pray when he knew that his soul lusted after its own destruction?

 Raw:
 Raw:
 Raw:
 Raw:
 Raw:
2300_131720_000012_000002 Text:
Of these two Brockton required the larger plant, but with the conductors placed underground. It was the first to complete its arrangements and close its contract. mr Henry Villard, it will be remembered, had married the daughter of Garrison, the famous abolitionist, and it was through his relationship with the Garrison family that Brockton came to have the honor of exemplifying so soon the principles of an entirely new art.

 Raw:
 Raw:
 Raw:
 Raw:
 Raw:
5683_32866_000035_000001 Text:
I assure you, upon my honour, I've no secret.

 Raw:
 Raw:
 Raw:
 Raw:
 Raw:
4446_2273_000037_000002 Text:
Have I told you about my new play?

 Raw:
 Raw:
 Raw:
 Raw:
 Raw:
7176_88083_000015_000003 Text:
With only head and wings above the mad smother, he flapped onward frantically, beating down the foam about him.

 Raw:
 Raw:
 Raw:
 Raw:
 Raw:
1580_141084_000054_000000 Text:
"I have told you everything, sir."

 Raw:
 Raw:
 Raw:
 Raw:
 Raw:
2830_3980_000026_000005 Text:
The most they could claim is that they were sent by others.

 Raw:
 Raw:
 Raw:
 Raw:
 Raw:
1995_1836_000041_000000 Text:
"I thought the only excuse for fighting was a great Right; if Right is on neither side or simultaneously on both, then War is not only Hell but Damnation."

 Raw:
 Raw:
 Raw:
 Raw:
 Raw:
237_134500_000007_000003 Text:
Please leave the roses till last, if you must cut them.

 Raw:
 Raw:
 Raw:
 Raw:
 Raw:
260_123288_000003_000002 Text:
The roarings become lost in the distance.

 Raw:
 Raw:
 Raw:
 Raw:
 Raw:
7729_102255_000021_000004 Text:
The general public interest in the controversy was shown by discussions carried on by press, pulpit, and in the daily conversation and comment of the people of the Union in every town, hamlet, and neighborhood.

 Raw:
 Raw:
 Raw:
 Raw:
 Raw:
1089_134686_000032_000008 Text:
A great saint, saint Francis Xavier!

 Raw:
 Raw:
 Raw:
 Raw:
 Raw:
4446_2275_000040_000002 Text:
"It's got to be a clean break, Hilda.

 Raw:
 Raw:
 Raw:
 Raw:
 Raw: