Reconstructing the Vocal Tract with Differentiable Acoustic Simulation

Supplementary Videos

Autoencoding Results

We encode speech into cross sectional areas of the vocal tract, then decode the cross sections to speech.

Autoencoding 1:

“On November 8, 1963”

Ground Truth

Reconstructed Audio and Cross-Sections

Autoencoding 2:

“It was the discovery of this flaw in the banking sys…”

Ground Truth

Reconstructed Audio and Cross-Sections


Autoencoding 3:

“He cried out, quote, ‘Oh no'"

Ground Truth

Reconstructed Audio and Cross-Sections

Autoencoding 4:

“It sought to obtain and diffuse useful informa…”

Ground Truth

Reconstructed Audio and Cross-Sections

Singing Synthesis

To highlight the expressivity of the model, we show how we can use our simulator to reconstruct popular songs. The area functions are parameterized by a neural field.

Video 1 (English Pop Song Synthesis) . A reconstruction of a popular English language song.

Sampling Rate: 32000 Hz

Original Source: https://www.youtube.com/watch?v=dQw4w9WgXcQ

Transcript: “We’re no strangers to love. You know the rules, and so do I.”

Video 3 (English Pop Song Synthesis). A reconstruction of “Hey Jude” by the Beatles

Sampling Rate: 3200 Hz

Original Source: https://www.youtube.com/watch?v=tmoil_ZEorg

Transcript: “Hey Jude, don’t make it bad. Take a sad song, and make it better.”

Video 4 (K-pop Synthesis): A reconstruction of the beginning of “OMG” by New Jeans.

Sampling Rate: 44100 Hz

Original Source: https://www.youtube.com/watch?v=BH1o66wl8qE

Transcript: “이 노래는 (this song) it’s about you baby… Only you… You, you, you, you, you, you, you.”

Generalization to Out of Distribution Sounds

Video 5 (Fitting a Piano). The acoustic simulator can even fit to out of distribution sounds like a piano, but still maintain a human timbre. For example, when fitting “Ode to Joy”, the model still sounds distinctly human, validating that the model’s physics are constrained to human voices.

Original Source: https://www.youtube.com/watch?v=4YV4B_qxl78

Sampling Rate: 32000 Hz

Visualizing MRI and cross-sectional areas

Video 6: We can synthesize speech on MRI volumes by segmenting the cross-sectional areas from the vocal tract. This allows us to fit a MRI video to a target utterance. The speech sample is from the LJSpeech dataset.

Transcript

“The problem is aggravated by the necessity”

Ground Truth

Area Function Reconstruction (SR: 16000 Hz)

MRI Reconstruction (SR: 16000 Hz)

MRI Comparison with Speech2rtMRI

We evaluate our MRI reconstruct technique on examples from the USC 75-Speaker Speech MRI dataset. Background noise from the MRI machine is removed with Adobe Express’s speech enhancement tool. By “analysis by synthesis”, our technique can reconstruct both a MRI video of the speaker, and the speech synthesized from our acoustic simulator. On the other hand, the SOTA MRI reconstruction method Speech2rtMRI [Nguyen et al., 2025] cannot synthesize audio. Speech2rtMRI suffers from significant degradation because it is a video diffusion model with no acoustic grounding for the human vocal tract. Video diffusion models are known to struggle when generating long sequences. Meanwhile, our method can faithfully reconstruct the MRIs. One major difficulty when solving this problem is that the mapping between vocal tract geometry to speech is not one-to-one. Multiple vocal tract shapes can product the same sound, causing ambiguity in the generated MRIs.

Audio is sampled at 24000 Hz.


Transcript

MRI 1: “The best place I’ve been to as I was saying is uh, New York”

Ground Truth

Speech2rtMRI (No Audio)

Our Reconstructed MRI and Audio


Transcript

MRI 2: “So, my favorite music right now…”

Ground Truth

Speech2rtMRI (No Audio)

Our Reconstructed MRI and Audio


Transcript

MRI 3: “My favorite restaurant, there is one…”

Ground Truth

Speech2rtMRI (No Audio)

Our Reconstructed MRI and Audio


Transcript

MRI 4: “I like a lot of different genres”

Ground Truth

Speech2rtMRI (No Audio)

Our Reconstructed MRI and Audio


Transcript

MRI 5: “Oh I really like LA. I’m from Connecticut so LA is really really”

Ground Truth

Speech2rtMRI (No Audio)

Our Reconstructed MRI and Audio

Auto-encoding Comparison to TensorTract2

Our work can faithfully reconstruct the identity of speakers, while TensorTract2 is restricted to an adult male vocal tract

Transcript

English 1. We say of course, somebody exclaimed, that they give two turns.

Ground Truth

TensorTract2 (SR: 44100 Hz)

Ours (SR: 16000 Hz)


English 2. I’m a good catholic. But I think I could get along with caring for trees, if I hadn’t anything else.

Ground Truth

TensorTract2 (SR: 44100 Hz)

Ours (SR: 16000 Hz)


English 3. What did it avail to pray, when he knew that his soul lusted after its own destruction?

Ground Truth

TensorTract2 (SR: 44100 Hz)

Ours (SR: 16000 Hz)


English 4. There is hardly one day out of a hundred, which is wholly joyous and sunny.”

Ground Truth

TensorTract2 (SR: 44100 Hz)

Ours (SR: 16000 Hz)


English 5. Among other things on, which she cast her eyes, was a small crucifix of solid silver, standing on a cabinet near the window.

Ground Truth

TensorTract2 (SR: 44100 Hz)

Ours (SR: 16000 Hz)


Korean 1. 너는 내가 아는 어떤 사람이랑 많이 닮았어.

Translation: You look like someone I know.

Ground Truth

TensorTract2 (SR: 44100 Hz)

Ours (SR: 16000 Hz)


Korean 2. 저는 부산에서 자랐어요.

Translation: I grew up in Busan.

Ground Truth

TensorTract2 (SR: 44100 Hz)

Ours (SR: 16000 Hz)


Korean 3. 여름 방학이 눈 깜짝할 사이에 지나갔어.

Translation: Summer vacation passed in a flash.

Ground Truth

TensorTract2 (SR: 44100 Hz)

Ours (SR: 16000 Hz)


Korean 4. 배가 고파 죽겠어.

Translation: I'm starving to death.

Ground Truth

TensorTract2 (SR: 44100 Hz)

Ours (SR: 16000 Hz)


Korean 5. 시험에 떨어져서 속상해 죽겠어요.

Translation: I'm so upset because I failed the test.

Ground Truth

TensorTract2 (SR: 44100 Hz)

Ours (SR: 16000 Hz)


Chinese 1. 现在后悔没带枪了吧。
Translation: You regret not bringing a gun now, don’t you?

Ground Truth

TensorTract2 (SR: 44100 Hz)

Ours (SR: 16000 Hz)


Chinese 2. 今日正式开启了第八十七届奥斯卡的投票大幕。
Translation: Today marks the official opening of voting for the 87th Oscars.

Ground Truth

TensorTract2 (SR: 44100 Hz)

Ours (SR: 16000 Hz)


Chinese 3. 周六没有分长短周。
Translation: Saturday does not distinguish between long and short weeks.

Ground Truth

TensorTract2 (SR: 44100 Hz)

Ours (SR: 16000 Hz)


Chinese 4. 房产证上最多可以登记几个人?
Translation: How many people can be registered on a property ownership certificate at most?

Ground Truth

TensorTract2 (SR: 44100 Hz)

Ours (SR: 16000 Hz)


Chinese 5: 但毕竟恋人还没有登记

Translation: But after all, the couple hasn’t officially registered yet.

Ground Truth

TensorTract2 (SR: 44100 Hz)

Ours (SR: 16000 Hz)


German 1. Dann, als er alles wusste, forderte er mich auf, ihm unsere Zeichnungen und das Tagebuch zu zeigen. Ich brauchte ihm diesen Wunsch nicht zu erfüllen, tat es aber dennoch, weil ich ihn sonst beleidigt hätte, und ich sah doch, dass er es gut mit mir meinte.

Translation: Then, once he knew everything, he asked me to show him our sketches and the diary. I did not have to grant him this request, but I did so anyway, because otherwise I would have offended him—and I could see that he meant well by me.

Ground Truth

TensorTract2 (SR: 44100 Hz)

Ours (SR: 16000 Hz)


German 2. Ich möchte nicht bloß, sondern ich werde es wirklich tun. Wenn er gefangen wird, werde ich ihn befreien, und wenn etwa gar die Waffen gegen ihn gebraucht werden sollen, so stelle ich mich auf seine Seite und kämpfe für ihn. Das will ich euch offen und ehrlich sagen. So, das werdet ihr tun? Wirklich? Ja, ich habe es einem Sterbenden in die Hand versprochen, und ein solches Gelöbnis ist mir der

Translation: I do not merely wish to do it, I truly will. If he is captured, I will free him, and if weapons are to be used against him, then I will take his side and fight for him. I want to tell you this openly and honestly. So, that is what you will do? Really? Yes, I promised it into the hand of a dying man, and such a vow is to me the

Ground Truth

TensorTract2 (SR: 44100 Hz)

Ours (SR: 16000 Hz)


German 3. Nicht, dass es ihm an irgendeinem Bedürfnis fehle; denn er weiß sich durchaus zu beschränken, und für das Notwendige habe ich gesorgt. Auch drückt es ihn nicht, etwas von mir anzunehmen.

Translation: It is not that he lacks any necessity, for he knows very well how to restrain himself, and I have provided for what is essential. Nor does it trouble him to accept something from me.

Ground Truth

TensorTract2 (SR: 44100 Hz)

Ours (SR: 16000 Hz)


German 4. Der Kaplan bat sich ein Glas Wasser aus, weil er sehr erschöpft war durch die Anstrengungen, von der Kirche ins Schloß zu kommen. Die Kammerfrau lief schnell und brachte ihm ein Glas Öl.

Translation: The chaplain asked for a glass of water, as he was very exhausted from the effort of getting from the church to the castle. The maid hurried off and brought him a glass of oil.

Ground Truth

TensorTract2 (SR: 44100 Hz)

Ours (SR: 16000 Hz)


German 5. Die wilden Bienen summten in Ginster und Klee, der Kuckuck rief von ferne, der Wind fuhr leise durch die blühenden Zweige der Linde. Das Weib sucht bei dem Manne, den es liebt, Schutz.

Translation: The wild bees buzzed among broom and clover, the cuckoo called from afar, the wind passed softly through the blossoming branches of the linden tree. The woman seeks protection from the man she loves.

Ground Truth

TensorTract2 (SR: 44100 Hz)

Ours (SR: 16000 Hz)


French 1. Ce regard est peut-être une comédie, pensa Julien; mais cette respiration pressée, mais tout ce trouble… Bah! se dit-il, qui suis-je pour juger de toutes ces choses? Il s’agit ici de ce qu’il y a de plus sublime et de plus fin parmi les femmes de Paris.

Translation: That look may perhaps be an act, thought Julien; but that hurried breathing, all that agitation… Bah! he said to himself, who am I to judge all these things? What is at stake here is what is most sublime and most refined among the women of Paris.

Ground Truth

TensorTract2 (SR: 44100 Hz)

Ours (SR: 16000 Hz)


French 2. Il n’y en avait pas pourtant; mais il y trouva un vase de cuivre jaune qui, à sa pesanteur, lui parut plein de quelque chose, et il remarqua qu’il était fermé et scellé de plomb avec l’empreinte d’un sceau.

Translation: There were none, however; but he found there a yellow copper vessel which, by its weight, seemed to him to be full of something, and he noticed that it was closed and sealed with lead bearing the impression of a seal.

Ground Truth

TensorTract2 (SR: 44100 Hz)

Ours (SR: 16000 Hz)


French 3. La science a déterminé sur le globe la direction de cinq courants principaux: un dans l’Atlantique nord, un second dans l’Atlantique sud, un troisième dans le Pacifique nord, un quatrième dans le Pacifique sud, et un cinquième dans l’océan Indien sud.

Translation: Science has determined, across the globe, the direction of five principal currents: one in the North Atlantic, a second in the South Atlantic, a third in the North Pacific, a fourth in the South Pacific, and a fifth in the southern Indian Ocean.

Ground Truth

TensorTract2 (SR: 44100 Hz)

Ours (SR: 16000 Hz)


French 4. Je me retournai vers le capitaine Nemo. Ce terrible justicier, véritable archange de la haine, regardait toujours. Quand tout fut fini, le capitaine Nemo, se dirigeant vers la porte de sa chambre, l’ouvrit et entra. Je le suivis des yeux.

Translation: I turned toward Captain Nemo. That terrible avenger, a true archangel of hatred, was still watching. When everything was over, Captain Nemo, heading toward the door of his cabin, opened it and went in. I followed him with my eyes.

Ground Truth

TensorTract2 (SR: 44100 Hz)

Ours (SR: 16000 Hz)


French 5. Pas de filaments, pas de rubans, si minces qu’ils fussent, qui ne se tinssent droits comme des tiges de fer. Les fucus et les lianes se développaient suivant une ligne rigide et perpendiculaire, commandée par la densité de l’élément qui les avait produits.

Translation: No filaments, no ribbons, however thin they might be, that did not stand straight like iron rods. The fucus and the lianas developed along a rigid, perpendicular line, governed by the density of the element that had produced them.

Ground Truth

TensorTract2 (SR: 44100 Hz)

Ours (SR: 16000 Hz)