An NVIDIA Employee Built a Garage Door That Talks Back in a Pirate Voice

An NVIDIA employee introduced a Halloween project called “MonsterHouse” on the Hugging Face blog on October 30 (local time). The setup projects a moving mouth onto a garage door, and when visitors speak to it, the house answers in the voice of a cranky pirate ship. It is built on NVIDIA’s Nemotron Voice Agent and runs on a single workstation.

A garage door that softens if you ask nicely

MonsterHouse comes with a backstory: the house was built from the timbers of a sunken pirate ship. At first it finds visitors annoying, but it mellows if you ask politely. Thank it, and it says a warm goodbye. The creator fed earlier parts of the conversation back into the system so the house would not rehash the same argument with someone who had already won it over.

Listening to visitors and composing replies is divided among several AI models. A Nemotron speech recognition model turns speech into text, and Nemotron 3.5 Bolt E4B 4GB writes the response. Magpie TTS Zeroshot reads that response in the character’s voice, and Audio2Face-3D moves the mouth in sync with the audio. The resulting mouth animation is layered onto a scene built in Unreal Engine 5.

Keeping the personality consistent took more than a prompt. The creator assembled a training set of more than 5,000 multi-turn conversations and fine-tuned the model on it. The checks included whether a reply followed naturally from what came before, whether it repeated itself, and whether the tone and pacing felt right. After testing more than a dozen models, Bolt E4B was chosen. It used fewer computing resources than the other candidates, Qwen3 30B-A3B and Qwen 3.6 35B-A3B, the creator explained.

How long is the wait for a reply?

In the creator’s own measurements, the median time from the end of a visitor’s sentence to the start of a freshly generated spoken reply was 1.70 seconds. When a pre-stored response was used, it was 0.72 seconds. For now, 0.3 seconds of silence is treated as the end of a turn. Frequently used lines, such as greetings and candy jokes, were prepared in advance as text or recordings to shorten the wait.

Image: huggingface.co

The voice can convey seven moods, including grumpiness, menace, and delight. The creator recorded voice samples three to 10 seconds long for each mood and let the AI pick the emotion that matched the line. The mouth animation is generated from clean audio, before effects are applied. Effects such as reverb are then added separately to the audio visitors actually hear, completing the character’s voice.

Could you build this at home?

It is one computer, but with two graphics cards. An RTX PRO 6000 Blackwell Max-Q handles conversation and speech, while an RTX PRO 4000 Blackwell takes care of graphics and mouth animation. The creator said the roles were split so the visuals would keep a steady frame rate even while conversation was being processed. The MonsterHouse code itself has not been cleaned up for distribution, but all of the NVIDIA models it uses are available to anyone, the creator explained.

Image: huggingface.co

There is also a camera feature that recognizes visitors’ costumes. Nemotron 3 Nano Omni describes the costume, while a separate component tracks the number of people and their movement. The costume recognition example that was published is a test on synthetic video featuring five sheep in costume. The creator said more testing is needed for real driveway conditions with groups of visitors arriving at once, and under nighttime lighting.

There is also a mode that lets an operator pick up a microphone and voice the house directly. What remains to be seen is how naturally it converses with actual visitors. The creator’s goals of unattended operation and automatic candy dispensing are still on the to-do list.