Project · 2026 · interactive
A voice assistant with nothing in the cloud
Speech in, speech out, with no server: voice detection, speech recognition, a language model and speech synthesis all run on the device. The hard part is not any one model but the wait between the end of your question and the first word of the answer. Below, one spoken turn is replayed stage by stage with the app's measured timings; change how it behaves and watch the first word move.
- First spoken word, real app on a laptop
- ~1.3 s
- Filler phrase if the model is slow
- after 1.2 s
- Barge-in
- talk over it
- Network calls
- none
Try it
How far away is the Moon?
Pick a question and press Ask.
First word 2.25 s after you stopped talking; the full reply is written at 2.84 s, so streaming starts speaking 0.84 s sooner than waiting.
Press Ask, then untick “speak sentence by sentence” and ask again to see what streaming buys. Tick “slow model” to see the filler phrase cover a late answer, and press “talk over it” while it speaks. Stage times are the app's measurements; the replies are examples written for this page.
The sentence splitter
- 1 Dr. Rao said the dose is 3.5 mg.
- 2 Take it twice a day.
- … Call if you feel diz still being written, held back
A sentence is sent to speech only when the next one has started, so “Dr.” and “3.5” never cut a sentence short. Type more and watch the last piece move up.
How it works
Know when you have stopped. A voice detector listens to the microphone in short frames. Speech starts when the level rises above the background; the turn ends after 600 ms of silence, the price of not cutting people off mid-pause. The app uses the Silero neural detector; this page uses a simpler loudness detector with the same timing.
Stream, don't wait. The language model writes its answer a few characters at a time. Each time a sentence is complete it goes straight to speech synthesis while the model keeps writing the next, so the first sentence is heard while the rest is still being generated. The splitter knows that “Dr.” and “3.5” do not end sentences.
Never leave silence or an error. If no sentence is ready 1.2 s after the question, a short filler phrase plays. If the model fails, a canned conversational line replaces it, never a raw error.
Let people interrupt. While the assistant talks, the microphone stays open with a stricter threshold, since its own voice leaks into the mic; speech that lasts 320 ms stops playback and the generation at once.
The real app
| Metric | Value | Source |
|---|---|---|
| End-of-speech window | 600 ms | ARCHITECTURE.md |
| Speech to text, faster-whisper base, int8, CPU | ~600 ms | ARCHITECTURE.md |
| Llama 3.2 3B first sentence, streamed | 570–1,160 ms | ARCHITECTURE.md |
| Piper speech for the first sentence | 100–540 ms | ARCHITECTURE.md |
| Model plus speech to first audio, mean | ~1.34 s | ARCHITECTURE.md |
Measured on an RTX 3050 laptop with Llama 3.2 3B in Ollama, faster-whisper and Piper, with nothing sent over the network. Code on GitHub.