Talk to Jarvis (local speech-to-text)

Jarvis can listen as well as speak. Speech recognition runs entirely on your machine with a local Whisper model: no account, no API key, no audio upload.

Why this needs a build flag

Voice output works everywhere out of the box, because every OS ships speech synthesis. Voice input is the harder half:

• The web SpeechRecognition API doesn’t exist in WebView2, the engine Jarvis uses on Windows. So on Windows there was simply no dictation.
• Where that API does exist, most browsers implement it by streaming your audio to a cloud service. That breaks the promise that nothing leaves your machine, so Jarvis prefers the local engine whenever it’s available.

The fix is to capture the microphone in Rust and transcribe locally. That pulls in a pure-Rust ML stack, which adds a few minutes of build time, so it’s opt-in rather than forced on everyone.

Enable it

Build with the local-whisper feature:

npm run tauri dev -- --features local-whisper

For a release build:

npm run tauri build -- --features local-whisper

Inference runs through candle, Hugging Face’s Rust ML framework, so no cmake and no LLVM/libclang are needed — whisper.cpp bindings require all three plus a LIBCLANG_PATH environment variable, which is why they aren’t used here.

One honest caveat: this isn’t a completely C-free build. candle-core depends on tokenizers with its default features, which compiles Oniguruma (onig_sys) from C. So you do need a working C compiler — the one that ships with the Rust MSVC toolchain on Windows, or the system cc on macOS and Linux. That’s a far lighter requirement than cmake plus LLVM, but it isn’t zero.

First run: one download

The model isn’t bundled, because shipping tens of megabytes to people who never use voice would be rude. The first time you click the mic button it shows a ⇩ and fetches the model:

• ~43 MB, one time, from Hugging Face’s public mirror (no account needed)
• Cached under your app data directory, namespaced per model
• Written to a .partial file and renamed into place, so an interrupted download can never be mistaken for a good one

Once it lands, the button becomes a real push-to-talk mic and everything after that is offline.

Using it

1. Click the mic (or press it while typing — dictation appends to whatever is already in the composer).
2. Speak.
3. Click again to stop. The button shows … while Whisper transcribes, then drops the text into the composer for you to edit before sending.

If you say nothing intelligible, Jarvis tells you it didn’t catch anything rather than inventing a sentence. That is deliberate: Whisper is known to hallucinate filler like “thank you” on silence, and putting words in your mouth is worse than admitting it heard nothing.

Hands-free (Voice v2)

Once the model is downloaded, the header shows a hands-free toggle. Turn it on and you never touch the machine again:

1. Say “hey jarvis” — the phrase is matched tolerantly, so “hay jarvis” and a mis-heard “jarvus” still work.
2. Ask in the same breath (“hey jarvis, what time is it”) or wait for “Listening.” and then ask.
3. After the answer there’s an 8-second follow-up window where you can just keep talking — no wake phrase needed. It shows a countdown, then goes back to waiting for the phrase.

Say “stop listening”, “never mind”, or “that’s all” to end the session by voice.

The badge tells you exactly what the microphone is doing, because that’s a privacy question, not a UI detail:

Badge	Meaning
hands-free	off — mic closed
armed (mint)	on, listening only for the wake phrase
listening (cyan, pulsing)	mic open, capturing you
thinking	mic closed, model working
speaking	mic closed — it never transcribes its own voice

You can change the wake phrase; it must be at least two words, because one-word phrases trigger constantly in ordinary speech.

Talking over it (Voice v3)

While Jarvis is reading an answer out, you can just start talking. It stops and listens, and what you say next goes straight to the model — no wake phrase, and no waiting for it to finish being wrong.

The hard part is that a microphone next to a speaker hears Jarvis at least as loudly as it hears you. With no echo cancellation, a fixed threshold either fires on Jarvis’s own voice (so it interrupts itself, forever) or is set so high that only shouting works.

So it measures instead of assuming. For the first half-second of playback the detector does nothing but listen to how loud Jarvis sounds in your microphone, in this room, at this volume. After that, only sound clearly above that measured level, held long enough to not be a cough or a door, counts as you talking.

Two consequences worth knowing:

• Headphones make it near-perfect. There’s no echo to clear, so anything you say wins immediately.
• A laptop at high volume may defeat it. If Jarvis is louder in the mic than you are, nothing you say can clear the measured floor. Turn the volume down or use headphones.

The microphone is released as soon as playback ends, not when the watcher’s time estimate runs out — otherwise the follow-up window that opens right after an answer would find the device still busy.

Nothing captured while it speaks is transcribed or kept. The detector reads loudness and throws the audio away frame by frame, so Jarvis still cannot hear its own voice into a request — that’s the property the closed mic protected in v2, and barge-in does not weaken it.

What it does to your audio

Before transcription, the take is conditioned locally:

Step	Why
Downmix to mono	Whisper takes one channel
Resample to 16 kHz	the only rate Whisper accepts
Trim silence	drops dead air at both ends
Normalize peak	a quiet talker isn’t read as silence
Energy gate	skips transcription entirely if nobody spoke

Long dictation is split on Whisper’s 30-second window and rejoined, so a long thought isn’t silently cut off.

Choosing a model

Two checkpoints are available, both quantized to keep the download small:

• tiny.en (default) — English only, fastest
• tiny — multilingual

The English-only model is sharper for English and is what you get unless you ask otherwise.

If it doesn’t work

• Mic button is dim with “not available in this build” — you’re running without --features local-whisper, and this window has no speech recognizer.
• “no microphone found” — check your OS input device; Jarvis uses the system default.
• Nothing transcribes but the mic lights up — try speaking closer; the energy gate treats very quiet takes as silence on purpose.