VoiceFlow: Local-First macOS Voice Assistant
Convert spoken commands into verified Mac actions with local Whisper, confirmation gates, and encrypted logs.
WHERE IT STANDS
- Push-to-talk Carbon global hotkey capture feeding 16kHz PCM audio to local memory
- Local faster-whisper CPU int8 transcription (p50: 2,003 ms, zero network egress)
- Deterministic zh-TW technical term normalization dictionary
- Rule-based intent classifier routing commands into read-only vs state-mutating paths
- HUD confirmation modal blocking external side effects (0 auto-submits over 160 soak runs)
- Encrypted local SQLite audit logging via Apple CryptoKit AES-GCM
- Out-of-process watchdog recovering crashed worker processes in 0.43 seconds
- Fine-tuned Whisper model discarded due to timestamp drift (+310ms) and marginal accuracy gains
- Browser automation integration remains an early DOM-level prototype without full AXUIElement integration
- Desktop OS orchestration limited to supported application adapters; not a generic OS automation layer
01 · Problem & Safety Concerns
Voice assistants that execute system-level commands have two major risks. First, continuously streaming microphone audio to third-party cloud servers compromises personal and workspace privacy. Second, speech recognition is never 100% accurate: a misheard word can accidentally delete files, send unintended messages, or run wrong terminal scripts.
VoiceFlow addresses both issues by running everything locally on the Mac CPU and requiring physical user confirmation for any action that modifies system state.
02 · System Constraints
Audio recording, speech-to-text, intent parsing, and command execution run entirely on macOS with no external API calls.
Any action with external effects (terminal scripts, file deletion, web posts) requires pressing a physical confirmation button.
Transcripts, resolved intents, and execution records are encrypted locally in SQLite using Apple CryptoKit AES-GCM.
03 · Architecture & Confirmation Gate
Audio captured via global hotkey is transcribed by local Whisper, cleaned with regex dictionary rules, and parsed for intent. Read-only queries answer directly; actions modifying state pause for confirmation.
Local Voice Input to Verified macOS Action
01. Audio Input: Push-to-talk hotkey captures microphone audio into memory with zero cloud transmission.
02. Local Speech-to-Text: Local faster-whisper CPU int8 transcribes audio (p50: 2,003 ms) without network requests.
03. Term Normalization: Rules fix common tech jargon and zh-TW phrasing errors in the transcript.
04. Intent Classification: Separates simple read-only queries from actions that change system state.
05. Confirmation Barrier: Actions that modify files or make network calls require explicit confirmation (0 auto-submits over 160 soak tests).
06. Execution & Logs: Executes the confirmed action. Watchdog restarts crashed workers in 0.43s. Logs are encrypted with AES-GCM in local SQLite.
04 · Three Architecture Decisions
1. Manual confirmation for actions that change state
ACTION SAFETYContext: Spoken words are easily misheard in noisy rooms or during technical discussions.
Decision: Require user confirmation for all irreversible actions (file changes, POST requests, shell scripts). For social post demos, the workflow stops before publish.
Outcome: Across an 8-hour soak test of 160 tasks, recorded exactly 0 unconfirmed high-risk executions.
2. Independent watchdog process with 0.43s recovery
PROCESS RELIABILITYContext: Local CTranslate2 speech processes or browser automation helpers can occasionally stall or crash under continuous use.
Decision: Split the menu bar UI from background worker processes using local UNIX domain sockets and heartbeat checks. Added an automated watchdog to restart failed workers.
Outcome: In fault injection tests, crashed workers recovered in 0.43 seconds without closing the menu bar UI or losing application state.
3. Encrypted logs with secret redaction
LOG PRIVACYContext: Debugging requires logs, but saving raw voice transcripts risks capturing passwords or sensitive personal details.
Decision: Added regex filters to strip API keys, tokens, and personal identifiers before saving. Encrypted the SQLite database with AES-GCM using keys in the macOS Keychain.
Outcome: No leaked credentials found across all 160 soak task log audits.
05 · Hardest Technical Problem
Preventing UI lockups when global hotkeys failed to register
Root Cause: Under specific macOS security settings, the global hotkey listener could fail during background startup, leaving the app waiting for user input that could never arrive.
Engineering Approach: Decoupled hotkey listener setup from action handling. Added a 500ms timeout fallback: if the hotkey listener fails to respond, the app switches to a standard macOS dialog and selects the safe default path.
Verification: Tested 100 consecutive keyboard listener crash events in the test suite; all 100 switched to standard dialogs without freezing the app.
06 · Discarded Prototype: Fine-Tuned Whisper
Engineering discipline means dropping prototypes that fail real-world tests, even when model training completes cleanly.
07 · Soak Testing & Metrics
8-hour continuous test executing automated voice tasks with zero UI crashes.
Flat memory usage over 8 hours of background operation, with no CoreAudio leaks.
Fast process recovery when worker processes were terminated with SIGKILL during tests.
Tested across 20 audio samples; transcribed locally on CPU within interactive desktop speeds.
08 · Retrospective & Learnings
Engineering Takeaway: Takeaway: In voice desktop tools, test performance on short static audio files does not predict system reliability. Fine-tuning the speech model gave minimal practical improvement (+0.0727), while simple dictionary normalization and a clear confirmation prompt provided 100% protection against wrong actions.
What I Would Do Differently: What I Would Do Differently: Use macOS accessibility APIs (AXUIElement) for UI control earlier, rather than web DOM scraping, which breaks whenever web layouts change.