Metadata-Version: 2.4
Name: heyvox
Version: 1.0.0
Summary: macOS voice layer for AI coding agents — wake word, STT, TTS, HUD, media control
Author-email: Franz Felberer <franz@felberer.at>
License-Expression: MIT
Project-URL: Homepage, https://heyvox.dev
Project-URL: Documentation, https://github.com/heyvox-dev/heyvox#readme
Project-URL: Repository, https://github.com/heyvox-dev/heyvox
Project-URL: Issues, https://github.com/heyvox-dev/heyvox/issues
Project-URL: Changelog, https://github.com/heyvox-dev/heyvox/releases
Keywords: voice,speech,tts,stt,wake-word,mcp,ai-coding,macos,local,privacy
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: MacOS X
Classifier: Intended Audience :: Developers
Classifier: Operating System :: MacOS
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Multimedia :: Sound/Audio :: Speech
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: openwakeword>=0.6.0
Requires-Dist: pyaudio>=0.2.14
Requires-Dist: numpy>=1.24.0
Requires-Dist: PyYAML>=6.0
Requires-Dist: pydantic>=2.0
Requires-Dist: platformdirs>=4.0
Requires-Dist: sherpa-onnx>=1.0
Requires-Dist: pyobjc-framework-Cocoa>=10.0
Requires-Dist: pyobjc-framework-Quartz>=10.0
Requires-Dist: mcp>=1.0
Requires-Dist: psutil>=5.9
Requires-Dist: rich>=13.0
Provides-Extra: apple-silicon
Requires-Dist: mlx-whisper>=0.1.0; extra == "apple-silicon"
Provides-Extra: tts
Requires-Dist: kokoro>=0.3.0; extra == "tts"
Requires-Dist: sounddevice>=0.4.0; extra == "tts"
Requires-Dist: huggingface_hub>=0.20.0; extra == "tts"
Provides-Extra: aec
Requires-Dist: livekit>=1.0.0; extra == "aec"
Provides-Extra: learn-vocab
Requires-Dist: anthropic<1,>=0.84; extra == "learn-vocab"
Provides-Extra: chrome
Requires-Dist: websockets>=13.0; extra == "chrome"
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: pytest-asyncio; extra == "dev"
Requires-Dist: pytest-mock; extra == "dev"
Requires-Dist: pytest-subprocess; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Dynamic: license-file

# HeyVox — Voice Coding, not Vibe Coding

> **Beta** — HeyVox is under active development. It works, but expect rough edges. If something breaks, please [open an issue](https://github.com/heyvox-dev/heyvox/issues) — your feedback shapes what gets fixed next. See [heyvox.dev](https://heyvox.dev) for the full picture.

Your AI coding agent doesn't just listen — it talks back. HeyVox adds a voice layer to any MCP-compatible AI coding agent (Claude Code, Cursor, Windsurf, Continue.dev). Wake word detection, speech-to-text, text-to-speech with configurable verbosity, and browser media control — all running locally on your Mac.

## How It Works

**Voice IN** — Speak to your agent:
1. Say the wake word (or hold your push-to-talk key)
2. HeyVox records and transcribes your speech locally (MLX Whisper / sherpa-onnx)
3. Your words are pasted into the agent's input field and sent

**Voice OUT** — Your agent speaks back:
1. Claude Code responses with `<tts>` blocks trigger Herald (TTS orchestration)
2. Herald generates speech locally via Kokoro TTS and plays it
3. System media (YouTube, Spotify) auto-pauses during playback via Hush
4. Alternatively, agents can call `voice_speak()` via MCP

**HUD** — See what's happening:
- Menu bar icon shows current state (idle/recording/transcribing/speaking)
- Frosted-glass pill overlay shows live waveform during recording
- Recent transcript history in dropdown menu

## What's Included

HeyVox is a monorepo with three integrated components:

| Component | What it does |
|-----------|-------------|
| **HeyVox Core** | Wake word detection, push-to-talk, local STT, text injection, HUD overlay, MCP server |
| **Herald** | TTS orchestration — Kokoro speech generation, playback queue, audio ducking, workspace-aware delivery |
| **Hush** | Browser media control — Chrome extension that pauses/resumes YouTube, Spotify Web, etc. |

One install gets everything. One setup command wires it all up.

## Requirements

- macOS 14+ (Apple Silicon recommended for MLX Whisper)
- Python 3.12+
- Microphone access, Accessibility, and Screen Recording permissions
- PortAudio (`brew install portaudio`)

## Install

### From source (recommended for now)

```bash
# Prerequisites
brew install portaudio

# Clone and install
git clone https://github.com/heyvox-dev/heyvox.git
cd heyvox
pip install -e ".[apple-silicon,chrome]"

# Run the setup wizard
heyvox setup
```

### From PyPI (coming soon)

```bash
pip install heyvox
pip install 'heyvox[apple-silicon]'   # Apple Silicon acceleration (MLX Whisper)
pip install 'heyvox[tts]'             # Kokoro TTS support
pip install 'heyvox[full]'            # Everything
```

### What `heyvox setup` does

1. Checks macOS permissions (Accessibility, Microphone, Screen Recording)
2. Installs PortAudio if missing
3. Downloads the Kokoro TTS model (~300 MB)
4. Tests your microphone
5. Creates config at `~/.config/heyvox/config.yaml`
6. Installs launchd service (auto-start at login)
7. Installs Herald TTS hooks for Claude Code (`~/.claude/settings.json`)
8. Registers the MCP voice server with your AI agent
9. Shows setup summary

## Usage

### Core commands

```bash
heyvox start              # Start in foreground
heyvox start --daemon     # Start as background service (launchd)
heyvox stop               # Stop the service
heyvox restart            # Restart the service
heyvox status             # Show status
heyvox logs               # Tail service logs
```

### Voice output (TTS)

```bash
heyvox speak "Hello"      # Speak text via Kokoro TTS
heyvox skip               # Skip current TTS playback
heyvox mute               # Toggle TTS mute

# Herald CLI (advanced TTS control)
herald status              # Queue, hold, playing, muted, paused
herald pause               # Pause TTS playback
herald resume              # Resume TTS playback
herald queue               # Show queued messages
herald skip                # Skip current message
```

### Transcription history

```bash
heyvox history             # Show recent transcriptions
heyvox history -c          # Copy last transcript to clipboard
```

### Browser media control (Hush)

Hush is a Chrome extension that pauses browser media during recording and TTS.

1. Open Chrome → `chrome://extensions/`
2. Enable Developer mode
3. Click "Load unpacked" → select the `heyvox/hush/extension/` folder
4. Run the install script to set up native messaging:
   ```bash
   bash heyvox/hush/scripts/install.sh
   ```

Hush is optional — without it, HeyVox falls back to native MediaRemote for Spotify/Apple Music, or media key simulation.

## MCP Tools

Once registered, your AI agent gets these voice tools:

| Tool | Description |
|------|-------------|
| `voice_speak(text, verbosity)` | Speak text aloud (full/summary/short/skip) |
| `voice_status()` | Get current state (idle/recording/speaking) |
| `voice_queue(action)` | Manage TTS queue (list/skip/stop/mute/unmute) |
| `voice_config(action, key, value)` | Get or set voice settings at runtime |

## Configuration

Edit `~/.config/heyvox/config.yaml`:

```yaml
wake_words:
  start: hey_jarvis_v0.1   # Wake word model (custom "hey_vox" coming soon)

threshold: 0.5
silence_timeout_secs: 5.0

stt:
  backend: local
  local:
    engine: mlx             # mlx (Apple Silicon GPU) or sherpa (CPU)
    mlx_model: mlx-community/whisper-small-mlx

tts:
  enabled: true
  voice: af_heart           # Kokoro voice (af_sarah, af_nova, af_sky, etc.)
  speed: 1.0
  verbosity: full           # full | short (first sentence only) | skip (mute)
  style: detailed           # detailed | concise | technical | casual
  pause_media: true         # Pause YouTube/Spotify during TTS

target_mode: last-agent     # last-agent | pinned | generic
agents:                     # Apps to track for last-agent mode
  - Claude
  - Conductor
  - Cursor
  - Terminal

push_to_talk:
  enabled: true
  key: fn                   # Push-to-talk key

mic_priority:               # Preferred mics (checked in order)
  - Jabra
  - G435
  - MacBook Pro Microphone
```

## Supported Agents

HeyVox works with any app that supports MCP (Model Context Protocol):

| Agent | Voice IN | Voice OUT (MCP) | Voice OUT (Herald hooks) | Auto-register |
|-------|----------|----------------|-------------------------|---------------|
| Claude Code | Yes | Yes | Yes | Yes |
| Cursor | Yes | Yes | — | Yes |
| Windsurf | Yes | Yes | — | Yes |
| Continue.dev | Yes | Yes | — | Yes |
| Any focused app | Yes | — | — | — |

**Voice IN** works with any app — HeyVox pastes transcribed text into whatever's focused.
**Voice OUT via MCP** — the agent calls `voice_speak()` to talk back.
**Voice OUT via Herald** — Claude Code hooks automatically speak `<tts>` blocks in responses.

## Pairs Well With Visual MCPs

HeyVox is at its best when combined with **external MCP servers that emit visual artifacts**. Voice keeps the discussion flowing; visuals anchor it — together they make architecture conversations dramatically easier to follow.

**Example: external Mermaid MCP**

Pair HeyVox with a Mermaid MCP server (any MCP that renders Mermaid diagrams to images or SVG) and your agent gains a synchronized visual side-channel:

1. You discuss a design change out loud — wake word → STT → agent receives the question
2. Your agent narrates the answer via Herald TTS, so your eyes stay on the diagram
3. In parallel, the agent calls the Mermaid MCP to regenerate the diagram with the proposed change
4. You see the delta at a glance — boxes added/removed, edges rerouted — instead of holding every structural change in your head while listening

This pattern generalizes to any external MCP that returns images, SVGs, or diagrams (PlantUML, Excalidraw, mind-map tools, even slide generators). HeyVox owns the conversation channel; the visual MCP owns the artifact. They don't need to know about each other — your AI agent orchestrates both.

> **Why it feels good**: voice-only design reasoning forces you to remember every change as you hear it. Visual-only design reasoning makes you keep typing to drive updates. With both, you can talk *and* look — and a 10-minute design discussion finishes with an up-to-date diagram already in front of you.

## Architecture

```
┌──────────────────────────────────────────────────┐
│  Your AI Agent (Claude Code, Cursor, etc.)       │
│  ┌────────────┐  ┌─────────────────────────┐     │
│  │ MCP Client │  │ Claude hooks (on-response)│    │
│  └─────┬──────┘  └──────────┬──────────────┘     │
└────────┼────────────────────┼────────────────────┘
         │ stdio              │ <tts> blocks
┌────────┼────────────────────┼────────────────────┐
│  HeyVox                    │                     │
│  ┌───────────┐  ┌──────────▼──────────┐          │
│  │ MCP Server│  │ Herald              │          │
│  │ (TTS out) │  │ (Kokoro → queue →   │          │
│  └───────────┘  │  play → duck audio) │          │
│                 └──────────┬──────────┘          │
│  ┌───────────┐             │ pause/resume        │
│  │ Wake Word │  ┌──────────▼──────────┐          │
│  │ (mic in)  │  │ Hush (browser media)│          │
│  └─────┬─────┘  └────────────────────┘          │
│        │                                         │
│  ┌─────▼───────┐  ┌───────────┐                 │
│  │  Local STT  │  │ HUD       │                 │
│  │ (MLX Whisper│  │ (overlay +│                 │
│  │  sherpa-onnx│  │  menu bar)│                 │
│  └─────┬───────┘  └───────────┘                 │
│        │                                         │
│  ┌─────▼───────┐                                │
│  │ Text Inject │                                │
│  │ (osascript) │                                │
│  └─────────────┘                                │
└──────────────────────────────────────────────────┘
         All processing runs locally.
         No audio leaves your machine.
```

## System Requirements

| Resource | Idle | Active (recording + TTS) |
|----------|------|--------------------------|
| RAM | ~150 MB | ~900 MB peak |
| GPU (Metal) | — | ~500 MB (Whisper STT) |
| Disk | ~800 MB (models) | — |
| CPU | <1% | ~15% during STT |

- **Whisper STT** runs on Apple Silicon GPU via MLX (not system RAM)
- **Kokoro TTS** runs on Metal GPU via mlx-audio (~5-10x faster than CPU), auto-unloads after idle timeout
- **Wake word** model is tiny (~5 MB), always loaded
- **Memory watchdog** auto-restarts if RSS exceeds 1 GB

## Platform Support

- **macOS** (Apple Silicon) — fully supported, primary target
- **macOS** (Intel) — supported via sherpa-onnx STT (no MLX)
- **Windows / Linux** — coming soon (core engine is cross-platform Python; only UI and system integration need porting)

## Audio Devices

| Device Type | Mic Quality | Playback Quality | Notes |
|-------------|-------------|------------------|-------|
| 2.4 GHz wireless (USB dongle) | High | High | Best option — bypasses Bluetooth entirely |
| USB/3.5mm wired headset | High | High | Great option |
| Built-in Mac mic | Good | N/A | Works fine for voice input |
| Bluetooth headset (incl. AirPods) | Reduced | Reduced | Works — with automatic mitigations (see below) |

**Bluetooth and HFP mode**: Bluetooth can either stream high-quality audio (A2DP) or do bidirectional mic+speaker (HFP) — not both simultaneously. When the mic activates, audio quality drops to phone-call level (~16 kHz). This is a macOS/Bluetooth limitation, not a HeyVox bug.

**HeyVox mitigations**: Bluetooth headsets work in practice because HeyVox handles the rough edges automatically:
- **Dead device filtering** — CoreAudio queries skip paired-but-disconnected Bluetooth devices
- **Silent mic recovery** — detects when a Bluetooth mic goes silent and auto-switches to an alternative (with exponential backoff)
- **Echo suppression** — TTS echo buffer strips recently spoken text from STT output, preventing feedback loops in speaker mode

**For best results**: Headsets with a 2.4 GHz USB dongle (Logitech G435, SteelSeries Arctis, Jabra) bypass Bluetooth entirely and appear as standard USB audio — no codec switching, no quality drop.

## Privacy

HeyVox processes everything locally:
- Wake word detection: openwakeword (on-device)
- Speech-to-text: MLX Whisper or sherpa-onnx (on-device)
- Text-to-speech: Kokoro (on-device)
- No audio is sent to any server, ever

## Development

```bash
# Install with dev extras
pip install -e ".[dev,apple-silicon,chrome]"

# Run tests
pytest tests/ -k "not e2e"

# Lint
ruff check heyvox/ tests/
```

## License

MIT
