Metadata-Version: 2.4
Name: sessionvoice
Version: 0.1.0
Summary: One listening engine for VoiceMode and the Delta bot: Apple / Nemotron / whisper -> one event stream -> VAD -> turns
License-Expression: AGPL-3.0-or-later
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy
Provides-Extra: capture
Requires-Dist: httpx; extra == "capture"
Requires-Dist: scipy; extra == "capture"
Requires-Dist: sounddevice; extra == "capture"
Requires-Dist: onnxruntime; extra == "capture"
Requires-Dist: webrtcvad-wheels; extra == "capture"
Provides-Extra: silero
Requires-Dist: onnxruntime; extra == "silero"
Provides-Extra: nemotron
Requires-Dist: mlx-audio>=0.5.4; extra == "nemotron"
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Requires-Dist: httpx; extra == "test"
Provides-Extra: all
Requires-Dist: httpx; extra == "all"
Requires-Dist: scipy; extra == "all"
Requires-Dist: sounddevice; extra == "all"
Requires-Dist: onnxruntime; extra == "all"
Requires-Dist: webrtcvad-wheels; extra == "all"
Requires-Dist: mlx-audio>=0.5.4; extra == "all"
Dynamic: license-file

# sessionvoice (the ears and the mouth)

sessionvoice is the distribution and the Claude Code plugin. The commands
and modules keep their names: `ears`, `ears-serve`, `mouth`, `src/ears`,
`src/mouth`.

One listening engine for VoiceMode and the Delta bot. It is always listening,
streams transcription, and decides when he has finished.

    PCM (s16le mono) -> backend ----------> events (docs/events.md) -> stdout
                          apple | nemotron | whisper       ^
                        VAD (Silero) ----- edges ----------+
                        turn stage (silence | smart) -- turn, preview

- **Backends**, all speaking one event format:
  - `apple`: SpeechTranscriber on macOS 27, through the `apple-speech` Swift
    helper.
  - `nemotron`: Nemotron 3.5 streaming on MLX, run as a child process.
  - `whisper`: any OpenAI-compatible transcription URL, for Linux or pre-27
    macOS.
- **One VAD** (Silero, falling back to energy). It produces `vad` edges, and
  Nemotron segments on them too.
- **Turn stages:** `silence` (the bot's join rules, plus commit) and `smart`
  (Cora's Smart Turn and Parakeet re-decode), both in-process with the audio.

## Components

The engine above is the core. Around it, these parts make it an always-on
pair of ears on a Mac (m5, 2026-10-01; depth in `docs/architecture/`):

- **Capture** (`src/ears/listen`): `ears.listen capture` reads one input
  device and writes every partial, turn and heartbeat to the **heard log**
  (`src/ears/heard.py`):
  `~/.voicemode/logs/conversations/heard_YYYY-MM-DD.jsonl`.
  - **A speaker change ends the turn** (`--speaker-split K`, EARS-14): when
    another named speaker holds K partials in a row (by the speaker
    side-car's `speakers_YYYY-MM-DD.jsonl`), the capture closes the turn
    before them and opens a new one, so each heard turn has one speaker. A
    split turn carries `speaker` (the side-car's label: `alice`, `mike`, ...).
  - **Every turn the side-car named carries `speaker`** (EARS-106): an
    unsplit turn is named by the same rule, from its own partials' labels
    (the one name that held K in a row; only a turn of fewer than K
    partials, too short to reach K at all, is named by its named labels
    if they are all one name; a longer turn no name held K in a row,
    such as a mostly-neutral TV turn with one stray `mike`, carries
    none), and marked
    `speaker_from: labels`. A turn of neutral labels only (the TV), no
    labels yet, or a change the split had not cut when the turn closed,
    carries none. **Default 2; 0 = off** (no `speaker` on any turn: today's
    turns exactly). No side-car file: turns as before. Labels that name
    nobody (`other`, `mixed`, `silent`, `unheard`) never count.
  - **Deploy turns it on.** `keeper/ears-keeper.sh` starts the capture
    without `--speaker-split`, so the default K=2 goes live at the next
    capture restart after merge. **Rollback:** add `--speaker-split 0` to
    the capture line in `start_capture` and restart the capture.
- **Keeper** (`keeper/ears-keeper.sh`, launchd `com.failmode.ears-keeper`):
  keeps the capture alive on a device that comes and goes. It follows the
  Mac's default input through `keeper/input-watch.swift`, and when the
  AirPods leave or return it stops the capture and starts one on the right
  device. It only manages the capture a keeper started; other captures
  (a call-tap, a call leg) are left alone.
- **Sounds** (`keeper/ears-sounds.py`, launchd `dev.voicemode.ears-sounds`):
  a short sound in his ears when the ears log an event, so he knows he was
  heard.
- **Turn mail** (`keeper/ears-mail.py`, launchd `dev.voicemode.ears-mail`,
  EARS-6): mails every finished turn of his To and From
  `heard+<device>@<host>`, with the `Ears-*` headers of the pager genie's
  contract (sessionmail `openspec/changes/pager-genie`). A session hears a
  device by subscribing to that address. The box is `heard` (`--sender`;
  Mike, 16:13 Thu 2026-10-01), because `ears` is this project's own box.
  sessionmail trusts only its keeper's From (`_voice.KEEPER`), so the
  plist loads only once that says `heard` too. `--to seats` is the older mail, To
  each seat a turn is for. **Never run `--to seats` and `--to device` for
  the same seat: it would get each call twice.**
  `Ears-Speaker` is whose turn it is. A turn with a `speaker` label (a
  split turn, EARS-14, or any turn the side-car named, EARS-106) is mailed as `<label>@<host>`: `alice` becomes
  `alice@m5`, the same form as `--speaker`. The label is lower-cased, and any
  character outside `[a-z0-9._-]` becomes `-`. Every other turn, including
  one whose label is empty or names nobody, keeps `--speaker`
  (`$EARS_SPEAKER`, else `unknown@<host>`: a voice the matcher could not
  place is never mailed as Mike, EARS-110).
  `Ears-Echo` says a mouth line heard back through the mic was cut out of
  the turn (EARS-98). On the speakers the mic hears the mouth, so a line's
  words are cut wherever they sit in his turn: at its start (up to `3 + 5
  words/s × the seconds the line began after his turn` of his words ahead of
  it), in the middle, or at the end. The aim is to cut only the echo, never
  his words around it. That is what the fixtures test (8 real echo turns, 8
  AirPods quotes), not a guarantee: the cut is a text match. A span starts
  on 2 or more matched words in a row, or on a lone word joined to them by
  one-for-one swaps (a misheard name), so a lone "the" of his does not drag
  his words into a clipped echo. Inside a span, up to 30% unmatched words
  (half, when the span is a word-for-word copy of the line) are taken as
  misheard echo. The header is ONE line, a comma list with an entry per line
  cut, in turn order: `Ears-Echo: cora/2d492fb06d3b 31 words cut,
  cora/6c681f74374e 30 words cut` (`<seat>/<utt> <n> words cut`; the seat is
  the line's agent). It goes on both the device mail and the `--to seats`
  mail; the seats mail also keeps its "heard whole" section. A mail with no
  cut has no `Ears-Echo`. A turn that is all echo is still not mailed.
  Neither is one word left BETWEEN two cuts: that is the seam misheard
  ("86%" for "eighty-six"). A lone word before or after both cuts is his and
  is mailed. Timing uses the line's play span (`said` − `played_s` ..
  `said`) and the turn's real span (the stream start, from the heartbeat or
  `listen-started`, + `t0`..`t1`), else `ts − (t1 − t0)`. Three guards keep
  his quotes of a line: a line played on headphones (AirPods, as the mouth's
  `BARGE_DEVICES`) is never cut; a cut needs half the line's played words;
  and, only when the stream start is known (the real span), the turn must
  have opened within 2 s of the line beginning, unless the turn is mostly
  the line. On the `ts − (t1 − t0)` estimate, only the first two guards
  apply. Numerals (`9%` for "nine percent") still slip past the word match.
- **Waiter** (`bin/heard-first`): an agent blocks on the heard log
  until a line is for it, by `--rule`: its name in a partial or a
  turn (`partial:REGEX`, `turn:REGEX`), a reply to its last line
  (`reply:AGENT`), or his first words (`first`); then it wakes.
  `bin/ears-turn` then reads the whole turn, one line.
- **Mouth** (`src/mouth`, `bin/mouth`): an agent speaks through one queue,
  and its lines go into the same log, so the waiters can gate out echo.
- **Calls:** `bin/call-tap` puts a call from any app into the ears, both
  ways; `keeper/call-ears` grows an ear for each Delta call and drops it
  at hang-up. The ears keeper runs it as its child: one launch agent,
  `com.voicemode.ears-keeper` (`ears services install`), holds the room's
  ear and every call's.
- **Speaker names** (not in this repo yet): a sidecar tags each heard line
  with who spoke, in `speakers_YYYY-MM-DD.jsonl` beside the heard log, and
  the waiters read it. Moving it into VoiceMode as an opt-in service is
  VM-2326.

## Use

    ears backends                          # what runs here
    ears install                           # fetch Silero (the engine: Parakeet on mlx-audio :8890)
    ears install --apple                   # also build the Apple helper (swift, macOS 27) + assets
    ears file talk.wav --realtime --turns silence --previews
    rec ... | ears listen --turns silence  # live
    ears serve 2023                        # a whisper-compatible URL on Apple STT

In Python:

    from ears.cli import Pipeline
    p = Pipeline("auto", turns="silence", previews=True, emit=my_handler)
    p.feed(pcm_bytes)   # every 20 ms
    p.close()

## Install

The distribution is `sessionvoice` (`sessionvoice[nemotron]`, `sessionvoice[all]`);
it installs the `ears`, `ears-serve` and `mouth` commands.

    uv venv && uv pip install -e '.[all]'  # onnxruntime + mlx-audio
    ears install

## Tests

One command runs the whole suite, offline:

    make test                  # uv run --locked pytest
    make test ARGS="-k mouth"  # pytest flags pass through

`uv sync` (or `make sync`) once installs the `dev` group from `uv.lock`; after
that nothing in the suite touches the network, and nothing plays a sound.
Without `sounddevice` (PortAudio) the four `tests/test_ears_sounds_call.py`
tests skip: `uv sync --extra capture` brings it (and `onnxruntime`, for the
silero test), and then they run, and three of them play a short tick through
the default output. Without VoiceMode on `PYTHONPATH` its
three tests skip.

## Measured (m5, 2026-09-19; Cora's eval set, her `score.py`, whisper normaliser)

WER through the ears pipeline:

| backend  | 20 turns (1858 w) | 150 singles | how                       |
|----------|-------------------|-------------|---------------------------|
| apple    | 1.51%             | 2.16%       | fast feed, finals joined  |
| apple    | 1.35%             | -           | one 827 s call, real time |
| nemotron | 3.01%             | 3.79%       | fast feed, finals joined  |

- **The real-time call** is the 20 turns joined with 3 s gaps, using silence
  turns with commits on. It produced 22 turns for 20 inputs.
- **The old helper**, which finalised the live analyzer at each commit, scored
  5.17% on the same call, with 54 deletions.
- **Turn latency on turn01, fed in real time:**
  - Apple fires 1.82 s after speech ends, including a 0.2 s commit wait.
  - Nemotron fires at 1.65 s.
  - VAD edges lag about 140 ms at onset and about 280 ms at offset (a 0.2 s
    offset debounce).
- **Memory:** Apple uses about 75 MB of RSS on our side; Nemotron about 2.2 GB.
- **Turn stages (Cora, 20 turns, real time):** `silence` cut 1 of 20, with a
  p50 of 1.59 s. `smart` veto cut 0 of 20, with a p50 of 2.43 s.

## The mouth (`src/mouth`)

The speaking half: `mouth say` queues a line (a maildir), a resident
player speaks it through Kokoro or a cloned voice, to the room or into a
Delta call, and writes `saying`/`said` rows to the heard log so the
waiters can tell echo from speech. Design: `src/mouth/DESIGN.md`.

    mouth say --voice pip --device default "Hello."
    mouth say --voice pip --device call --hold turn-end "On the call."
    mouth --help            # stop, retract, amend, status, devices, serve

Moved here from VoiceMode `feat/mouth-say` @ 64cccd80 (Mike, 00:43 Mon
2026-09-28: "we put the mouth with the ears"). The live player still runs
the VoiceMode copy until cutover. Three ties to VoiceMode remain, and
they come over with the capture:

- `voice_mode.heard`: the heard log's reader and writer, shared with the
  capture.
- `voice_mode.voice_profiles`: resolves cloned voices (`pip`, `laurie`).
- `voice_mode.dj`: optional, loaded only for music ducking.

Tests: `make test ARGS=tests/mouth` (see Tests above); put a VoiceMode checkout
on `PYTHONPATH` to run the three tests that need it.
