Metadata-Version: 2.4
Name: syne-voice
Version: 0.1.0
Summary: Voice and avatar conscious companion runtime on top of the Mnemosyne memory substrate. 4-process tree (supervisor, cortex, voice, swarm), FLAME point-cloud avatar, LiveKit voice loop, Darwin-Godel self-editing with Tier-7 filesystem lock. CLI and Python API.
Author: Geass Labs
License: MIT License
        
        Copyright (c) 2026 Geass Labs
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: mnemosyne
Requires-Dist: numpy>=1.26.0
Requires-Dist: hypothesis>=6.100.0
Requires-Dist: anyio>=4.0.0
Provides-Extra: dev
Requires-Dist: pytest>=8.0.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.23.0; extra == "dev"
Requires-Dist: pytest-xdist>=3.5.0; extra == "dev"
Requires-Dist: ruff>=0.5.0; extra == "dev"
Requires-Dist: mypy>=1.10.0; extra == "dev"
Requires-Dist: httpx>=0.27.0; extra == "dev"
Requires-Dist: respx>=0.21.0; extra == "dev"
Provides-Extra: avatar
Requires-Dist: PyDrawille==0.0.1.2; extra == "avatar"
Requires-Dist: drawille>=0.1.0; extra == "avatar"
Provides-Extra: voice
Requires-Dist: livekit-agents>=0.11.0; extra == "voice"
Requires-Dist: livekit-plugins-turn-detector>=0.4.0; extra == "voice"
Requires-Dist: livekit-plugins-silero>=0.7.0; extra == "voice"
Requires-Dist: livekit-plugins-fishaudio>=0.1.0; extra == "voice"
Requires-Dist: onnxruntime>=1.18.0; extra == "voice"
Requires-Dist: openwakeword>=0.6.0; extra == "voice"
Requires-Dist: pynvml>=11.5.0; extra == "voice"
Requires-Dist: pyjwt>=2.8.0; extra == "voice"
Requires-Dist: llama-cpp-python>=0.3.0; extra == "voice"
Requires-Dist: soundfile>=0.12.0; extra == "voice"
Requires-Dist: kokoro-onnx>=0.4.0; extra == "voice"
Requires-Dist: transformers>=4.48.0; extra == "voice"
Requires-Dist: websockets>=15.0; extra == "voice"
Requires-Dist: faster-whisper>=1.0.0; extra == "voice"
Provides-Extra: transport
Requires-Dist: aioquic>=1.0.0; extra == "transport"
Requires-Dist: qh3>=1.0.0; extra == "transport"
Provides-Extra: ui
Requires-Dist: textual>=0.85.0; extra == "ui"
Requires-Dist: tomlkit>=0.13.0; extra == "ui"
Requires-Dist: watchdog>=4.0.0; extra == "ui"
Dynamic: license-file

# Syne

**Voice + avatar conscious companion. The other half of the real-time local voice agent experiment.**

> Built on top of [Mnemosyne](https://github.com/synet-systems/mnemosyne). This is an experiment in pushing the latency floor of fully-local, real-time voice agents on a single consumer GPU.

---

## What this is

Syne is the **mouth, ears, and face** of a voice agent. It runs as a 5-process tree (supervisor → cortex / voice / swarms / avatar / emotion) on top of a single Mnemosyne engine, with one LiveKit WebRTC session at a time.

The goal of the experiment:

> *Can a fully-local voice agent on a single RTX 3060 beat a hosted OpenAI Realtime API on perceived latency, while running 100% offline?*

We don't have a final answer. The repo shows where we are.

---

## What works today

- **LiveKit voice loop** with VAD + dual STT (Moonshine partials, Parakeet final)
- **LFM 1.2B** Tier-1 responder (60+ tok/s on Pascal, fits in 1 GB VRAM)
- **Qwen 2B** Tier-2 planner with JSON-schema-constrained output
- **Tiered degradation**:
  - Tier 1: pod-routed (MI300X, ~270ms network)
  - Tier 2: local Ollama when pod unreachable
  - Tier 3: WAL queue when both local + pod fail
- **Kokoro + CosyVoice2 + Orpheus** TTS pools with barge-in cancellation
- **FLAME point-cloud avatar** in the terminal (or rendered to a frame file)
- **Adaptive backchannel pool** that learns from your engagement (mhm-hmm frequency, not random)
- **Hedge injection** at low classifier confidence
- **Tool dispatch with consent**: set_timer always prompts, web.fetch prompts once per session, memory.search is safe
- **LiveKit QR pairing** for phone-as-microphone
- **Persistent supervisor** with 3-crash-in-30s halt, heartbeat watcher, WAL replay on recover

**Perceived latency** (local, warm, voice loop): **~500-700 ms**. Hard ceiling 1.2s.

---

## What we're still figuring out

- Whether **specular backchannel** (predicting the user's "uh huh" before they say it) actually improves the UX or just feels weird
- Whether **memory residues** (failed tool calls, abandoned plans) should become first-class memories or stay ephemeral
- Whether **the persona drift timer** (daily background task that nudges the agent's voice/personality based on long-term memory) is net-positive or just weird at 3am
- How to **scale to 4 concurrent voice sessions** on one GPU without losing latency (the 1-session benchmark is at the limit of Pascal)
- Whether **Qwen flash-attention** can be safely enabled on sm_70+ (it segfaults on Pascal sm_61 — see `llm/pool.py:qwen_flash_attn`)

---

## TL;DR

```bash
# Build the system
cd syne_rs && cargo build --release
cd ../syne
./bootstrap.sh                         # creates .venv, installs deps
source .venv/bin/activate

# Run the supervisor (Python fallback mode for now)
SYNE_SUPERVISOR_PYTHON=1 python -m syne.supervisor
# → starts cortex, voice, swarms, avatar, emotion as children

# Drive it via the LiveKit dev server (or your own)
# Open the WebRTC client at http://localhost:7880
# Talk. The agent listens, thinks, talks.
```

For the full setup, see [the in-repo .kiro/specs/syne/](syne/.kiro/specs/syne/) (the captured plan log).

---

## Architecture (60-second tour)

```
┌───────────── LiveKit WebRTC session ─────────────┐
│                                                  │
│   You speak ──► 20ms int16 frames @ 16kHz       │
│                                                  │
│   Mic ──► Silero VAD ──► TurnDetector ──► Dual-STT
│                                       (Moonshine + Parakeet)
│                                                  │
│   ──► Prosody extractor (YIN pitch, RMS)       │
│   ──► Ambient monitor (YAMNet)                  │
│   ──► AudioCapture (sha, mnemosyne ref)         │
│                                                  │
│   On turn end:                                   │
│   Tier-0 classifier (~50ms)                     │
│     ├ Tier 1: LFM 1.2B stream (~200ms)         │
│     └ Tier 2: Qwen 2B planner (when tools)     │
│   ──► OutputVerifier                             │
│   ──► TTSRouter (Kokoro / CosyVoice2 / Orpheus) │
│   ──► Speaker                                    │
│                                                  │
└──────────────────────────────────────────────────┘
                ▲                       ▲
                │ HTTP/MCP              │ in-process
                │                       │
   ┌────────────┴───────────┐  ┌────────┴────────┐
   │   Mnemosyne (sister)   │  │  Rust supervisor │
   │   memory + retrieval   │  │  tree (5 procs)  │
   └────────────────────────┘  └──────────────────┘
```

---

## Repository layout

```
syne/
├── syne/                      # The companion package (Python)
│   ├── voice/                 # LiveKit loop, STT, TTS, prosody, sentence buffer
│   ├── cortex/                # BrainServer, MemoryReflex, opt-out, retention
│   ├── avatar/                # FLAME point-cloud renderer
│   ├── swarms/                # Multi-agent (Claude Code, Kiro, OpenClaw, Hermes)
│   ├── safety/                # BudgetLedger, WAL, rollback, tripwires
│   ├── self_edits/            # Tier-7 self-edit gates (Darwin Gödel Machine)
│   ├── scripts/               # Operational scripts + systemd units
│   ├── cli.py                 # `syne` CLI
│   ├── supervisor.py          # Python fallback for the Rust supervisor
│   ├── bootstrap.sh           # One-shot setup
│   ├── pyproject.toml
│   └── ...
├── syne_rs/                   # Rust workspace (6 crates)
│   ├── crates/
│   │   ├── supervisor/        # PID spawn + heartbeat watcher
│   │   ├── router/            # Sub-millisecond LLM-call router
│   │   ├── swarm_core/        # Per-swarm state machine
│   │   ├── audio_dsp/         # AEC, noise suppression, AGC
│   │   ├── wal/               # Write-ahead log (SQLite + Rust)
│   │   └── pyo3_bindings/     # PyO3 for Python interop
│   ├── Cargo.toml
│   └── ...
├── .kiro/specs/syne/          # The phase log (P0..P9.4)
├── tests/                     # 1736 tests, 99.3% pass
├── packages/syne-py/          # Thin PyPI wrapper around ../syne/
├── .github/workflows/         # release.yml (binary build)
├── LICENSE                    # MSAL v1.0
├── CHANGELOG.md
└── README.md                  # ← you are here
```

---

## The "real-time voice agents locally" experiment

**Stack we picked and why**:

| Choice | Why |
|---|---|
| **LiveKit** over raw WebRTC | Mature Python client, server in Go/Rust, free for self-host, scales to mobile later |
| **Parakeet TDT 0.6B** as primary STT | Sub-100ms partials, onnxruntime, runs on Pascal sm_61 (GTX 1050 Ti) |
| **Moonshine** as fallback STT | 4x faster than Whisper, similar quality, smaller |
| **LFM 2.5 1.2B Q4_K_M** as Tier-1 | 65 tok/s on Pascal, fits 1 GB VRAM, good enough for short replies |
| **Qwen 2B Q4_K_M** as Tier-2 planner | Structured JSON output via GBNF grammar |
| **Kokoro 82M ONNX** as TTS-1 | 0.2s TTFB, 24 voices, runs on CPU |
| **CosyVoice2 0.5B Q8** as TTS-2 | Higher quality, slower, 600 MB VRAM |
| **Ollama** as fallback LLM server | Bundles CUDA 12 libs (Arch ships CUDA 13.2 which is Pascal-broken) |
| **Postgres + Qdrant** as the substrate | Battle-tested, can be self-hosted |
| **SQLite WAL** as Tier-3 queue | No deps, durable, replayable |

**What we tried that didn't work**:

- Whisper large-v3 — too slow (1.2s partials)
- Bark TTS — robotic, slow
- LFM 7B — too big for Pascal, OOMs
- Qwen 7B planner — same
- PyTorch on Pascal sm_61 — segfaults under flash-attention
- Building llama-cpp-python from PyPI — links against CUDA 13 (incompatible)
- Self-hosting Whisper.cpp — too slow

The full Q&A log is in `syne/.kiro/specs/syne/`.

---

## For the impatient

```bash
git clone https://github.com/synet-systems/syne.git
cd syne
./bootstrap.sh
source .venv/bin/activate
SYNE_SUPERVISOR_PYTHON=1 python -m syne.supervisor
```

You should see:

```
2026-06-14 12:40:36  syne.supervisor spawning syne-cortex
2026-06-14 12:40:36  syne.supervisor spawning syne-voice
2026-06-14 12:40:36  syne.supervisor spawning syne-swarms
2026-06-14 12:40:36  syne.supervisor spawning syne-avatar
2026-06-14 12:40:36  syne.supervisor spawning syne-emotion
2026-06-14 12:40:37  [voice] syne-voice starting
2026-06-14 12:40:37  [swarms] syne-swarms starting
2026-06-14 12:40:37  [avatar] syne-avatar starting
2026-06-14 12:40:37  [emotion] syne-emotion: backend=stub ready
2026-06-14 12:40:37  [emotion] listening on /run/user/1000/syne/emotion.sock
```

(One of the 5 will likely fail to fully start unless you've built llama-cpp-python against the bundled CUDA 12 libs — see `syne/llm/llama_cuda_runtime.py` for the recipe. The rest degrades gracefully.)

---

## Contributing

This is an experiment. **PRs welcome**, especially if you have:
- Latency measurements on different GPUs
- A working LiveKit deployment you can test against
- Backchannel-pool training data (recordings of *you* using a voice agent)
- Better sentence buffer / prosody algorithms
- A working flash-attn kernel for Pascal (please)

Issues with the `voice-loop` label get the fastest response.

## License

This monorepo: **MSAL v1.0** — see [LICENSE](LICENSE).
The published `syne` PyPI package: **MIT** — see [LICENSE-MIT](../synet-systems/blob/main/packages/syne-py/LICENSE-MIT).
Dual-license explanation: [NOTICE](NOTICE).
