Metadata-Version: 2.4
Name: audient
Version: 0.1.0
Summary: An audio perception layer for agents — continuous-listening, event-gated, with a concept memory that grows through use.
Project-URL: Homepage, https://github.com/es617/audient
Project-URL: Repository, https://github.com/es617/audient
Project-URL: Issues, https://github.com/es617/audient/issues
Author: Enrico Santagati
License: Apache-2.0
Requires-Python: <3.14,>=3.12
Requires-Dist: birdnetlib>=0.17
Requires-Dist: fast-agent-mcp>=0.7.12
Requires-Dist: fastapi>=0.110
Requires-Dist: faster-whisper>=1.0
Requires-Dist: fastmcp>=0.4
Requires-Dist: librosa>=0.10
Requires-Dist: numpy
Requires-Dist: pydantic>=2
Requires-Dist: pyyaml
Requires-Dist: scipy
Requires-Dist: silero-vad>=5.1
Requires-Dist: sounddevice>=0.5.5
Requires-Dist: soundfile
Requires-Dist: sqlite-vec>=0.1
Requires-Dist: sse-starlette>=2.0
Requires-Dist: tensorflow>=2.16; sys_platform == 'darwin'
Requires-Dist: tflite-runtime>=2.14; sys_platform == 'linux' and python_version < '3.12'
Requires-Dist: torch>=2.2
Requires-Dist: transformers>=4.40
Requires-Dist: uvicorn>=0.27
Requires-Dist: websockets>=12
Description-Content-Type: text/markdown

# audient

An experimental **audio perception layer for agents**: continuous-listening,
event-gated, multi-layer classification, and dynamic memory.

LLMs handle text natively. They handle audio when it's speech, via
transcription or audio-native models. They don't hear the rest — a bird
singing outside while you read, a glass breaking two rooms away, a
smoke-alarm ringing. audient gives an agent ears to detect and classify
audio events using specialized models and reasoning.

Under the hood it's an **MCP server** (stdio + HTTP at `/mcp`) plus
a **FastAPI REST + WS + SSE adapter** on `127.0.0.1:8089`. Drive it
from an MCP-capable agent (Claude Desktop, `fast-agent`), access the
knowledge through the HTTP/SSE endpoints, or run the bundled
[`frontend/`](frontend/) UI to watch the loop in real time and close
it in a browser.

![audient's Live surface: audio scope on top, live classifier reasoning streaming in a side panel, memory grid pulsing as recognized events land](https://raw.githubusercontent.com/es617/audient/main/docs/hero.gif)

For the story behind the design, the biology parallel, and a walkthrough
of the growth loop with screenshots: [Let the AI Out: audient — an
ambient audio perception layer that grows](https://es617.dev/2026/07/07/audient-ears/).
For a 90-second demo of the memory going from empty → auto-recognizing
in real time: [demo on YouTube](https://www.youtube.com/watch?v=6ofyj-2gQ8c).

---

## What audio actually needs

An agent that only handles audio-as-speech misses most of what audio
carries. Useful things it can't reach today:

- **Ambient awareness** — what's happening around the agent. Music
  in another room. Footsteps in the hall. The HVAC kicking on.
- **Event detection** — alarms, doorbells, a phone ringing two
  floors down, glass breaking, a baby crying.
- **Voice beyond the words** — stress in a voice on a call, tone
  shifts, who's actually speaking, the background of a meeting.
- **Anomaly detection** — equipment that just changed pitch. A pet
  that sounds different. Construction next door that wasn't there
  yesterday.

There ARE specialized models for each of these. The problem isn't
models, it's orchestration: today you decide which one to run on
which clip and stitch the answers together yourself. The agent isn't
in the loop.

What this project aims at: an agent with the audio channel always
on. It decides what's worth attending to, recognizes what it's heard
before, escalates the rest to whatever specialist fits, and learns
the things it didn't know. The longer view: an agent that can deploy
new specialists on its own — and eventually train them.


## Four stages

```
    continuous audio
          │
          ▼
  ┌─────────────────────────────────────┐
  │  Gate                               │
  │  energy / novelty / VAD             │
  └──────────────┬──────────────────────┘
                 ▼
  ┌─────────────────────────────────────┐
  │  Fingerprint                        │
  │  CLAP embedding + symbolic features │
  └──────────────┬──────────────────────┘
                 ▼
  ┌─────────────────────────────────────┐
  │  Memory  (top-k cosine)             │
  └──┬──────────────────────────────┬───┘
     │ strong match                 │ weak / miss
     │                              ▼
     │            ┌─────────────────────────────┐
     │            │  Agent                      │
     │            │   AST coarse (+ spectral)   │
     │            │   → Specialist if AST hints │
     │            │     at a known domain       │
     │            │     (e.g. Whisper, BirdNET) │
     │            └─┬───────────────────────┬───┘
     │              │ recognized            │ nothing fits
     ▼              ▼                       ▼
  ┌──────────────────────────┐      ┌──────────────┐
  │  Record event            │      │  Unhandled   │
  │  (extends memory)        │      └──────┬───────┘
  └──────────────────────────┘             │ labeled
                                           ▼
                                       (Memory)
```

The pipeline coarsely mirrors the way human hearing works.

1. **Gate** — `detect_events`, `vad`, energy/novelty/spectral
   segmentation. Trims the continuous signal to candidate regions.
   Most of the audio never reaches the rest of the system. *(Hearing:
   cochlea + brainstem amplify onsets and transients, attenuate
   steady background.)*

2. **Fingerprint** — each candidate gets a CLAP embedding (dense,
   comparable) plus a panel of symbolic features (spectral centroid,
   peaks with prominence, harmonics, noise floor, mains-hum,
   clipping, temporal stability). Embedding is for retrieval;
   symbolic features are what the agent reads and reasons about in
   plain English.
   *(Hearing: primary auditory cortex turns the signal into a
   representation downstream areas can use.)*

3. **Recognize** — sqlite-vec top-k cosine over learned concepts.
   Strong match against a calibrated concept → labeled server-side.
   On a miss, the agent reaches for AST (527-label AudioSet
   classifier) as broad routing, sometimes alongside spectral
   features for disambiguation. AST is coarse — the agent chains a
   narrow specialist (e.g. Whisper for speech, BirdNET for bird
   vocs; the set is arbitrary) when AST hints at a known domain. *(Hearing: specialized regions
   — speech, music, environmental sound — see the fingerprint in
   parallel, each with its own memory; "recognize" is what lights up
   where.)*

4. **Unhandled** — if nothing recognizes it, the event lands in
   `list_unhandled` with whatever the system thinks it heard.
   Someone (an agent or a person) labels it once via
   `label_unhandled`; that label becomes a concept; next time the
   memory check matches. *(Hearing: you don't recognize it; you look
   at the source or ask. Once you have a label, you recognize it
   next time.)*

The agent always *perceives* — every event flows out as it happens,
whichever stage produced it. What varies is who's *classifying*: the
server for strong memory matches (cheap, no LLM call), the agent for
everything else.

Stage 1 also has a **fast lane for speech.** Silero VAD runs in
parallel with energy/novelty segmentation; when it fires, the region
skips CLAP → memory → specialist routing and goes straight to
Whisper. This treats VAD as a frozen, always-on "speech" concept —
the analog of recognizing a familiar voice before parsing the words.
If Whisper returns a transcript, the event is recorded as speech
directly. If Whisper returns nothing usable (silence, or non-speech
audio that fooled VAD — music, animal calls, sustained tones), the
region falls through to the same ambient classifier path everything
else takes; no audio is silently dropped.

## Three ways to reason about a sound

**The agent reasons over symbolic representations, never raw
audio.** Three layers are exposed for it to reason across:

- **The signal.** `summarize`, `spectrum`, `stats` return plain
  numbers — RMS, SNR, noise floor, spectral centroid, rolloff,
  bandwidth, flatness, top spectral peaks with prominence in dB,
  harmonic ratio, zero-crossing rate, mains-hum harmonics, kurtosis,
  clipping, and a temporal-stability block. The agent reads these as
  JSON and reasons about them the way an audio engineer would.
- **The embedding.** The agent never sees the CLAP vector itself.
  It sees the *similarity score* against everything in concept
  memory — `query_memory` returns the top-k matches with cosine
  similarity, threshold, and recognizer kind.
- **The labels.** Specialists return structured labels. Whisper
  transcripts. BirdNET species with confidence. AST's coarse guess
  across 527 categories. The agent hands a sound off via
  `invoke_specialist` and reads back the structured answer alongside
  the specialist's plain-English contract for interpretation.

The agent can disagree with any single layer — override an embedding
similarity if the labels suggest divergent categories, override a
specialist if the signal features don't fit. No single layer is
authoritative.

---

## Install

### From PyPI (usual path)

Requires Python 3.12 or 3.13 on macOS or Linux. The wheel bundles both
the server and the browser UI, so one install gets you everything.

```bash
pip install audient
export ANTHROPIC_API_KEY=sk-...   # only for --classifier=internal (default)
audient --no-stdio
```

Then open <http://127.0.0.1:8089/> and grant microphone permission.
First run downloads ~600MB of model weights from Hugging Face; expect
20–40s of warm-up on cold cache.

Skip the API key by running the server without the built-in classifier
(memory-first fast path still works for promoted concepts; new sounds
land in the unhandled bucket for later labeling):

```bash
audient --no-stdio --classifier=external
```

### From source (dev workflow)

Clone + `make demo` for the two-process dev setup with Vite hot
reload and the server side-by-side.

```bash
git clone https://github.com/es617/audient
cd audient
(cd server && uv sync)
(cd frontend && pnpm install)
export ANTHROPIC_API_KEY=sk-...
make demo
```

Server on `:8089`, Vite dev on `:5173`. Open <http://localhost:5173/>
for the hot-reloading UI. For server-only smoke tests, headless mic
mode, or the run-mode matrix, see
[`server/README.md`](server/README.md#run-modes); for the per-surface
frontend walkthrough, see [`frontend/README.md`](frontend/README.md).

## Driving audient from Claude Code (or any stdio MCP client)

The default entry point (`uv run audient`) speaks stdio MCP, which is
what Claude Code and most MCP hosts consume. Add it to your client
config:

```json
{
  "mcpServers": {
    "audient": {
      "command": "uv",
      "args": ["--directory", "/absolute/path/to/audient/server", "run", "audient"]
    }
  }
}
```

The agent gets the full MCP tool surface — `load_audio`, `summarize`,
`spectrum`, `detect_events`, `vad`, `invoke_specialist`,
`query_memory`, `record_event`, `teach`, `list_unhandled`,
`label_unhandled`, etc. Tool calls land server-side and modify the
SQLite concept memory.

A couple of caveats worth knowing:

- **Claude Code does not surface MCP server notifications.** The
  protocol supports `ResourceUpdatedNotification` (and audient fires
  them on new events / candidates), but Claude Code's UI doesn't
  render them as in-conversation messages. To see "what happened
  since I last asked," have the agent call `list_events` or
  `list_stream_candidates`.
- **The bundled frontend still works in parallel.** The FastAPI / SSE
  adapter on `127.0.0.1:8089` is part of the same process. So anything
  Claude Code writes (a new event, a corrected label, a taught concept)
  shows up live in the frontend timeline the moment it lands. The two
  clients drive the same server; only Claude Code is blind to live push.

## Known limitations

audient is experimental. It's been lightly evaluated on recorded data
during development; no formal benchmark suite or public-dataset numbers
yet. Other limitations:

- **Single-source assumption.** The pipeline treats each detected
  region as one event. Overlapping sounds — a bird call while music
  plays, a doorbell mid-conversation — are classified as whichever
  signal dominates the embedding or wins the VAD fast lane. The
  non-dominant event is lost. Source separation isn't wired up.

- **macOS-first.** Everything has been tested on macOS. The
  dependencies (torch, transformers, faster-whisper,
  sounddevice/portaudio) all have Linux wheels, so Linux should work
  with minimal fuss; I just haven't verified. Windows needs a Windows
  branch added to the BirdNET dep gate in `pyproject.toml` and
  probably a look at the mic-source path; nothing about the models
  themselves is Windows-incompatible. The WebSocket `/ingest`
  endpoint and the bundled frontend's browser-mic capture are
  platform-agnostic, so a Windows user can drive audient through a
  browser today — the backend side is the friction.

- **Classifier prompt tuned for Anthropic.** The in-process
  classifier runs through [fast-agent](https://github.com/Evalstate/fast-agent),
  so any supported provider (Anthropic, OpenAI, Gemini, Ollama-local)
  will execute, but the prompt was written and validated against
  Claude models. Switching providers may need prompt adjustments.

## Acknowledgements

audient stands on a lot of other people's work. The load-bearing pieces:

- **MCP + agent infra** — [FastMCP](https://github.com/jlowin/fastmcp)
  (Jeremiah Lowin), [fast-agent](https://github.com/Evalstate/fast-agent)
  (Shaun Smith / Evalstate).
- **Models** — [LAION-CLAP](https://github.com/LAION-AI/CLAP) for audio
  embeddings; [AST](https://github.com/YuanGongND/ast) (Yuan Gong et al)
  as the AudioSet coarse classifier;
  [Whisper](https://github.com/openai/whisper) (OpenAI) via
  [faster-whisper](https://github.com/SYSTRAN/faster-whisper);
  [BirdNET](https://github.com/kahst/birdnetlib) (Cornell Lab of
  Ornithology) via birdnetlib;
  [Silero VAD](https://github.com/snakers4/silero-vad).
- **Model hosting** — [Hugging Face](https://huggingface.co/) +
  `transformers` for CLAP loading.
- **Storage** — [sqlite-vec](https://github.com/asg017/sqlite-vec)
  (Alex Garcia).
- **Audio infra** — librosa, soundfile, sounddevice.
- **UI** ([`frontend/`](frontend/)) — React, Vite, Tailwind,
  TanStack Query, shadcn/ui, Zustand.

## License

Apache License 2.0 — see [LICENSE](LICENSE).
