Metadata-Version: 2.3
Name: ark-audio-agent
Version: 0.1.11
Summary: Autonomous text-to-music composer driven by a multi-agent ReAct harness (Claude API, with an offline deterministic fallback).
Keywords: music,generative,agents,react,claude,midi,synthesizer
Author: Immanuel R
Author-email: Immanuel R <raj@immanuel.co>
License: MIT
Requires-Dist: numpy>=1.24
Requires-Dist: mido>=1.3
Requires-Dist: rich>=13.7
Requires-Dist: anthropic>=0.107.1 ; extra == 'llm'
Requires-Python: >=3.10
Project-URL: Homepage, https://github.com/ir-dev/ark-audio-agent
Project-URL: Repository, https://github.com/ir-dev/ark-audio-agent
Project-URL: Issues, https://github.com/ir-dev/ark-audio-agent/issues
Provides-Extra: llm
Description-Content-Type: text/markdown

# ark-audio-agent

**Autonomous text → music.** Give it a sentence; it reasons about the feeling,
composes a structured piece, and renders it to MIDI + audio — with no human in
the loop. Ask it to, and it sings your words too.

The interesting part is the *harness*: composition is driven by a team of
specialist agents running a **ReAct loop** (reason → act → observe) over a shared
musical blackboard. It runs on the **Claude API** when a key is available, and on
a **fully-offline deterministic music-theory brain** otherwise, so it always
produces a result.

```bash
ark-audio "a rainy night in a neon city, lonely but hopeful"
# → renders/20261011-132450_a-rainy-night-in-a-neon-city-lonely-but/song.mp3  (+ .wav, .mid, report.md, trace.jsonl)
```

Add a **direction** to say how the piece must be built, in plain words. The text
still sets the mood:

```bash
ark-audio "The Lord is my shepherd; I shall not want." \
  --direction "a few secs of intro and a bit of rising notes before the main melody, \
  structured, fewer long leaps, strongly leaning toward one chord and tone; after the \
  climax a falling melody, then a few sec of outro to end it"
```

Name **reference music** to draw on: a piece from the built-in library, a style, or your
own MIDI file. The piece borrows its traits (chords, melodic shape, palette, groove), not
its notes:

```bash
ark-audio "a rainy afternoon in an empty flat" -r satie --borrow harmony,melody,tempo
ark-audio "lamps lit at dusk by the river" -r raga-yaman --borrow all
ark-audio "develop my sketch" -r ./idea.mid --strength 0.8
```

Add `--sing` to turn the text into a **song**: the words become the lyrics, a built-in voice
sings them one syllable per note, and the MIDI and a `lyrics.lrc` sheet carry them. It is off
by default, so a piece is instrumental unless you ask:

```bash
ark-audio --text-file psalm23.txt --sing
```

> **Building an agent or app on top of this?** Start with the
> [prompting guide](#prompting-guide-for-developers) below. [docs/AI_SPEC.md](docs/AI_SPEC.md)
> is the full contract: every parameter, nuance and conflict. [docs/PROMPTS.md](docs/PROMPTS.md)
> has the system prompt and Claude, OpenAI-compatible, MCP and CLI wiring.
> `ark_audio_agent.spec` ships the tool definitions and a `run_tool` dispatcher.

**Contents:** [What the AI does](#what-the-ai-does) ·
[Quick start](#quick-start) · [CLI cookbook](#cli-cookbook) ·
[Prompting guide for developers](#prompting-guide-for-developers) ·
[Python library](#python-library) · [Musical direction](#musical-direction) ·
[Reference music](#reference-music) · [Sung lyrics](#sung-lyrics) ·
[Tokens and cost](#tokens-and-cost) · [Outputs](#outputs) ·
[How it works](#how-it-works) · [Development](#development)

---

## What the AI does

With an Anthropic API key (or Claude on Microsoft Foundry), five Claude agents compose
the piece. Each one reasons with adaptive thinking and acts only through tools on a
shared score, so every musical decision is a tool call you can inspect afterwards.

| Capability | What happens |
|---|---|
| **Reads feeling from any text** | The Interpreter reads a verse, a poem, a scene or three words for mood, imagery, emotional arc and genre. It then picks a key, mode and tempo, and names the piece. |
| **Works as a team of specialists** | The Composer handles form, chords and the dynamic arc; the Melodist, contour, register and motifs; the Arranger, instruments, bass and drums. Each runs its own reason → act → observe loop. |
| **Follows plain-English direction** | "a few secs of intro, rising notes, fewer long leaps" becomes a section plan with hard limits. The Interpreter checks the parser's reading against your words and corrects it with `set_direction`. |
| **Chooses its own references** | It searches 29 references (public-domain works, genre archetypes and Indian styles), pins the ones that fit, and records why and what to borrow. It can also develop your own MIDI sketch. |
| **Sings the words, when asked** | With `--sing` the text becomes the lyric. The Melodist decides which lines each section sings (`set_lyrics`), the Composer gives them room, and the engine sets one syllable per note. A formant voice sings the result, and the MIDI carries the lyrics. |
| **Critiques and revises** | The Critic runs `analyze_score`, which measures the rendered piece against every requirement, the sung words included. It sends one clear problem back to the agent that owns it, for up to `--max-revisions` rounds. |
| **Shows its work** | `trace.jsonl` holds every thought, tool call and observation. `report.md` marks which decisions an LLM made, with the model and effort, and lists every agent run with its tokens and a total. The CLI prints the same table. |
| **Plugs into your own AI** | Tool definitions for Claude and OpenAI-compatible models, a `run_tool` dispatcher that never raises, a tested system prompt, an MCP server sketch and `ark-audio --spec`. |
| **Always finishes** | Effort runs from `low` to `max`. Calls degrade step by step on a capability mismatch, and with no key a deterministic brain makes the same tool calls offline. |

### Claude or offline: what changes

| | Claude multi-agent | Offline deterministic |
|---|---|---|
| Reading the text | open language understanding | a lexicon of ten feelings ([the words it knows](#writing-the-text)) |
| Direction | parsed, then reviewed and corrected by the Interpreter | the parsed plan is final |
| Reference music | searched, chosen and justified by the Interpreter | the best match, pinned at strength 0.5 |
| Critic | judges mood, arc and balance, plus every direction and lyric check | direction and lyric checks and a few balance rules |
| Sung lyrics (`--sing`) | the Melodist places the lines, choosing refrains | lines in order through the verse and chorus, the chorus repeating them |
| Reproducible | the seed fixes the notes; the agents' choices vary | byte-identical `score.json` for the same inputs and seed |
| Time and cost | minutes and real tokens, counted per agent run (`--effort low` is much cheaper) | about 1–3 s, free (0 tokens) |

---

## Quick start

Install from PyPI. The `[llm]` extra adds the anthropic SDK for the Claude brain:

```bash
pip install "ark-audio-agent[llm]"
ark-audio "a quiet dawn over still water" --offline     # no key needed: ~2 s, reproducible
```

Then turn on Claude. Put **one** provider in a `.env` file in the folder you run from;
the CLI loads it by itself, so there is nothing to `export`:

```bash
# .env  (from a checkout: cp .env.example .env)
ANTHROPIC_API_KEY=sk-ant-...
```

```bash
ark-audio "triumphant sunrise over the mountains"
```

Check the start-up panel: it should read `brain: claude multi-agent (...)` and
`env: .env`. If it says `deterministic (offline — ...)`, the text after the dash
names what's missing. A variable already set in your shell overrides the file.

- **Offline only:** `pip install ark-audio-agent`.
- **MP3 export:** install `ffmpeg` (`brew install ffmpeg`, `sudo apt install ffmpeg`,
  or `winget install Gyan.FFmpeg`). Without it you still get the WAV and MIDI.
- **From a checkout:** `uv sync` installs everything, the anthropic SDK included (see
  [Development](#development)). Prefix commands with `uv run` unless the venv is active.

---

## CLI cookbook

Every command below was checked against this release (offline). The ones that name a file
(`psalm23.txt`, `brief.txt`, `./idea.mid`, an env file) need that file to exist. Add
`--offline` to any of them for a fast, reproducible run without a key.

### First runs

```bash
ark-audio "a quiet dawn over still water"                     # Claude if a key is set, else offline
ark-audio "a quiet dawn over still water" --offline --seed 7  # reproducible: same seed, same piece
ark-audio "a quiet dawn over still water" --offline --seed 8  # another take of the same idea
echo "a storm breaking over the sea" | ark-audio --offline    # text from stdin
ark-audio --text-file psalm23.txt                             # text from a file
```

### Shape it with a direction

```bash
# a song form with a build and a release
ark-audio "For God so loved the world" \
  -d "a 6 second intro, rising notes into the verse, a climax, then a falling melody and a short outro"

# background music to a length
ark-audio "a calm, still meditation; slow breathing, soft light" \
  -d "background music for a meditation video: a short intro, a main part, then a long outro. 60 seconds long overall, slow"

# a pop structure at a set tempo
ark-audio "a summer night, friends, windows down" -d "120 bpm, a verse, a big chorus, a bridge, then a short outro"

# a stated key, a singable line and phrase structure
ark-audio "a lullaby for a sleeping child" -d "in F major, stepwise, structured 4-bar phrases, a long outro"

# a dark piece with an arch
ark-audio "a lonely lighthouse in winter" \
  -d "in D minor, 80 bpm, a short intro, a verse, a climax, then a falling melody and a long outro"

# hypnotic: grounded on one chord, no intro
ark-audio "a hypnotic desert night" -d "no intro, a main part, strongly leaning toward one chord and tone, a short outro, 70 bpm"

# podcast bed
ark-audio "a podcast about the ocean" -d "bgm for a podcast, a 6 second intro, a verse, a short outro"

# a long direction kept in a file
ark-audio "For God so loved the world" --direction-file brief.txt
```

The CLI prints each requirement with ✓ or ✗ when the piece is done, and `report.md`
keeps the same table.

### Draw on reference music

```bash
ark-audio --list-references                   # the whole library: id, key, tempo, meter, character
ark-audio --list-references rainy piano       # ranked for those words

# the same chords, your own tune
ark-audio "a sad farewell at a train station in the rain" -r pachelbel-canon --borrow harmony

# a waltz: borrowing tempo brings the 3/4 meter with it
ark-audio "wistful, a memory of an old dance hall" -r greensleeves --borrow tempo,harmony,melody,instruments

# two references, followed loosely
ark-audio "a quiet evening walk" -r satie -r debussy-clair-de-lune --strength 0.4

# everything from one reference, key and tempo included
ark-audio "victory at dawn" -r epic-trailer-build --borrow all

# develop your own MIDI sketch and keep its key
ark-audio "develop my sketch" -r ./idea.mid --borrow tonality,harmony,melody,tempo --strength 0.8

# purely original: stop the agents picking a reference themselves
ark-audio "a quiet dawn over still water" --no-reference
```

### Indian styles

A raga is its scale, so borrow `tonality` too (or `all`). Otherwise the text's mood picks
the mode and only the phrases, palette and groove carry over.

```bash
ark-audio "lamps lit at dusk by the river" -r raga-yaman --borrow tonality,melody,instruments
ark-audio "a farewell at dawn" -r raga-bhairavi --borrow tonality,tempo,melody,instruments   # 6/8 dadra
ark-audio "a temple festival morning" -r carnatic-kriti-mohanam --borrow all
ark-audio "a monsoon love song" -r bollywood-romantic-ballad
ark-audio "a punjabi wedding baraat" -r bhangra-dhol --borrow all
ark-audio "a night of devotion at the dargah" -r sufi-qawwali -d "a short intro, build up, a climax, then a short outro"
```

### Sing the words

```bash
ark-audio "The Lord is my shepherd; I shall not want." --sing                 # a short lyric, sung as a refrain
ark-audio --text-file poem.txt --sing                                         # line breaks become sung lines
ark-audio --text-file poem.txt --sing -d "a short intro, a verse, a chorus, a short outro, 90 bpm"
ark-audio --text-file psalm23.txt --sing -r worship-ballad                     # a hymn-like setting
ark-audio "Twinkle, twinkle, little star" --sing --offline --seed 3          # another take, reproducible
```

The CLI shows the line and syllable count up front, and after the run a ✓ or ✗ for each lyric
check. Open `song.mp3` to listen, `lyrics.lrc` in a player that shows lyrics, or `song.mid` in a
DAW, where the `vocal` track carries the words. More under [Sung lyrics](#sung-lyrics).

### Tune the Claude brain

```bash
ark-audio "a cathedral of light at the edge of the world" --effort high      # deeper reasoning
ark-audio "a cathedral of light at the edge of the world" --effort low --max-revisions 0   # fastest, cheapest
ark-audio "a cathedral of light at the edge of the world" --model claude-sonnet-5-5
ark-audio "a cathedral of light at the edge of the world" --env-file ~/.config/ark-audio/foundry.env
```

`ARK_MODEL`, `ARK_EFFORT` and `ARK_MAX_TOKENS` in `.env` set the same defaults. On
Foundry, `--model` is your deployment name. Each agent's line shows the tokens it used, and the
run ends with a table of every agent run and the total (see [Tokens and cost](#tokens-and-cost)).

### Output and sound

```bash
ark-audio "lonely synthwave drive at 2am" --out renders/drive --no-mp3   # pick the folder, skip ffmpeg
ark-audio "a storm breaking over the sea" --reverb 0.35                  # wetter (default 0.16)
ark-audio "a storm breaking over the sea" --reverb 0                     # dry
```

### Scripts and other AI tools

```bash
ark-audio --spec > ark-audio.spec.json        # tools, enums, precedence, conflicts, system prompt
ark-audio --spec | jq '.tools[].name'         # "compose_music", "list_reference_music"

# three takes of one idea, side by side
for seed in 1 2 3; do
  ark-audio "a quiet dawn over still water" --offline --seed $seed --out renders/takes/$seed
done
```

Exit codes: `0` success · `2` bad input (no text, unknown reference, bad `--borrow`,
strength out of range, more than 3 references, unreadable env file) · `130` interrupted.

### Every flag

| Flag | What it does | Example |
|---|---|---|
| `TEXT` · stdin · `--text-file F` | the mood | `ark-audio "a storm at sea"` |
| `-d`, `--direction` · `--direction-file F` | how the piece is built (the file wins over `-d`) | `-d "a short intro, a climax, a long outro"` |
| `-r`, `--reference` | music to draw on: id, title, composer or `.mid` path; up to 3 | `-r satie -r ./idea.mid` |
| `--borrow` | what to take: aspects, `all` or `none` | `--borrow harmony,melody` |
| `--strength` | how closely, 0–1 (default 0.7 for named references, 0.5 for the agents' picks) | `--strength 0.4` |
| `--no-reference` | stop the agents picking references themselves | |
| `--sing` | sing the text as the lyrics (default: off, instrumental) | `--sing` |
| `--list-references [WORDS]` | print the library, ranked for WORDS, and exit | `--list-references blues` |
| `--offline` | force the deterministic brain | |
| `--seed N` | the take (default 42) | `--seed 7` |
| `--model ID` | Claude model, or Foundry deployment (default `claude-opus-5-5`) | `--model claude-sonnet-5-5` |
| `--effort LEVEL` | `low`, `medium` (default), `high`, `xhigh`, `max` | `--effort high` |
| `--max-revisions N` | Critic revision rounds (default 2) | `--max-revisions 0` |
| `--env-file F` | credentials file (default `./.env`) | `--env-file ~/.ark.env` |
| `-o`, `--out DIR` | output folder | `--out renders/drive` |
| `--reverb X` | reverb mix 0–1 (default 0.16) | `--reverb 0.35` |
| `--no-mp3` | skip the ffmpeg MP3 export | |
| `--spec` | print the JSON tool spec and exit | |

Defaults and interactions for every flag are in [docs/AI_SPEC.md](docs/AI_SPEC.md#32-cli).

---

## Prompting guide for developers

The same three inputs do the work whether a person types the command or your model
builds the call. Putting each request into the right input matters more than the
wording.

### The three inputs

| Input | Its job | Put here | Keep out |
|---|---|---|---|
| `text` | the **mood** | the user's own words: a verse, a scene, imagery, feelings | structure ("8 bars", "an intro") |
| `direction` | **how it's built** | sections and their lengths, rising or falling shape, leap limits, a tonal anchor, the purpose, a stated key, tempo or length | moods and imagery |
| `references` | **music to draw on** | a library id, a title or composer, a `.mid` path | modern copyrighted songs (pick the closest archetype and say so) |

One switch changes what the text is for: with `sing`, the text is also the **lyric**, so pass
the user's words exactly, with their line breaks.

When they disagree: **direction > text > references**. A reference never changes the
key, mode or tempo unless you borrow `tonality` or `tempo`, and a key or tempo stated in
the direction beats even that.

### Writing the text

Pass the user's words as they are. Concrete imagery ("rain on a tin roof at 2am") gives
Claude more to work with than adjectives alone. Offline, the brain matches words against
ten feelings, so it helps to know which words it reacts to:

| Feeling | Some words it reacts to | Offline it leans to |
|---|---|---|
| devotional | lord, grace, prayer, psalm, hymn, soul, mercy | major, about 76 BPM |
| dark | sad, lonely, rain, winter, goodbye, broken | minor, about 74 BPM |
| bright | happy, joy, sun, dance, love, spring | major, about 118 BPM |
| epic | triumph, battle, hero, rise, storm, victory | minor, about 140 BPM |
| calm | calm, peace, gentle, dream, sleep, breeze | lydian, about 68 BPM |
| tense | angry, fear, chase, danger, panic | phrygian, about 150 BPM |
| mystery | mysterious, eerie, shadow, ghost, fog, ancient | dorian, about 92 BPM |
| hope | hope, dawn, new, believe, future, wonder | mixolydian, about 108 BPM |
| nostalgic | remember, memory, childhood, home, letter | major, about 84 BPM |
| urban | city, neon, street, night, lights, midnight | minor, about 96 BPM |

A text with none of these words still composes. Offline it gets a neutral reading and no
automatic reference.

### Writing a direction

- **Name the sections you want:** "a short intro", "rising notes", "a verse", "a chorus",
  "a bridge", "a climax", "a falling melody", "a long outro". They always play in this
  order: intro, rise, main, chorus, bridge, climax, fall, outro.
- **Put a length next to its section:** "a 6 second intro", "an outro of 2 bars".
  Seconds are rounded to whole bars at the chosen tempo.
- **State a key as `in` + a capital letter + a mode:** "in D minor" works; "in d minor"
  does nothing.
- **Shape the melody:** "stepwise" (no leap over a third), "fewer long leaps" (a fourth),
  "structured, 4-bar phrases" (motifs repeated A A′ B A′).
- **Ground it:** "strongly leaning toward one chord and tone" gives tonic-heavy harmony,
  a pedal bass and a melody drawn home.
- **Say what it's for:** "background music for a podcast" gives a lighter arrangement.
- **Name at least one section when you give a total length or a tonal anchor.** Offline,
  both are fitted to the planned sections. With none named, the default form is used and
  that check can fail.
- **Don't ask for** a time signature (borrow `tempo` from a 3/4 or 6/8 reference
  instead), or for vocals: singing is the `sing` switch, not a direction. Avoid the word
  "phrases" unless you want phrase structure.
- **With `sing`, think twice about a total length.** A stated length wins over the words, so a
  long lyric is sung faster and may lose lines. Leave it out when every word matters.

The full vocabulary is under [Musical direction](#musical-direction), and
[docs/PROMPTS.md §4](docs/PROMPTS.md#4-phrasebook-for-directions-that-parse-reliably) has
a phrasebook.

### Choosing references and what to borrow

Search first (`ark-audio --list-references WORDS` or `list_references("WORDS")`), then
pass an id. Map what the user says onto `borrow` and `strength`:

| The user says | Pass |
|---|---|
| "the same chords as …" | `borrow: ["harmony"]` |
| "the same tune", "that motif" | `["melody"]` |
| "the same structure" | `["form"]` |
| "that sound", "those instruments" | `["instruments"]` |
| "that beat", "that rhythm section" | `["groove"]` |
| "in the same key" | `["tonality"]` |
| "the same speed", "a waltz", "in 6/8" | `["tempo"]` (it brings the meter) |
| "a nod to" · "clearly inspired by" · "very close to" | `strength` 0.3 · 0.6 · 0.9 |

Pass `borrow` or `strength` only when the user says so: passing either locks it against
the agents. For a modern copyrighted song, use the nearest archetype and tell the user
it's modelled on the style, not the song.

### Requests turned into calls

These are `compose_music` arguments. On the CLI they become the positional text, `-d`, `-r`,
`--borrow` and `--strength`.

| Request | Arguments |
|---|---|
| "Music for this verse: The Lord is my shepherd; I shall not want." | `{"text": "The Lord is my shepherd; I shall not want."}` |
| "Same chords as Pachelbel's Canon, but sadder, my own melody." | `{"text": "a sad farewell at a train station in the rain", "references": ["pachelbel-canon"], "borrow": ["harmony"]}` |
| "A wistful waltz, like Greensleeves." | `{"text": "wistful, a memory of an old dance hall", "references": ["greensleeves"], "borrow": ["tempo", "harmony", "melody", "instruments"]}` |
| "An evening-raga feel for my diya-lighting video." | `{"text": "lamps lit at dusk by the river", "references": ["raga-yaman"], "borrow": ["tonality", "melody", "instruments"]}` |
| "30 seconds of bhangra for a wedding reel." | `{"text": "a punjabi wedding baraat", "references": ["bhangra-dhol"], "borrow": "all", "direction": "a short intro, a chorus, a short outro. 30 seconds long overall"}` |
| "Totally original, in F# minor, stepwise." | `{"text": "<their mood>", "direction": "in F# minor, structured, 4-bar phrases, stepwise", "auto_reference": false}` |
| "Sing this: The Lord is my shepherd; I shall not want." | `{"text": "The Lord is my shepherd; I shall not want.", "sing": true}` |
| "Again, but different." | the same arguments with `seed` + 1 |

More worked examples, with the reasoning behind each, are in
[docs/PROMPTS.md §3](docs/PROMPTS.md#3-turning-requests-into-calls-worked-examples).

### Put it behind your own model

`ark_audio_agent.spec` gives you all three pieces: the **system prompt**
(`system_prompt()`, reproduced in [docs/PROMPTS.md](docs/PROMPTS.md#1-the-system-prompt)),
the **tools** (`anthropic_tools()`, `openai_tools()`, or `ark-audio --spec` for any other
stack) and the **dispatcher** (`run_tool(name, args)`: JSON in, JSON out, errors returned as
`{"ok": false, "error": ...}`). A complete Claude loop:

```python
import json

import anthropic

from ark_audio_agent.spec import anthropic_tools, run_tool, system_prompt

client = anthropic.Anthropic()          # ANTHROPIC_API_KEY, or an `ant auth login` profile
messages = [{"role": "user", "content": "Gentle background music for my morning-prayer "
                                        "podcast, about 45 seconds, a bit like Amazing Grace"}]
while True:
    response = client.beta.messages.create(
        model="claude-opus-5-5",
        max_tokens=16000,
        system=system_prompt(),
        tools=anthropic_tools(),
        output_config={"effort": "medium"},
        betas=["server-side-fallback-2026-07-01"],   # re-run a refused request on a
        fallbacks="default",                           # fallback model (Claude API only)
        messages=messages,
    )
    if response.stop_reason == "refusal":
        raise RuntimeError(f"declined: {response.stop_details}")
    messages.append({"role": "assistant", "content": response.content})   # unchanged
    if response.stop_reason != "tool_use":
        break
    results = []
    for block in response.content:
        if block.type == "tool_use":
            out = run_tool(block.name, {**block.input, "offline": True})   # fast and free
            results.append({"type": "tool_result", "tool_use_id": block.id,
                            "content": json.dumps(out), "is_error": not out.get("ok", False)})
    messages.append({"role": "user", "content": results})   # all results in one message

print(next(b.text for b in response.content if b.type == "text"))
```

Things that matter in production:

- **Who composes.** `run_tool` uses the brain of the machine it runs on. With a key
  present, each music request would start its own multi-agent Claude run (minutes, real
  tokens), so the loop above forces `offline`. Remove that to let Claude compose too.
- **Don't force the tool.** `tool_choice` `any` or `tool` returns a 400 on Claude Opus 5.5.
  The system prompt is what steers the model to call `compose_music`.
- **Keep the history unchanged.** Append `response.content` as it came back, so the
  thinking blocks stay valid.
- **Other platforms.** On Microsoft Foundry, Bedrock or Vertex, use that platform's client
  and `client.messages.create`, without `betas` and `fallbacks`.
- **Async servers.** `run_tool` blocks while it renders, so call
  `await asyncio.to_thread(run_tool, name, args)`.
- **Tell the user what happened.** The result carries the title, key, tempo, form, file
  paths, `direction.failed`, the references used and, for a sung piece, `lyrics.failed`.
- **Track spend.** The result also carries `tokens` (API calls, input, output, cache read,
  cache write, `total_tokens`) and `agent_runs` with each run's tokens. Offline they are 0.
- **More wiring.** OpenAI-compatible function calling, an MCP server and the
  any-language CLI pattern are in [docs/PROMPTS.md §2](docs/PROMPTS.md#2-wiring-it-up),
  with a production checklist in §5.

---

## Python library

```python
from ark_audio_agent import compose

result = compose("a melancholy rainy afternoon", out_dir="renders/rain", offline=True)
print(result.outputs["wav"], "-", result.score.title)

# text for the mood, direction for the form and constraints
result = compose("For God so loved the world", out_dir="renders/bgm",
                 direction="short intro, rising notes into the verse, a climax, "
                           "then a falling melody and a few seconds of outro")
for check in result.direction_checks:
    print("✓" if check["ok"] else "✗", check["requirement"], "-", check["detail"])

# reference music: a library id/title or a .mid path; borrow picks what to take
result = compose("a rainy afternoon", references=["satie-gymnopedie-1"],
                 borrow=["harmony", "melody"], strength=0.6, offline=True)
print(result.inspiration_checks)   # how closely each borrowed aspect was followed

# sing the text as the lyrics (off by default)
result = compose("The Lord is my shepherd; I shall not want.", sing=True, offline=True)
print(result.outputs["lyrics"], result.lyric_checks)   # lyrics.lrc, and every line sung?

# tokens: per agent run, and the total (all 0 offline)
for run in result.agent_results:
    print(run.agent, run.total_tokens, run.usage)
print(result.token_usage()["total_tokens"])
```

The library doesn't read `.env` (the CLI does). Load it with
`ark_audio_agent.envfile.load_env_file(".env")`, and check `result.brain_kind` to see which
brain ran. `list_references("rainy piano")` searches the library from Python.

---

## Musical direction

Two inputs, with separate jobs:

- **The text** (the positional prompt) sets the **mood**: key, mode, tempo, genre,
  instruments.
- **The direction** (`--direction`) sets **how the piece is built**. The engine
  treats it as requirements and lets the agents be creative inside them.

A rule-based reader turns the direction into a plan the agents can see, so it
works offline as well. With the Claude brain, the Interpreter reviews that plan
against your words and corrects it with `set_direction`. It understands:

| You write | It becomes |
|---|---|
| "a few secs of intro", "a 6 second intro", "intro of 2 bars", "no intro" | an `intro` section of ~4 s / 6 s / 2 bars, or none |
| "rising notes", "build up", "crescendo" | a `rise`: an ascending run with a crescendo, leading in |
| "main part of the melody", "verse", "chorus", "bridge" | the main sections, in that order |
| "climax", "peak" | a `climax` that holds the piece's highest note |
| "falling melody", "wind down" | a `fall`: a descending line that resolves to the tonic |
| "outro", "end it", "fade out", "long outro" | an `outro` that settles on the tonic and fades |
| "fewer long leaps", "smooth" / "stepwise", "no leaps" | no interval wider than a fourth / a third |
| "leaning toward one chord and tone", "drone", "grounded" | a tonal anchor: tonic-heavy harmony, a tonic pedal bass, a melody drawn to the tonic |
| "structured", "memorable phrases", "4-bar phrases" | phrase-structured melody: motifs repeated as A A′ B A′ |
| "bgm", "background music for …" | a lighter arrangement (fewer notes, soft or no drums) |
| "in D minor", "90 bpm", "slow", "45 seconds long" | key, tempo and length |

Durations given in seconds are converted to whole bars at the chosen tempo.

**How the engine enforces it.** The melody engine guarantees the hard limits:

- No leap is wider than the limit, and that includes the step from one section
  into the next.
- Rises and falls move one way.
- The climax reaches above everything before it, and nothing after it goes higher.
- Cadences resolve to the tonic.

Then `analyze_score` measures the realized piece against each requirement. The
Critic sends any unmet one back to the agent responsible, and the CLI and
`report.md` show the result:

```
direction: 11/11 requirements met
  ✓ form follows the plan        sections: intro -> rise -> verse -> climax -> fall -> outro
  ✓ intro lasts ~4s              3.3s (1 bars at 73 BPM)
  ✓ rise rises                   G4 -> D5, trend +5.0 semitones
  ✓ climax holds the highest note
  ✓ fall falls                   C6 -> C5, trend -10.5 semitones
  ✓ no leap larger than 3 scale steps
  ✓ harmony leans on the tonic chord   tonic on 58% of bars (need 45%)
  ✓ melody ends on the tonic     last note C5
  ...
```

---

## Reference music

A third input, with its own job: **music to draw on**. A reference is a set of
traits, not notes:

- a tonality, a tempo and meter, and a form;
- chord progressions as Roman numerals, so they move to any key;
- melodic traits plus the interval shape of a motif;
- an instrument palette and a groove.

The library ships 29 references:

- **Public-domain works:** Pachelbel's Canon, Satie's Gymnopédie No. 1, Bach's
  Prelude in C and Toccata in D minor, Beethoven's Moonlight Sonata, Ode to Joy and
  Für Elise, Debussy's Clair de lune, Grieg's Mountain King and Morning Mood, Amazing
  Grace, Greensleeves, Scarborough Fair and the Dies Irae.
- **Genre archetypes**, which describe a style rather than any one song: pop anthem,
  12-bar blues, synthwave, lo-fi, epic trailer, ambient drone, flamenco, celtic reel
  and worship ballad.
- **Indian styles**, also archetypes: Hindustani ragas Yaman (evening) and Bhairavi
  (morning), each with its pakad as the motif, a Carnatic kriti in raga Mohanam, a
  Bollywood romantic ballad, bhangra and Sufi qawwali. Their palettes use the `sitar`,
  `shehnai` and `harmonium` instruments.

```bash
ark-audio --list-references            # the whole library
ark-audio --list-references rainy piano   # ranked for those words
```

Choose what to take with `--borrow`, and how closely with `--strength` (0..1):

| Aspect | Takes |
|---|---|
| `tonality` | key and mode |
| `tempo` | tempo and meter (how to get 3/4 or 6/8) |
| `form` | the section layout and its intensity arc |
| `harmony` | the progressions per section, moved into the piece's key and mode (I–V–vi–IV becomes i–v–VI–iv in minor) |
| `melody` | contour, density, phrase length, and the motif's interval shape, which the engine re-rhythms and answers with its inversion |
| `instruments` | the palette |
| `groove` | bass and drum styles |

By default a reference you name lends form, harmony, melody, instruments and groove,
but not tonality or tempo, because the text sets the mood. What you set is locked
against the agents.

Without `-r`, the Interpreter may pick a reference itself when one clearly fits the
text. Offline it pins the best match at strength 0.5; `--no-reference` turns this off.

Point `-r` at a `.mid` file and it is read into the same traits: key, tempo, meter,
one chord per bar, melodic traits and a motif, palette, bass and drum styles. The
reader only knows major and minor, so a modal piece comes back as the major or minor
key with the same notes.

The precedence is **direction > text > references**. A direction's sections replace
a reference's form, and a tonal anchor keeps the harmony tonic-heavy. `analyze_score`
and `report.md` show an *echo* per borrowed aspect: how closely the piece follows its
reference, from 0 to 1. It is a measure, not a target.

---

## Sung lyrics

Off by default. With `--sing` (`sing=True`, `"sing": true`) the text is the lyric as well as
the mood:

- **Lines.** Each line break or sentence (`. ; : ! ?`) is a sung line, and anything over 14
  syllables is split at a comma or in half. Words are split into syllables by English spelling
  rules, with each syllable's stress and sounds.
- **Placement.** The Melodist puts the lines in the singing sections (verse, chorus, bridge,
  climax, fall) with the `set_lyrics` tool. By default they run in order through the verse,
  and a chorus sings them again as a refrain; the intro and outro stay instrumental, and a
  short lyric repeats until the words move at a natural pace.
- **Room.** The Composer sizes the singing sections for about 2.5 syllables a second. A length
  in the direction wins: the words are sung faster, and lines that still don't fit are left out.
- **The line.** Each syllable gets its own note, a line per phrase: stressed syllables longer,
  the last one held, a breath between lines. The melody engine picks the pitches, so the
  direction's guarantees still hold, with leaps capped at a fifth so it stays singable.
- **The voice.** A formant singing synth (`voice.py`) sings the lead line, moved by octaves into
  a comfortable range, while the lead instrument doubles it softly. It has clear vowels and
  rough consonants, like early speech synthesis. For a real voice, load `song.mid`'s `vocal`
  track (it carries the lyrics) into a vocal synth.

Two checks are measured and shown on the CLI and in `report.md`, and a failure goes back to
the Critic like a direction requirement:

```
lyrics: 2/2 checks met
  ✓ every lyric line is sung   4/4 lines, 62 syllables sung
  ✓ words fit the tempo (at most 5 syllables a second)   fastest line 2.0 syllables/s, in 'verse'
```

English is read best. Other Latin-script languages are sung approximately; a word with no
Latin letters is sung on one neutral vowel. The full rules are in
[docs/AI_SPEC.md §7](docs/AI_SPEC.md#7-sung-lyrics-sing).

---

## Tokens and cost

With the Claude brain, every agent run records the API calls and tokens it used: input,
output, cache read and cache write, as the API reports them. They show up in four places:

- **While it runs:** each agent's line, e.g. `12 actions · 18,402 tokens · Interpreted as ...`.
- **After it runs:** a `tokens` row in the summary, then a table of every agent run (the
  pipeline, the Critic's reviews and any revisions) with its tokens and a total row.
- **In `report.md`:** the header line, an **Agent runs** table with a **Total** row, each
  agent's trace heading and the settings.
- **From code:** `run.usage` and `run.total_tokens` per `result.agent_results` entry,
  `result.token_usage()` and `result.total_tokens()` for the run, and `tokens` and
  `agent_runs` in the `compose_music` tool result.

The total is input + output + cache read + cache write (input excludes cached input). Offline
runs make no calls, so every count is 0.

---

## Outputs

Each run gets its own folder, `renders/<YYYYMMDD-HHMMSS>_<slug of the text>`, so
runs sort by time and never overwrite each other (`--out` or `out_dir=` picks
a folder yourself). It contains:

| File | What |
|---|---|
| `song.mid` | Standard MIDI file (open in any DAW / soundfont player); when sung, a `vocal` track with lyric events |
| `song.wav` | 44.1 kHz stereo audio from the built-in synth (and voice, when sung) |
| `song.mp3` | 192 kbps MP3 (if `ffmpeg` is installed) |
| `score.json` | the full declarative Score (reload / tweak / re-render) |
| `brief.json` | compact human/LLM summary of the piece |
| `lyrics.lrc` | only when sung: each sung line with the time it starts |
| `trace.jsonl` | every agent step: thought · action · observation |
| `report.md` | a readable write-up: which steps an LLM made and which were deterministic, every setting used (brain, model, effort, thinking, tokens, seed, revisions, references, singing, reverb, versions), the brief, the direction and how it was met, the references and what was borrowed, the lyrics and how they were sung, the structure, every agent run with its tokens and a total, and the agent trace |

---

## How it works

### Why it's built this way

Rather than ask one model to emit raw notes (which drifts out of key and is hard
to steer), the system separates **musical intent** from **musical realization**:

- **Agents decide intent** — key, mode, tempo, song form, chord progressions, a
  melody's contour/register/density, instrument roles, drum and bass styles.
- **Deterministic engines realize it** — pure music-theory functions turn that
  intent into in-key notes, which a `mido` writer turns into MIDI and a
  from-scratch `numpy` synthesizer turns into audio.

Because every decision an agent can make is a **tool**, both brains (Claude and
the offline planner) drive the exact same action surface and produce the exact
same trace format.

### Architecture

```
            text prompt
                │
     ┌──────────▼───────────────────────────────────────────────┐
     │  Orchestrator (the harness)                                │
     │                                                            │
     │   Interpreter → Composer → Melodist → Arranger             │  each agent runs a
     │                               │                            │  ReAct loop:
     │                               ▼                            │   reason (thinking)
     │                            Critic ──reject──┐              │   → act  (tool call)
     │                               │approve      │ one targeted │   → observe (result)
     │                               │        revision (bounded)  │
     └───────────────────────────────┼────────────┘              │
                                      │                           │
          ┌───────────────────────────▼───────────────────────┐  │
          │  Tools  (the ReAct action space)                   │  │
          │  set_brief · set_direction · get_theory ·          │  │
          │  find_references · choose_references · set_global ·│  │
          │  set_structure · set_progression · design_melody · │  │
          │  set_lyrics · set_instruments · set_bass ·         │  │
          │  set_drums ·                                       │  │
          │  analyze_score · finish / approve / request_revision│ │
          └───────────────────────────┬───────────────────────┘  │
                                       ▼                          │
                            Score blackboard (shared state)       │
     └──────────────────────────────┬─────────────────────────────┘
                                     ▼
          performance.py  →  { melody, harmony, bass, drums } notes
                             (+ the sung line, set by lyrics.py)
                                     │
                 ┌───────────────────┴────────────────────┐
                 ▼                                         ▼
        midi_render.py (mido)                   synth.py (+ voice.py when sung)
                 │                                         │
              song.mid                        song.wav ──(ffmpeg)──► song.mp3
```

### The agents

| Agent | Decides | Tools |
|---|---|---|
| **Interpreter** | mood, imagery, genre, key/mode/tempo, emotional arc; reviews the direction's plan; picks reference music | `set_brief`, `set_title`, `set_direction`, `find_references`, `choose_references` |
| **Composer** | time signature, section form + dynamic arc, chord progressions | `get_theory`, `set_global`, `set_structure`, `set_progression` |
| **Melodist** | per-section melody contour, register, density, syncopation; which lines each section sings, when sung | `design_melody`, `set_lyrics`, `get_theory` |
| **Arranger** | instruments, per-section bass & drum styles | `set_instruments`, `set_bass`, `set_drums` |
| **Critic** | judges the whole piece; approves or bounces back one revision | `analyze_score`, `approve`, `request_revision` |

Every agent also has `analyze_score` (a sensing action) and a terminal `finish`.

### The two brains

**Claude multi-agent** (default when an API key is set). Each agent runs
the canonical Messages-API tool-use loop with adaptive thinking; the model
reasons, calls a tool, reads the observation, and continues until it finishes.
Defaults to `claude-opus-5-5`. The API call degrades gracefully across
SDK/account differences (server-side refusal fallback → plain streaming → drop
effort → drop thinking), so the run never hard-fails on a capability mismatch.

```bash
# .env
ANTHROPIC_API_KEY=sk-ant-...
```
```bash
ark-audio "a cathedral of light at the edge of the world" --effort high
```

**Claude on Microsoft Foundry (Azure).** Point the brain at a Foundry resource
instead and it uses the SDK's `AnthropicFoundry` client with API-key auth. A
Foundry endpoint takes precedence over `ANTHROPIC_API_KEY`:

```bash
# .env
ANTHROPIC_FOUNDRY_API_KEY=...
ANTHROPIC_FOUNDRY_RESOURCE=my-resource   # or ANTHROPIC_FOUNDRY_BASE_URL=https://my-resource.services.ai.azure.com/anthropic/
ARK_MODEL=claude-opus-5-5                # your Foundry deployment name
```

If both `ANTHROPIC_FOUNDRY_RESOURCE` and `ANTHROPIC_FOUNDRY_BASE_URL` are set,
the base URL is used.

Server-side refusal fallback is a first-party API feature, so it is skipped on
Foundry. Adaptive thinking and effort still apply.

**Deterministic** (offline, always available). A rule-based planner maps the
text to a feeling via a lexicon, then applies music-theory heuristics to make the
same tool calls an LLM would. Reproducible given a `--seed`. This is what runs in
CI and what powers `--offline`.

> If no key is found (and `--offline` isn't set), the CLI automatically uses the
> deterministic brain and says so.

### How the music is realized

- **Theory** (`theory.py`): scales/modes, diatonic chords, Roman-numeral
  resolution, scale-degree → MIDI, equal-temperament frequencies.
- **Blackboard** (`blackboard.py`): the declarative `Score` the agents edit.
- **Direction** (`direction.py`): reads a direction prompt into a section plan
  and constraints, and checks a realized piece against it.
- **References** (`references.py`, `midi_analysis.py`): the reference library,
  its search and name resolution, the measure of how closely a piece echoes its
  references, and the reader that turns a MIDI file into reference traits.
- **Lyrics** (`lyrics.py`): with `--sing`, reads the text into lines and syllables (stress
  and sounds), places the lines in sections, sets one syllable per note, checks the result and
  writes `lyrics.lrc`.
- **Spec** (`spec.py`): the library as tools for another model, with the
  definitions, a `run_tool` dispatcher, the recommended system prompt and the full
  machine-readable contract.
- **Performance** (`performance.py`): realizes the Score into notes:
  - a contour-driven melody that snaps to chord tones on strong beats, or a
    phrase-structured melody when a direction (or the Melodist) asks for one.
    That melody is built from motifs, keeps leaps within a limit, leans on the
    tonic, flows across sections and ends on a cadence;
  - intensity ramps within a section;
  - chord comping or sustained pads;
  - several bass styles (roots, root-fifth, pulse, walking, arp, tonic pedal);
  - synthesized drum patterns (four-on-floor, backbeat, breakbeat, a riser, …).

  All of it is seeded and reproducible.
- **Synth** (`synth.py`): additive (band-limited) oscillator voices with ADSR,
  per-instrument patches, synthesized percussion, an FFT-convolution reverb, and
  a soft-clipping master bus.
- **Voice** (`voice.py`): a formant singing voice. Vowels are harmonics shaped by five
  formants, with a scoop into each note and vibrato; consonants are noise bursts, hums and
  glides timed so the vowel lands on the beat.

---

## Development

### First-time setup (virtual environment)

You need **Python 3.10+** (the repo pins 3.12 in `.python-version`). On macOS
the system `python3` is 3.9, which is too old, so use the Python that uv
installs or one from Homebrew.

**Option A: uv (recommended)**

```bash
git clone https://github.com/ir-dev/ark-audio-agent.git
cd ark-audio-agent
uv sync                      # creates .venv: the package (editable), anthropic, pytest
source .venv/bin/activate    # Windows: .venv\Scripts\activate
```

If Python 3.12 is missing, uv downloads it. `uv sync` also installs the `dev`
group (pytest and the anthropic SDK) by default, so a later `uv sync` never
drops the Claude brain.

**Option B: plain `venv` + pip**

```bash
git clone https://github.com/ir-dev/ark-audio-agent.git
cd ark-audio-agent
python3.12 -m venv .venv     # any Python 3.10+
source .venv/bin/activate    # Windows: .venv\Scripts\activate
python -m pip install --upgrade pip
pip install -e ".[llm]" pytest
```

`-e` is an editable install, so changes under `src/` take effect without
reinstalling.

### Check it works

```bash
pytest -q                                                 # theory + pipeline tests (offline, no key needed)
ark-audio "a quiet dawn over still water" --offline --seed 7
./examples/run_examples.sh                                # a spread of moods, offline
```

With the venv activated, `ark-audio` and `pytest` run directly. Without
activating it, prefix them with `uv run`.

### API key (optional)

A key is only needed for the Claude brain. Copy the template and fill in **one**
provider:

```bash
cp .env.example .env         # .env is git-ignored; never commit it
```

The CLI reads `./.env` itself, on every OS, so just run it from the repo root:

```bash
uv run ark-audio "a storm breaking over the sea"
```

## License

MIT.
