Metadata-Version: 2.5
Name: zeli-avatar
Version: 0.1.0
Summary: Python SDK for real-time, lip-synced talking-head avatars over WebRTC
Project-URL: Homepage, https://zeligate.com
Author: Zeligate
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: avatar,lip-sync,real-time,streaming,text-to-speech,webrtc
Classifier: Development Status :: 4 - Beta
Classifier: Framework :: AsyncIO
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Multimedia :: Video
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: aiohttp>=3.9
Requires-Dist: aiortc>=1.9.0
Requires-Dist: av>=11.0
Requires-Dist: numpy>=1.23
Provides-Extra: dev
Requires-Dist: mypy>=1.8; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest>=7.4; extra == 'dev'
Requires-Dist: pyyaml>=6.0; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Provides-Extra: display
Requires-Dist: opencv-python>=4.8; extra == 'display'
Requires-Dist: sounddevice>=0.4; extra == 'display'
Description-Content-Type: text/markdown

# Zeli Avatar SDK (Python)

Real-time, lip-synced talking-head avatars streamed to your app over **WebRTC**.
Connect to a Zeli avatar server, receive synchronized audio + video frames, and
drive the avatar with text, either through the conversational model or straight
to text-to-speech.

📖 **Documentation:** this README covers the whole surface, and runnable
programs for each capability ship in `examples/`. A hosted documentation
site is coming; it is deliberately not linked here yet, because a link in
PyPI metadata cannot be corrected after a version is published.

```bash
pip install zeli-avatar
# with display helpers (OpenCV / sounddevice) for the examples:
pip install "zeli-avatar[display]"
```

Requires Python 3.10+. You also need a running Zeli avatar server, and an API
key to authenticate against it.

```python
client = ZeliClient(
    api_key="zsk_live_...",                       # or os.environ["ZELI_API_KEY"]
    avatar_config=AvatarConfig(avatar_id="01-presenter-male__confident"),
    options=ClientOptions(server_url="https://your-zeli-host"),
)
```

The key is sent as an `X-Api-Key` header on every request. A rejected key raises
`AuthenticationError`.

## Which box am I talking to

There is **no default base URL**, and that is deliberate. This product is
deployed per environment, so a client that picks one for you picks an
environment for you, and the only way to find out which is to look at the
traffic. Name it one of two ways:

```python
# In the code, when the app knows its own target:
client = ZeliClient(api_key=..., options=ClientOptions(server_url="https://your-box.example.com"))

# Or from the environment, when the deployment knows and the code should not:
#   export ZELI_SERVER_URL=https://your-box.example.com
client = ZeliClient(api_key=...)
```

Say neither and `ZeliClient(...)` raises `ConfigurationError` naming both
options. Say both and they must agree: two different URLs raise rather than one
silently beating the other, because the developer who exported
`ZELI_SERVER_URL` to aim a script somewhere else would otherwise be overruled
with nothing printed. The URL must start with `http://` or `https://`; a bare
hostname is refused at construction rather than becoming an `InvalidURL` deep
inside a request, or a silent `ws://` downgrade on the control channel.

The variable name is exported as `zeli.SERVER_URL_ENV_VAR`, and the JavaScript
SDK reads the same one.

## Quickstart

```python
import asyncio, os
from zeli import ZeliClient, AvatarConfig, ClientOptions, ZeliEvent

client = ZeliClient(
    api_key=os.environ["ZELI_API_KEY"],
    avatar_config=AvatarConfig(avatar_id="01-presenter-male__confident", voice_id="your-voice-id"),
    options=ClientOptions(server_url="http://your-server:8080"),
)

@client.on(ZeliEvent.MESSAGE_RECEIVED)
async def on_message(message):
    print(f"{message.role.value}: {message.content}")

# send_message() and talk() hand work to the server and return immediately, so a
# server-side refusal ("TTS not configured", unknown voice) arrives here and
# nowhere else. Without this handler the script prints nothing and looks hung.
@client.on(ZeliEvent.ERROR)
async def on_error(err):
    print("server error:", err)

async def main():
    async with client.connect() as session:
        await session.send_message("Hi! Introduce yourself in one sentence.")
        await session.wait_until_closed()

asyncio.run(main())
```

## Core concepts

| Object | What it is |
|--------|------------|
| `ZeliClient` | Entry point. Holds config + event handlers; opens sessions. |
| `ManagementApi` | `client.management`: avatars, voices, settings, session tokens. Needs a full API key, so it belongs on your server. |
| `AvatarConfig` | Which avatar/voice/persona to bring to life. |
| `ClientOptions` | Where the server is (`server_url`), ICE servers, timeouts. |
| `Session` | One live connection: media streams + control methods. |
| `TalkStream` | Push text to the avatar's voice incrementally. |
| `ZeliEvent` | The events you can subscribe to with `@client.on(...)`. |

## Connecting

```python
# Use an already-prepared avatar by id (uploaded files are keyed by filename stem):
client = ZeliClient(
    api_key=os.environ["ZELI_API_KEY"],
    avatar_id="01-presenter-male__confident",
    options=ClientOptions(server_url="https://your-box.example.com"),
)

# Or pass the config object, which also carries the voice:
client = ZeliClient(
    api_key=os.environ["ZELI_API_KEY"],
    avatar_config=AvatarConfig(
        avatar_id="01-presenter-male__confident",
        voice_id="your-voice-id",
    ),
    options=ClientOptions(server_url="https://your-box.example.com"),
)

async with client.connect() as session:
    print(session.session_id, session.avatar)
    await session.wait_until_closed()
```

An avatar id the server cannot prepare is **not an error there**. It answers
`200`, echoes your id back, and streams a clip it does have instead. The SDK
turns that into a `SERVER_WARNING` and exposes the substitute as
`session.substituted_avatar` (`None` when you got what you asked for), because
`session.avatar` only reflects what you *requested*. Check ids against
`await client.list_avatars()`.

## Receiving media

Video and audio arrive as [PyAV](https://pyav.org) frames you can turn into NumPy arrays.

```python
async with client.connect() as session:
    async def show_video():
        async for frame in session.video_frames():
            img = frame.to_ndarray(format="rgb24")   # (H, W, 3) uint8

    async def play_audio():
        async for frame in session.audio_frames():
            samples = frame.to_ndarray()              # int16 PCM, 48 kHz stereo

    await asyncio.gather(show_video(), play_audio())
```

## Driving the avatar

```python
# Through the conversational model (LLM -> reply -> TTS -> avatar):
await session.send_message("What's the weather like on Mars?")

# Straight to text-to-speech, bypassing the model:
await session.talk("This line is spoken immediately.")

# Incrementally, as text becomes available:
async with session.create_talk_stream() as talk:
    await talk.send("Streaming ")
    await talk.send("this ")
    await talk.send("out loud.", end_of_speech=True)

# Cut the avatar off mid-sentence (barge-in):
await session.interrupt()

# Start a fresh conversation without dropping the session. The avatar stays on
# screen and connected; it just stops remembering what was said.
conversation_id = await session.new_conversation()
```

`new_conversation()` erases nothing. The box keeps the conversation you closed,
so its turns stay browsable, and opens a new one beside it. The persona and the
system prompt survive, because only the turn list rotates: the avatar keeps its
character and loses its memory.

It returns the new conversation id, or `None` from a box old enough not to
report one. `None` is not a failure. The conversation still rotated; the box
simply did not say which one replaced it.

It also clears the local transcript, so `client.get_message_history()` agrees
with the box rather than continuing to show a conversation the avatar has
forgotten. Handlers registered on `MESSAGE_HISTORY_UPDATED` receive the empty
history, and the clear happens only after the box confirms: a refused reset
leaves your view intact, because what is on screen is still the truth.

Scoped to your credential. A box is shared, and the conversation id is never
taken from the caller, so this cannot reach anybody else's history.

## Events

Register handlers as decorators or with `add_listener`. Handlers may be sync or async.

```python
from zeli import ZeliEvent

@client.on(ZeliEvent.SESSION_READY)
async def on_ready(info):
    print("ready:", info.session_id)

@client.on(ZeliEvent.MESSAGE_STREAM_EVENT_RECEIVED)
async def on_chunk(event):
    print(event.content, end="", flush=True)   # transcript, clause by clause

@client.on(ZeliEvent.AVATAR_SPEECH_STARTED)
async def on_speaking(correlation_id):
    print("avatar started speaking")
```

| Event | Fires when |
|-------|-----------|
| `CONNECTION_ESTABLISHED` | The media connection is up. |
| `SESSION_READY` | The control channel handshake completed. |
| `MESSAGE_RECEIVED` | A user or assistant message is finalized. |
| `MESSAGE_STREAM_EVENT_RECEIVED` | An incremental transcript chunk arrives. |
| `MESSAGE_HISTORY_UPDATED` | The transcript changed. |
| `AVATAR_SPEECH_STARTED` / `AVATAR_SPEECH_ENDED` | The avatar starts / stops speaking. |
| `TALK_STREAM_INTERRUPTED` | A barge-in interrupted playback. |
| `CONNECTION_CLOSED` | The session ended. Carries a `ConnectionCloseCode`: `normal` when you closed it, `server_closed` when the server did, `webrtc_failure` when the transport broke. |
| `ERROR` / `SERVER_WARNING` | A server-side error / warning. |
| `EMOTION_DETECTED`, `USER_SPEECH_STARTED`, `USER_SPEECH_ENDED` | **Reserved, not currently emitted.** No Zeli server sends the frames these map to. |

`SESSION_READY` hands you a `SessionInfo` carrying `session_id`, the `avatar` the
server bound the session to, and that avatar's `tones`, the only place to learn
which values `session.talk(tone=...)` will accept on this server.

The box holds **one live session at a time**, so anyone else connecting takes
your slot. That arrives as `CONNECTION_CLOSED` with `server_closed`, and
`session.is_active` goes `False`; handle it if your process is long-lived.

Read the running transcript any time with `client.get_message_history()`.

## Server-side management

Everything that is not the live conversation lives on `client.management`:
creating avatars and voices, reading and writing the box's configuration, and
minting the short-lived token a browser streams with.

**It is a separate namespace because most of it needs a full API key.** A
`zsk_live_` key can create avatars, upload voices and mint tokens, so it belongs
on your server and must never reach page JavaScript. The session half of the SDK
is what runs beside a viewer, on a token your server mints. Keeping the
privileged calls behind one name makes that boundary visible.

```python
client = ZeliClient(
    api_key=os.environ["ZELI_API_KEY"],
    options=ClientOptions(server_url="https://your-box.example.com"),
)

# Mint the credential a browser streams with. Your backend does this, never the page.
token = await client.management.create_session_token(expires_in_seconds=600)
print(token.token, token.seconds_remaining(now=time.time()))

# Build an avatar from a portrait or a video, then wait for it.
status = await client.management.create_avatar("alice.png", name="alice", tones=["confident"])
while not avatar_is_ready(await client.management.list_avatars(), status.avatar_id):
    await asyncio.sleep(2)
```

### Keeping a session alive past its token

A session token lives for minutes, and a conversation can easily outlive it.
When it runs out the avatar simply stops answering: the next request is refused
and the caller is told the API key is wrong, which is both unhelpful and untrue.

The SDK cannot mint a replacement, and that is by design rather than an
omission. Minting needs the full API key, and the whole point of a session token
is that the full key never reaches the client holding it. So the renewal has to
come from a server, and `token_provider` is how it gets there:

```python
async def fetch_token() -> str:
    async with aiohttp.ClientSession() as http:          # YOUR endpoint
        async with http.get("https://your-app.example.com/api/zeli-token") as resp:
            return (await resp.json())["token"]

client = ZeliClient(
    session_token=first_token,
    token_provider=fetch_token,
    options=ClientOptions(server_url="https://your-box.example.com"),
)
```

That endpoint is yours: it calls `create_session_token()` with the full key,
exactly as above, and returns the token. The SDK then calls the provider when
the token it holds is within fifteen seconds of expiring, and again if the box
refuses it anyway. Both paths are needed: the second covers a client clock and a
box clock that disagree, which would otherwise strand a session this SDK
believed still had time left.

Four things are worth knowing:

* **It is optional.** Leave it out and the client behaves exactly as it always
  has.
* **It is never called twice at once.** Two requests refused at the same moment
  share one refresh, and both then proceed. Your endpoint is not hammered, and
  you do not end up with a spare token nobody used.
* **A failure is named as yours.** If your endpoint is down the SDK raises
  `TokenProviderError`, not `AuthenticationError`, so nobody goes looking at an
  API key that was never part of the call. The original failure is the
  `__cause__`.
* **It is refused beside `api_key`.** A full key does not expire, so there is
  nothing to refresh, and refreshing one would mean treating the full key and
  the browser token as the same kind of credential. That pairing raises a
  `ConfigurationError`.

The provider may return a `SessionToken` rather than a bare string, which is
what `create_session_token()` already hands back. Worth doing: a bare string
cannot say when it expires, so the SDK can only react to a refusal, while the
expiry keeps the renewal ahead of the failure for every token after the first.

`client.credential` exposes `seconds_remaining`, `presented` and `refresh()` for
a caller who wants to drive it, for instance to renew before something slow
rather than during it.

| Operation | What it does |
|-----------|--------------|
| `create_session_token(expires_in_seconds=None)` | Mint the browser credential. Returns a `SessionToken`; the box clamps the TTL, currently to 600s. |
| `get_settings()` / `update_settings(patch)` | Read and merge the box's configuration for this caller. A partial patch: omitted fields keep their value. |
| `get_status()` | Whether the language model and the voice engine are both ready. |
| `clear_conversation()` | Forget this caller's history. Also interrupts a reply in progress. |
| `list_voices()` | Every voice the engine offers. An id from here goes into `AvatarConfig.voice_id`. |
| `preview_voice(voice_id=None)` | Audition a voice. Returns a `BinaryAsset` (WAV bytes plus content type). |
| `create_voice(file_path, name=...)` | Clone a voice from a reference clip of at least 6 seconds. Minutes of work. |
| `prepare_voice(voice_id)` | Ask the engine to make a voice ready to speak. |
| `list_avatars()` | Every avatar, with its tone variants and what is still preparing. |
| `create_avatar(file_path, name=None, tones=None, consent_token=None, framing=None, gesture_amplitude=None)` | Create from a portrait or a video. Answers immediately with `preparing`. |
| `create_photo_avatar(photo_path, avatar_id=...)` | Build from a single portrait and wait for every clip. Gated by a feature flag and a consent check. |
| `delete_avatar(avatar_id)` | Delete an uploaded avatar. Never a stock one. |
| `list_avatar_clips(avatar_id)` | How each emotional variant is getting on. Sparse: a missing tone was never requested. |
| `get_avatar_clip(avatar_id, tone=None)` | One rendered clip, as MP4 bytes. |
| `retry_avatar_clip(avatar_id, tone)` | Regenerate one tone. Answers 202; a 409 means one is already running. |
| `get_avatar_preview(avatar_id)` | A still frame, as image bytes. |
| `upload_audio(file_path, filename=None)` | Upload an audio file for the box to play. |

**The JavaScript SDK offers the same eighteen operations under the same names**,
spelled `client.management.createSessionToken(...)` and so on: every name here is
that name in snake_case, with no exceptions, so you can predict one from the
other rather than look it up. Results are the box's own JSON with the box's own
field names in both SDKs. The two signature differences are deliberate: this SDK
takes a filesystem path where the browser one takes a `Blob`, because a Python
caller is the server.

`create_avatar`, `create_avatar_from_photo`, `create_session_token` and
`list_avatars` are also still reachable directly on `ZeliClient`. They keep
working and delegate to `client.management`, but they are deprecated. Note the
rename on the way across: `client.create_avatar_from_photo(...)` is
`client.management.create_photo_avatar(...)`, which is the name the other SDK
uses.

## Configuration reference

**`ClientOptions`**

| Field | Default | Meaning |
|-------|---------|---------|
| `server_url` | **none.** `ZELI_SERVER_URL`, or an error. | Base URL of the avatar server. |
| `connect_path` | `/connect` | Persistent WebRTC offer/answer path. |
| `ice_servers` | Google STUN | ICE servers for NAT traversal. |
| `connect_timeout` | `30.0` | Seconds to wait for the media connection. |

**`ZeliClient`** credential arguments, all keyword only:

| Argument | Meaning |
|----------|---------|
| `api_key` | A full API key (`zsk_live_...`). Server side only: it can create avatars, upload voices and mint tokens. |
| `session_token` | A short lived `zsk_temp_...` credential, as a string or as the `SessionToken` the mint route returns. Wins over `api_key` when both are set, because the only reason to have both is a copy paste mid migration and the safer one should win. |
| `token_provider` | An async callable that asks YOUR backend for a fresh session token. Called as expiry approaches and on a refusal, never twice at once. Optional; omitting it leaves today's behaviour unchanged. Refused beside `api_key`. See above. |

**`AvatarConfig`**: **only `avatar_id` and `voice_id` do anything.** `avatar_id`
is sent on `POST /connect`; `voice_id` rides along with every `send_message` /
`talk` / `talkstream`. The remaining fields (`name`, `system_prompt`, `llm_id`,
`language_code`, `emotion_responsive`, `enhance`, `enhance_strength`,
`loop_mode`, `max_session_length_seconds`) have no representation on the wire
protocol: the SDK never transmits them and no Zeli server reads them. Setting one
logs a warning on the `zeli` logger rather than silently doing nothing. Their
server-side equivalents (system prompt, model, TTS provider and voice) are
configured on the box itself, per box rather than per session, and this SDK can
read and write them: `await client.management.get_settings()` and
`await client.management.update_settings({...})`.

## Error handling

```python
from zeli import (
    ZeliError, ConnectionError, ConsentError, SessionError,
    AuthenticationError, TokenProviderError,
)

try:
    async with client.connect() as session:
        await session.wait_until_closed()
except AuthenticationError:
    ...   # server rejected the connection
except TimeoutError:
    ...   # a transport didn't come up in time; also a ConnectionError
except ConnectionError:
    ...   # couldn't reach the server / WebRTC failed
except SessionError:
    ...   # server refused or a session operation failed
except ConsentError:
    ...   # a photo avatar was refused because consent isn't verified
except TokenProviderError:
    ...   # YOUR token endpoint failed; the box and the API key were not involved
except ZeliError as e:
    print(f"[{e.code}] {e.message}")
```

`ConsentError` is deliberately **not** an `AuthenticationError`. The key was
fine; the consent record was not, and reporting a rejected key would send you to
rotate a credential that was never the problem. The JavaScript SDK has no
equivalent yet and reports that refusal as an authentication failure, so if you
are moving code across, catch it here and check the message there.

## Compatibility

The SDK connects over the server's `POST /connect` (media) and, when present, the
`/api/session/ws` control gateway (`talk`, incremental transcripts, streaming
events). Against a server without the gateway it automatically falls back to
`POST /api/chat` for `send_message`. `talk`/`create_talk_stream` then require an
upgraded server.

## Examples

See [`examples/`](examples): `quickstart.py`, `save_video.py`, `talk_stream.py`.
All three take their API key from `ZELI_API_KEY` and let the SDK itself read
`ZELI_SERVER_URL`, so none of them names a box in code:

```bash
export ZELI_SERVER_URL=https://your-box.example.com
export ZELI_API_KEY=zsk_live_...
export ZELI_AVATAR_ID=01-presenter-male__confident   # optional
python examples/quickstart.py
```

## License

Apache-2.0.
