Metadata-Version: 2.5
Name: exuvia
Version: 0.1.0
Summary: An AI-first, file-backed DAG workbench for analysts: Python/SQL nodes, per-node schedules, a live graph UI, and a Claude agent that builds pipelines with you.
License-Expression: Apache-2.0
License-File: LICENSE
Requires-Python: >=3.11
Requires-Dist: anthropic>=1.0
Requires-Dist: croniter>=3.0
Requires-Dist: duckdb>=1.1
Requires-Dist: fastapi>=0.115
Requires-Dist: httpx>=0.27
Requires-Dist: openai>=1.40
Requires-Dist: pandas>=2.2
Requires-Dist: pyarrow>=17
Requires-Dist: uvicorn[standard]>=0.30
Requires-Dist: watchfiles>=0.24
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == 'dev'
Description-Content-Type: text/markdown

# exuvia

**An AI-first DAG workbench for analysts and data scientists.**

Marimo showed that notebooks work better as a reactive graph than as a list of cells.
exuvia takes the next step: the graph *is* the interface. Each node is a plain
file (Python or SQL) that can be scheduled on its own, every output is cached
and content-addressed, and an AI agent (Claude, DeepSeek, or any OpenRouter
model) can read, write, run, and debug nodes alongside you on a live canvas.

*Exuvia* (n.): the shell a lobster leaves behind when it molts. Every run
leaves one too: an immutable, content-addressed output you can go back to.

```
universe ──► prices (cron 17:00 weekdays) ──► returns ──► rolling_vol ──► sector_summary.sql
    └──────────────────────────────────────────────────────────────────────────┘
```

> Lives in the Lobster monorepo under `exuvia/` but is fully self-contained —
> it imports nothing from the rest of the repo and reaches Lobster only over
> HTTP, so it can be split into its own repository at any time. Run every
> command below from `exuvia/`.

## Why

| | notebooks | orchestrators (Airflow, Dagster) | **exuvia** |
|---|---|---|---|
| Unit of work | cell | task / asset | **node file** |
| Dependencies | hidden state / reactive cells | declared in code | **function parameters & SQL table names** |
| Scheduling | none | whole DAG / asset jobs | **per node, cron, cascades downstream** |
| Iteration loop | fast | slow (deploy) | **fast — edit, run, see data** |
| AI | sidebar chat | none | **agent with tools over the graph** |
| Storage | `.ipynb` JSON | Python package | **one plain file per node, git-friendly** |

## Quick start

```bash
pip install exuvia          # from PyPI, once released; in this repo: uv sync
exuvia serve examples/markets
# → http://127.0.0.1:7420
```

Set `ANTHROPIC_API_KEY` (Claude) or `OPENROUTER_API_KEY` (DeepSeek and any
other OpenRouter model) in the environment or the project's `.env` to turn on
the agent. See [The agent](#the-agent) for providers.

Start a fresh project:

```bash
exuvia init my-research
exuvia serve my-research
```

## Nodes

One file in `nodes/` = one node. The file name is the node name.

**Python** — parameters are upstream nodes; return a DataFrame (or JSON-able value):

```python
"""20-day annualized realized volatility per ticker."""
import exuvia

@exuvia.node(
    schedule="0 18 * * 1-5",          # optional: re-run on this cron (local time)
    cascade=True,                      # re-run when upstream output changes (default)
    chart={"mark": "line", "encoding": {...}},   # optional Vega-Lite spec, no `data`
)
def rolling_vol(returns):
    out = returns.sort_values("date").copy()
    out["vol_20d"] = out.groupby("ticker")["ret"].transform(lambda s: s.rolling(20).std() * 252**0.5)
    return out.dropna()
```

**SQL** (DuckDB) — upstream nodes are tables:

```sql
-- Latest vol by sector.
-- schedule: */30 9-16 * * 1-5
-- chart: {"mark": "bar", "encoding": {...}}
select u.sector, avg(v.vol_20d) as avg_vol
from universe u join rolling_vol v using (ticker)
group by 1
```

The graph is read **statically** (`ast` for Python, identifier matching for
SQL), so a broken node still shows up in the graph with its error, and nothing
runs until you ask.

## Execution model

- **Content-addressed outputs.** Each run writes its result to
  `.exuvia/artifacts/<node>/<sha>.parquet` (or `.json` / `.pkl`).
- **Freshness is derived, not flagged.** A node is *fresh* when its code hash
  matches its last successful run and each upstream's current artifact is the
  one that run consumed. Otherwise it is *stale*, and the reason says why.
- **Run = target + stale ancestors**, in dependency order, in parallel where
  the graph allows (`EXUVIA_MAX_PARALLEL`, default 4).
- **Subprocess isolation.** Every node runs in its own Python process: crash
  isolation, clean per-run logs streamed live to the UI, killable timeouts
  (`EXUVIA_RUN_TIMEOUT`, default 1800 s).
- **Per-node schedules + cascade.** A scheduled node is forced to re-run on its
  cron. If it produces *different* output, downstream nodes with
  `cascade=True` refresh automatically; identical output stops the cascade.
  Schedules can be paused from the UI without editing code.
- **Run history** lives in `.exuvia/exuvia.db` (SQLite): status, trigger
  (manual / schedule / cascade / agent / cli), timings, logs, schema.
  Schedule state (last fire per node) is persisted too, so restarts neither
  drop nor double-fire schedules.

## Architecture: control plane + executors

```
            ┌──────────── control plane (exuvia serve) ────────────┐
 UI / API ──►  graph · freshness · scheduler · run history · agent │
            │  artifacts (.exuvia/)                                 │
            └───────────────┬──────────────────────────────────────┘
                            │ Executor interface
              ┌─────────────┴──────────────┐
     LocalExecutor (default)        RemoteExecutor ──HTTP──► exuvia executor
     warm subprocess pool                                   (stateless container)
```

The **control plane** is the only stateful part. **Executors** run one node
at a time and return a content-addressed artifact; they hold no project state.

- **Local (default):** a pool of pre-warmed, single-use subprocesses on the
  same machine.
- **Remote:** `exuvia executor` is a plain HTTP service — the same image runs
  on Cloudflare Containers, Fly, Cloud Run, Kubernetes, or a box under a desk.
  Point a control plane at it with `--executor URL` / `EXUVIA_EXECUTOR_URL`.
  Every request flows control plane → executor (project zip, upstream
  artifacts in; logs and artifact out), so a laptop can drive a cloud
  executor without being reachable from it. Project bundles are hash-verified
  and cached on the executor, so unchanged code is not re-sent.

Executor protocol (bearer `EXUVIA_EXECUTOR_TOKEN`):

| Route | |
|---|---|
| `HEAD/PUT /v1/bundles/{hash}` | is this project version cached / upload it (zip, verified against the hash) |
| `PUT /v1/runs/{id}/inputs/{dep}` | upload an upstream artifact |
| `POST /v1/runs/{id}` | run; NDJSON response: `{"type":"log"}` lines then one `{"type":"result"}` |
| `GET /v1/runs/{id}/output` | download the artifact (control plane re-verifies its hash) |
| `POST /v1/runs/{id}/cancel`, `DELETE /v1/runs/{id}` | cancel / clean up |
| `GET /healthz` | liveness |

## Deploying

```bash
docker build -t exuvia .
# executor (default command)
docker run -p 8080:8080 -e EXUVIA_EXECUTOR_TOKEN=change-me exuvia
# control plane on your machine, nodes running in the container
EXUVIA_EXECUTOR_TOKEN=change-me exuvia serve my-project --executor http://localhost:8080
```

Nodes need their packages in the executor image — extend it:

```dockerfile
FROM exuvia
RUN uv pip install --python /opt/venv/bin/python scipy statsmodels
```

**Hosts that sleep or scale to zero.** An in-process scheduler can't fire while
its process is asleep. Run the control plane with `--scheduler external` and
have something call `POST /api/schedule/tick` every minute or so: a Cloudflare
Durable Object alarm or Cron Trigger, a Kubernetes CronJob, or system cron.
Ticks are idempotent; fires missed while asleep are coalesced into one
catch-up run, and paused schedules skip their backlog.

**Auth.** Set `EXUVIA_API_TOKEN` on a hosted control plane: API clients send
`Authorization: Bearer <token>`; browsers open `/?token=<token>` once and get
an HttpOnly cookie. Always set `EXUVIA_EXECUTOR_TOKEN` on a networked executor
— it runs arbitrary code by design.

| Env | Default | |
|---|---|---|
| `EXUVIA_EXECUTOR_URL` / `EXUVIA_EXECUTOR_TOKEN` | local pool | remote executor + its token |
| `EXUVIA_SCHEDULER` | `internal` | `external` = only `POST /api/schedule/tick` advances schedules |
| `EXUVIA_API_TOKEN` | off | shared-secret auth for the control plane API |
| `EXUVIA_MAX_PARALLEL` | 4 | concurrent node runs (control plane and executor) |
| `EXUVIA_RUN_TIMEOUT` | 1800 | seconds before a node run is killed |
| `EXUVIA_BUNDLE_MAX_MB` | 50 | max project size shipped to a remote executor |
| `EXUVIA_EXECUTOR_DIR` | temp | executor scratch space |
| `EXUVIA_PROVIDER` | auto | `anthropic`, `openrouter`, or `openai` (any OpenAI-compatible server) |
| `EXUVIA_MODEL` | per provider | agent model (see below) |
| `EXUVIA_EFFORT` | `medium` | Claude effort level |
| `EXUVIA_BASE_URL` | OpenRouter | base URL for the `openai` provider (Ollama, vLLM, LiteLLM, …) |

## The agent

The agent runs a tool loop over the live workspace: `list_nodes`, `read_node`,
`write_node`, `delete_node`, `run_node`, and `query_data` (DuckDB over every
node's current output). It looks at real column names before writing, runs
what it writes, reads the traceback when something fails, and fixes it. Every
tool call shows up in the chat, and nodes it touches flash on the canvas.

Pick the model with environment variables; the provider is auto-detected from
whichever key is set (Claude wins if both are):

| Provider | Key | Default model |
|---|---|---|
| `anthropic` | `ANTHROPIC_API_KEY` | `claude-opus-5-5` (adaptive thinking, server-side fallback) |
| `openrouter` | `OPENROUTER_API_KEY` (or `OPEN_ROUTER_KEY`) | `deepseek/deepseek-v4-pro` |
| `openai` | `OPENAI_API_KEY` + `EXUVIA_BASE_URL` + `EXUVIA_MODEL` | none (must be set) |

```bash
# DeepSeek through OpenRouter
OPENROUTER_API_KEY=sk-or-... exuvia serve my-project
# any other OpenRouter model
EXUVIA_PROVIDER=openrouter EXUVIA_MODEL=qwen/qwen3-coder exuvia serve my-project
# a local model
EXUVIA_PROVIDER=openai EXUVIA_BASE_URL=http://localhost:11434/v1 EXUVIA_MODEL=qwen3:32b OPENAI_API_KEY=x exuvia serve .
```

The model has to support function calling. The UI's agent header shows which
model is active.

Things to ask it:

- "Pull in daily prices and add a 60-day rolling correlation to SPY"
- "Why did `returns` fail?" (with the node selected)
- "Schedule `prices` every 30 minutes during market hours"
- "Chart the drawdown for each ticker"

## Headless

```bash
exuvia run examples/markets                 # run everything stale
exuvia run examples/markets returns --force # force one node (+ stale upstream)
exuvia status examples/markets
```

`exuvia run` exits non-zero when any node fails, so it drops straight into CI
or a system cron.

## Embedding / using it inside another app

Everything the UI does is a plain HTTP call, so a host app can drive exuvia
headlessly or iframe the UI:

| Route | |
|---|---|
| `GET /api/project` | graph + per-node status |
| `GET/PUT/DELETE /api/nodes/{name}` | read / write / delete a node file |
| `POST /api/nodes` | create from a template |
| `POST /api/nodes/{name}/run` | run a node |
| `POST /api/run-stale` | refresh everything stale |
| `GET /api/nodes/{name}/data` | output preview |
| `POST /api/query` | DuckDB SQL over all outputs |
| `POST /api/nodes/{name}/schedule` | pause / resume a schedule |
| `POST /api/schedule/tick` | evaluate schedules now (for external schedulers) |
| `GET /api/events` | SSE: graph changes, run status, live logs |
| `POST /api/agent` | SSE: agent turn |

Set `EXUVIA_CORS_ORIGINS` for cross-origin hosts. `exuvia.connectors.lobster`
queries the Lobster lake (`LOBSTER_TOKEN`) from any node.

## Development

```bash
mise run sync      # uv sync + npm install
mise run ui-dev    # Vite on :5180 proxying /api → :7420
mise run demo      # exuvia serve examples/markets
mise run ui-build  # bundle the UI into exuvia/static
mise run test
```

Layout:

```
exuvia/
  project.py    static graph discovery (ast / SQL identifiers)
  workspace.py  control plane: freshness, run orchestration, cascade, scheduler, watcher
  executor.py   Executor interface: LocalExecutor (warm pool) + RemoteExecutor (HTTP client)
  executor_service.py  `exuvia executor` HTTP service
  pool.py       pre-warmed single-use worker processes
  runner.py     per-node subprocess entry point
  store.py      SQLite run history
  agent.py      agent tools + provider-neutral tool loop
  llm.py        model providers: Anthropic, OpenAI-compatible (OpenRouter / DeepSeek)
  server.py     FastAPI routes + SSE + static UI
ui/             React + React Flow + CodeMirror + Vega-Lite
examples/       demo projects
```

## Releasing

PyPI releases go through `.github/workflows/exuvia-publish.yml` with trusted
publishing, so no token is stored anywhere. Bump `version` in `pyproject.toml`,
merge, then push a matching tag:

```bash
git tag exuvia-v0.1.0 && git push origin exuvia-v0.1.0
```

The workflow checks the tag matches the version, runs the tests, bundles the
UI into the wheel, and publishes.

## Status

Prototype. Not yet: multi-user auth (only a shared token), executor autoscaling, partitioned
or incremental nodes, non-Python runtimes, secrets management beyond env vars.

## License

Apache-2.0
