Metadata-Version: 2.3
Name: modal-modo
Version: 0.2.1
Summary: Persistent, keyed actors on Modal backed by SQLite
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Typing :: Typed
Requires-Dist: modal>=1.5.4,<1.6
Requires-Dist: modal-mosql>=0.2.1,<0.3
Requires-Python: >=3.10
Description-Content-Type: text/markdown

# modo

Install from PyPI with `pip install modal-modo`. The Python import name remains
`modo`.

Keyed, stateful actors on Modal. Each actor ID owns a SQLite database and runs
one call at a time.

```python
import modo

app = modo.App("counters")


@app.actor(name="counter")
def counter(ctx: modo.Context, amount: int = 1) -> int:
    return int(
        ctx.sql.exec(
            "INSERT INTO state VALUES (1, ?) "
            "ON CONFLICT(id) DO UPDATE SET value = value + excluded.value "
            "RETURNING value",
            amount,
        ).one()["value"]
    )


@counter.on_entry
def initialize_counter(ctx: modo.Context) -> None:
    ctx.sql.exec(
        "CREATE TABLE IF NOT EXISTS state "
        "(id INTEGER PRIMARY KEY CHECK (id = 1), value INTEGER NOT NULL)"
    )


@app.local_entrypoint()
def main() -> None:
    print(counter.get("user:2").remote(1))
    print(counter.get("user:4").remote(10))
```

`name` is the durable namespace for the actor type's state; it defaults to the
function name, so set it explicitly if state must survive a rename. `get(id)`
selects one actor. Each ID is a Modal class parameter with its own container
pool, routing ownership, database, lock, journal, and transactional KV state.

Actors initialize their native storage on first use. An atomic create-only claim
of an unused ID selects an empty-storage path that skips checkpoint searches and
journal recovery. Duplicate opens and uncertain claim retries recover existing
state. No `prepare()` call or legacy-state migration is needed. Recovery reads
only the actor's own routing and metadata catalog, plus its journal payloads.
See [actor activation](docs/actor-activation.md) for initialization
ordering and [tracing](docs/activation-tracing.md) for cold-call instrumentation.

The call surface follows Modal:

```python
actor = counter.get("user:2")
value = actor.remote(1)
value = await actor.remote.aio(1)
call = actor.spawn(1)
value = actor.local(1)  # runs in-process under .modo/ (or $MODO_LOCAL_ROOT)
```

The default actor image includes the native checksum dependency. Custom images
must install `fastcrc>=0.3.6,<0.4` before calling Modal’s `add_local_*` methods.

Remaining `@app.actor()` keyword arguments (`image`, `timeout`, `secrets`,
`min_containers`, ...) are forwarded to `app.cls()`; `max_containers` is one
per actor ID pool. Setting `min_containers=1` keeps a container warm for each
instantiated ID, so resource usage grows with the number of IDs.

## Contract

- An app named `counters` owns a v2 Volume `counters-actors` and journal
  Dict `counters-actors-wal`.
  Each managed actor also has a `modo-route-…` routing Dict and a small
  `modo-meta-…` Dict for receipts and
  archive boundaries. Each actor uses local SQLite, a Dict journal
  under `modo/wal/<actor>/<id>`, and Volume archives/checkpoints.
- A successful command commits the handler's SQL, KV changes, and serialized
  result in one transaction. The LTX journal record and an immutable commit
  receipt must reach Dict before the call returns. Volume archiving stays
  asynchronous; the first initialization of an actor commits a protocol marker
  to Volume before accepting writes.
- A background flusher batches records into LTX segments and periodically
  checkpoints SQLite: after 64 uncheckpointed transactions, 4 MiB of new archive
  data, or five quiet seconds with outstanding writes. These are maintenance
  targets, not hard limits during publication failures or busy snapshots.
  Only a restored or successfully published checkpoint advances the maintenance
  frontier; failed or busy checkpoints remain eligible for retry. Retained
  fallback archives do not count toward the next checkpoint. An idle actor with
  no outstanding writes does not poll storage. It retains the most recent 128
  transactions in Dict and
  two checkpoint generations, plus archive segments needed by the older image.
- Recovery merges archive and retained journal copies, verifies checksums,
  and checks the committed transaction receipts. Missing required history or
  missing receipts on an initialized actor raises an error instead of silently
  restoring older state. Repair is possible while a valid retained copy exists.
- Commands to one ID are serialized; read-only calls use concurrent snapshots.
  A new resident claims a unique journal
  fencing slot. Every call checks its epoch's immutable ownership and retirement
  keys, including requests returning cached results. Publication checks ownership
  again after the create-only journal write. Recovery validates commit receipts.
- `remote` and `remote.aio` call the host directly using a cached generation and
  hydrated method handle. A rejection refreshes the route and retries the same
  request. `spawn` uses a routing Function so its eventual result includes retries.
  Failed owners retire only their own epoch. A replacement reserves a new epoch
  in the same physical pool and recovers before serving; a fresh pool is the fallback.
- Hosts scale to zero by default; no standby or keeper is provisioned.
- Hosts default to `region="us-east"` to stay near Dict. Pass an explicit
  `region` to `@app.actor()` to choose another placement.
- Different IDs run in separate container pools. Each pool accepts 100 concurrent
  inputs for its actor ID; commands serialize while read-only snapshots can overlap.

See [the durability protocol](docs/durability.md) for failure semantics,
assumptions, retention, and upgrade instructions.

Handlers are synchronous. `Context` exposes `ctx.id`, `ctx.sql`, `ctx.dict`,
and `ctx.files`.

### Dict

```python
ctx.dict.put("status", "ready")
status = ctx.dict.get("status")
ctx.dict.pop("status", None)
```

KV values live in the actor's SQLite database and roll back with SQL on a
handler exception. They persist under `.local()` too. Keys are strings; values
must support Python pickle. Mutating a retrieved value requires a subsequent
`put()` to persist it. Tables prefixed `_modo_` are reserved for library state.

### Files

`ctx.files` is a durable, actor-scoped directory under `/__modo__/files` in the
app's Volume. The container's working directory is the shared files directory,
so relative writes are durable but not actor-isolated. File writes are not
journaled: they become durable at the next background or shutdown
`Volume.commit()`, not when the handler returns.

### Entry hooks

`@actor.on_entry` registers synchronous hooks that run, in order and inside
one SQL transaction, whenever an actor ID becomes resident. If a hook fails,
its SQL rolls back and the handler does not run. A new host runs
the hooks again, so keep them idempotent. `.local()` opens a fresh connection
per call and therefore runs the hooks before each call.

## Alembic migrations

Install `modal-mosql[alembic]` alongside modo to run ordinary Alembic revisions during
actor initialization:

```python
from mosql.alembic import upgrade


@counter.on_entry
def migrate(ctx: modo.Context) -> None:
    upgrade(ctx.sql, config="alembic.ini", revision="head")
```

Each actor's database tracks its own revision. Migrations run when its container
starts; inactive actors migrate when next started. All entry hooks share one
transaction, so a later hook failure also rolls back the migration and its
revision update. Install `modal-mosql[alembic]>=0.2.1,<0.3` in the actor image,
and include the Alembic config and revision files.

Alembic's `env.py` must use the connection supplied by mosql. See the
[setup instructions and supported operations](mosql/README.md#alembic-migrations)
and [example migrations](mosql/examples/migrations).

## Limits

- Each actor ID has its own active host pool with at most one container.
  Failover is scoped to that ID. Retired containers can remain alive while draining.
- Cold starts apply to replacement containers and, for `spawn`, the routing Function.
  Failover depends on Modal detecting/replacing failed containers and provisioning
  the successor; it is not instantaneous. Startup ownership checks and observing
  retirement during pending RPCs avoid waiting for old inputs to be redelivered.
- Modal Dict entries still expire after seven inactive days. This library
  cannot remove that platform limit: expired journal data may be unrecoverable,
  and expired commit receipts cause recovery to fail closed even if archives
  remain. Dict acknowledgment also leaves the newest tail dependent on Dict's
  storage failure domain until archiving completes.
- Temporary Volume failures pause actor calls while the host retries with
  exponential backoff (0.1 to 5 seconds). Successful archiving resumes calls;
  integrity failures remain isolated to the affected actor.
- Deploy callers and hosts with the same Modo API. Native actor restarts and
  container replacement use automatic ownership-based failover. These guarantees
  protect SQL and KV; arbitrary file writes and external network effects require
  application-level coordination.
- Transport retries carry the same generated request ID. For retries made by
  your application after a timeout, reuse `actor.with_request_id(id)`; a new
  ordinary call generates a new ID. Command results are retained indefinitely
  in SQLite, even if the handler only queries state. Explicit `read_only()`
  invocations do not retain results. External effects are not exactly-once.
- Warm calls do no metadata enumeration. Receipts and fresh ownership checks
  still add Dict RPCs. Activation and recovery enumerate only the actor’s own
  metadata. Background
  cleanup tracks known keys instead of scanning the Dict. `spawn` adds a Function
  invocation and RPC hop. Pending calls check retirement every 250 ms; fast
  calls add no such reads. Benchmark your own workload; see the measured
  comparison in [the protocol](docs/durability.md).
- Synchronous input cancellation terminates that actor ID’s Modal container.
  Requests waiting for its lock occupy that ID’s input slots.
- CPU and memory options apply to each actor ID's container. Size them for one
  actor's database and handler working set; total resource usage grows with active IDs.

## Native storage

Modo 0.2 uses a single native actor storage path. Legacy shared-Dict KV and
metadata migration are not supported. `actor.prepare` and the generated
preparation Function have been removed; callers invoke actors directly.
Update dependent callers together with the actor runtime. Ownership fencing,
receipt verification, and the initial Volume commit remain required. Current
catalog keys and actor storage paths retain their format.

Calls now use one request envelope for both commands and reads. Deploy callers
and hosts together. Previously stored request fingerprints are not converted;
reusing an old request ID with the new encoding raises `RequestConflictError`.
Resolve outstanding requests before switching callers. This change does not
reset or delete actor data. The bundled mosql constructor is now `SQLite`;
custom checkpoint stores must implement `validate_recovery(position)`.

## Retrying requests

```python
request = counter.get("user:2").with_request_id("order-123")
value = request.remote(1)
value_again = request.remote(1)  # same stored result, no second mutation
```

Reusing the ID with different serialized arguments raises
`modo.RequestConflictError`. Arguments and results must support Python pickle;
result serialization happens before commit so serialization failures roll back
SQL and KV. Reuse the same argument structure on retries. Successful command
results are retained for the lifetime of the actor; handler exceptions are not
cached. A lost publication response can still represent a committed request.

The router makes at most four host attempts for retirement and selected Modal
transport failures. Application exceptions and corrupt-history errors propagate
without automatic retry. If failover cannot converge, `modo.ActorUnavailableError`
includes `actor_id` and `request_id`; retry using that request ID to resolve an
uncertain outcome. Use your own ID before calling if you must also survive loss
of the caller or the routing Function itself.

## Read-only invocations

Use a new read-only stub for status checks and history queries:

```python
reader = room.get("general").read_only()
history = reader.remote("history", limit=50)
history = await reader.remote.aio("history", limit=50)
call = reader.spawn("history", limit=50)
```

The handler must use a query path. SQLite rejects SQL writes, KV mutations,
DDL, PRAGMAs (including attempts to disable restrictions), and ATTACH/DETACH
before they execute, including nested transactions and cached SQL statements.
This covers managed `ctx.sql` and `ctx.dict`; it cannot prevent arbitrary
Python filesystem or network effects. `.local()` enforces the same restriction.

The read handler does not write or retain its response; activation may perform
durable initialization. A nonresident actor claims ownership, starts empty after
an atomic unused-ID claim or recovers existing state, then initializes storage
and runs `.on_entry` hooks. Hooks may
create tables or migrate state. Only after activation finishes does the
read-only snapshot begin. Repeated reads of an initialized, resident actor add
no request-ledger rows or transaction-journal records. Background maintenance
of earlier commands and activation are separate from that guarantee.

Remote and spawned reads run concurrently with each other and with commands.
Each uses a separate SQL/KV connection pinned to a durably acknowledged snapshot
before the handler starts. A read can return the prior state while an overlapping
command is still executing. Results are serialized and detached inside the
snapshot, then ownership is checked again before returning. A fenced result is
discarded and managed routing retries against the current owner. The snapshot
is consistent during the invocation, not necessarily the latest state at delivery.

Commands still serialize. A writer-priority gate prevents snapshot admission
between local commit and durable journal/receipt acknowledgment; it does not
cover read handler execution. If publication remains unresolved, new reads fail
with `mosql.DurabilityError`. An already admitted snapshot may finish while a
later command publishes. Readers never retry a command's publication themselves.

At most eight snapshots per resident actor execute at once; excess reads fail
immediately with `TimeoutError`, avoiding a queue of reads occupying input slots.
`read_only(timeout=30.0)` sets the snapshot admission/execution budget in seconds.
Long SQL queries are interrupted at the deadline; arbitrary Python work is
checked when it returns and cannot be forcibly interrupted. Activation, ownership
RPCs, and cleanup are outside that budget. A stuck Python handler can therefore
retain its snapshot until it exits. Shared mutable Python state in handlers must
be safe for concurrent use; only managed SQL/KV has snapshot isolation.

Active snapshots delay host shutdown and defer WAL truncation, so commands do not
wait for readers at checkpoint time. Closing a host drains readers before closing
or deleting their database. Long reads can grow WAL storage and compete for CPU
and disk bandwidth. `.local()` retains its existing per-call activation and
serialization behavior; it still enforces read-only SQL/KV and retains no response.

Read mode neither consults nor reserves request IDs in `_modo_requests`, even
when an ID already belongs to a command. Every fresh invocation executes again
and may return newer state despite identical IDs and arguments. Transport
retries also execute again. Retrieving an existing spawned call's result is
still retrieval, not a new invocation. Existing command deduplication is unchanged.

`read_only()` returns a new stub and leaves the original stub's behavior intact.
It composes with `with_request_id()` in either order:

```python
room.get("general").read_only().with_request_id("status-check")
room.get("general").with_request_id("status-check").read_only()
```

The local contention harness in `integration/local_read_contention.py` measured
write p95 of 109.8 ms with serialized reads versus 11.1 ms with concurrent snapshots
under four readers doing 20 ms of work. With no readers, median write latency was
about 2.0 ms in both versions. These are single-run, 60-command measurements using
in-memory Dicts and local fsync, not Modal latency estimates. See the
[raw measurements](docs/concurrent-read-latency.json).

## Examples and verification

[`examples/chatroom.py`](examples/chatroom.py) serves a FastAPI WebSocket chat
where each room ID is an actor owning that room's message history:

```bash
uv run modal run examples/chatroom.py
uv run modal serve examples/chatroom.py
```

[`examples/agent_chat.py`](examples/agent_chat.py) serves an agent-chat app
where each session ID is an actor owning a durable prompt queue and an
append-only event timeline (tool calls, streamed tokens, completion), with a
FastAPI WebSocket frontend:

```bash
uv run modal run examples/agent_chat.py
uv run modal serve examples/agent_chat.py
```

```bash
uv run examples/counter.py
uv run pytest
uv run modal run integration/modal_actor.py
uv run modal run integration/modal_safety.py
uv run modal run integration/modal_failover.py
uv run python integration/modal_rollout.py
uv run modal run integration/modal_dense_actor.py
uv run modal run integration/modal_benchmark.py
uv run modal run integration/modal_write_benchmark.py
```

The single-actor host's command/read/shutdown protocol is model-checked in
`specs/Host.tla` (mosql's durability protocols have their own specs in
`mosql/specs/`). With a JDK and `mosql/specs/bin/tla2tools.jar` (see mosql's
README):

```bash
java -cp mosql/specs/bin/tla2tools.jar tlc2.TLC -deadlock -config specs/Host.cfg specs/Host.tla
```

`specs/HostNoRecheck.cfg` is expected to fail: it keeps the close-race
counterexample as a regression check for the recheck under the host command lock.

`integration/modal_sandbox_actor.py` and its benchmarks prototype one named
Sandbox per actor ID and are independent of the library.

`specs/ActorSafety.tla` checks receipt-based recovery and frontier-checked reads
under takeover, lost responses, and payload loss. `ActorSafetyNoReadCheck.cfg`
and `ActorSafetyNoReceipts.cfg` must fail to demonstrate the corresponding
counterexamples. This is a bounded protocol model, not verification of Modal's
backend or a proof of availability.

`specs/ActorRouting.tla` checks ownership admission and delayed retirement.
`ActorRouting.cfg` must pass; `ActorRoutingUnsafe.cfg` must fail because an old
failure is allowed to retire the current generation instead of its own.

`specs/ActorEpochs.tla` checks the current protocol: separate ownership epochs
and pool addresses, atomic reservations, and delayed startup contenders. Its
positive configuration passes; the unsafe-retirement and missing-admission-check
configurations must fail their respective invariants. See the
[durability protocol and recovery measurements](docs/durability.md).
