TensorSketch

A code-first, visually-editable, durable agentic framework.

TensorSketch is a framework for building AI agents and agentic workflows where:

• Code is the single ground truth. A visual canvas — TensorSketch Studio — is a losslessly-synced projection of your code, never a second, competing source of truth. Edits on the canvas write straight back into the source.
• One type abstraction runs through everything. The same Schema describes tool inputs, structured LLM output, your graph's state, and every node's ports.
• Execution is durable and parallel by construction. A BSP (bulk-synchronous parallel) runtime gives you cycles, deterministic fan-out, dynamic fan-out (Send), and clean checkpoint boundaries — so a run resumes exactly where it left off.
• The core knows interfaces, never implementations. Every provider, tool, database backend, and protocol is chosen by name or passed in, so TensorSketch absorbs new research without a rewrite.

Status: Phases 0, 2, and 3 complete; Phase 1 (code⇄canvas) in progress. Built and tested: the type spine and BSP runtime; durable execution (checkpoints, resume/fork, exactly-once effects); streaming; the full agent layer (tools, three providers, the durable agent loop, structured output, dynamic fan-out); interop and observability (MCP, middleware, tracing + exporters, a name registry, OpenAI/A2A/AG-UI serving, an eval harness with drift detection); and TensorSketch Studio — the visual canvas with a live trace overlay. The public API is pre-1.0 and may still change. See the roadmap and build status.

Install (development)

cd tensorsketch
uv sync
uv run pytest        # the test suite (green on 3.11 + 3.12)

See Installation for details.

Hello, graph

A graph is typed nodes wired over typed state. Here's a two-step pipeline:

import asyncio
from tensorsketch import Schema, Node, Graph, Context, START, END


class State(Schema):        # the shared, typed state — each field is a channel
    text: str
    shout: str = ""


class Shout(Node):
    class In(Schema):       # this node reads state.text ...
        text: str
    class Out(Schema):      # ... and writes state.shout
        shout: str

    async def run(self, ctx: Context, inp: In) -> Out:
        return self.Out(shout=inp.text.upper() + "!")


app = (
    Graph(State)
    .add(Shout)
    .edge(START, "Shout")
    .edge("Shout", END)
).compile()

out = asyncio.run(app.invoke({"text": "hello"}))
print(out.shout)   # HELLO!

out is a fully-typed State, so out.shout is known to be a str.

Next: Getting started builds a routing agent step by step.

Learn the model

• State & channels — how state is stored and reduced.
• Nodes & graphs — typed ports, edges, routing, holes.
• Execution model — how the BSP superstep runtime runs.
• Durability — checkpoints, resume/fork, and exactly-once effects.
• Streaming — live namespaced events, ctx.emit, resumable replay.
• Tools · Providers · Agents — build a durable model+tools agent.
• Composition patterns — map/reduce, parallel, and subgraphs.
• Tracing · Evaluation · Serving — observe, grade, and expose an agent.
• Code ⇄ canvas and Studio — the visual projection of your code.

Design

• Architecture plan — the full "TensorSketch" design.
• Roadmap — phases and what's built so far.

Installation

TensorSketch is pre-1.0 and not yet published to PyPI; install from the source tree. The public API may still change between minor versions.

Requirements

• Python 3.11+ (the runtime uses asyncio.TaskGroup and modern typing).
• uv for environment and dependency management.

Set up the environment

cd tensorsketch
uv sync

This creates a virtual environment in .venv/ and installs TensorSketch (editable) plus the dev tools. The core has only two runtime dependencies — pydantic and typing-extensions — by design: providers, protocols, and storage backends are optional packages, so installing TensorSketch never pulls in an LLM SDK or a database driver.

Optional extras

Everything third-party is opt-in — install only what you use. import tensorsketch never pulls in an LLM SDK, a database driver, or a web framework unless you ask for it.

uv sync --extra anthropic --extra canvas --extra serve
# or, as a dependency:  pip install "tensorsketch-core[anthropic,canvas,serve]"

Verify

uv run pytest            # the test suite
uv run python examples/support_router.py
uv run python examples/counting_loop.py

Development commands

uv run ruff check src tests examples benchmarks   # lint
uv run ruff format src tests examples benchmarks   # format
uv run mypy                                        # strict type-check (src + tests)
uv run pytest                                      # tests
uv run python benchmarks/bench.py                  # micro-benchmarks

Or use the Makefile:

make check    # lint · format check · strict types · tests (exactly what CI runs)
make bench    # micro-benchmarks

The project is configured for strict mypy and a broad ruff ruleset — TensorSketch aims for zero-compromise, fully-typed code. CI (GitHub Actions) runs make check on Python 3.11 and 3.12.

Getting started

This guide builds a small support router: it classifies a user's query and routes it to a specialist. Along the way you'll meet the core of TensorSketch's authoring API — typed state, typed nodes, sequential and conditional edges, and running a graph.

The finished example lives at examples/support_router.py.

1. Define the state

A graph has one state Schema. Every field is a channel the runtime stores and updates.

from typing import Literal
from tensorsketch import Schema

Intent = Literal["billing", "tech", "other"]

class Support(Schema):
    query: str
    intent: Intent = "other"
    answer: str = ""

query is required (the caller provides it); intent and answer have defaults, so they start populated and get overwritten as the graph runs.

2. Write typed nodes

A Node declares an In Schema (the state fields it reads) and an Out Schema (the fields it writes). The body is ordinary async code — TensorSketch never looks inside it.

from tensorsketch import Node, Context

class Classify(Node):
    class In(Schema):
        query: str
    class Out(Schema):
        intent: Intent

    async def run(self, ctx: Context, inp: In) -> Out:
        q = inp.query.lower()
        if any(w in q for w in ("refund", "charge", "invoice")):
            return self.Out(intent="billing")
        if any(w in q for w in ("error", "crash", "bug")):
            return self.Out(intent="tech")
        return self.Out(intent="other")

Add specialist nodes the same way — Billing, Tech, and Fallback, each reading query and writing answer. (In a real agent, these bodies would call an LLM or a tool; the graph would look identical.)

Don't have the body yet? Leave a typed hole

You can declare a node's interface and defer its body:

from tensorsketch import Hole

class Billing(Node):
    class In(Schema):
        query: str
    class Out(Schema):
        answer: str

    async def run(self, ctx: Context, inp: In) -> Out:
        raise Hole("Answer billing questions using the KB tool")

The graph still compiles and type-checks; running it will stop at the hole. Hole is a greppable, type-checked marker for "this node needs code" — later phases turn that description into a real body.

3. Wire the graph

Add nodes, set the entry with START, and connect edges. Use conditional to route dynamically based on state:

from tensorsketch import Graph, START, END

def route(state: Support) -> str:
    return {"billing": "Billing", "tech": "Tech"}.get(state.intent, "Fallback")

app = (
    Graph(Support)
    .add(Classify).add(Billing).add(Tech).add(Fallback)
    .edge(START, "Classify")
    .conditional("Classify", route)
    .edge("Billing", END).edge("Tech", END).edge("Fallback", END)
).compile()

compile() validates the whole graph: every port maps to a real state field of a compatible type, and every edge points at a real node. Mistakes are caught here, before anything runs.

4. Run it

import asyncio

out = asyncio.run(app.invoke({"query": "I'd like a refund on my invoice"}))
print(out.intent)   # billing
print(out.answer)   # [Billing] Looking into your billing question: ...

invoke seeds the state with your input, runs the graph to completion, and returns the final Support state — fully typed, so your editor knows out.intent and out.answer.

Where to go next

• State & channels — accumulate values with reducers.
• Nodes & graphs — fan-out, joins, and holes in depth.
• Execution model — how supersteps actually run.

Studio — the visual canvas

TensorSketch Studio is the visual projection of your code. It renders the graph a source file defines and writes edits straight back into that file. There's no separate diagram to keep in sync: the code is the source of truth, and the canvas is a lossless view of it.

A support router: start → Classify, a route conditional fanning out to Billing / Tech / Fallback, each converging on end — hand-drawn boxes, typed ports, dashed conditional arrows labelled with their routing key.

Launch it

The Studio ships with the canvas extra (it's authoring-time tooling, kept out of the runtime):

pip install tensorsketch-core[canvas]
python -m tensorsketch.canvas examples/support_router.py
# → TensorSketch Studio → http://127.0.0.1:8765   (editing examples/support_router.py)

Open the printed URL. The little bridge is stdlib-only and binds to localhost — it's a developer tool, not a service. It holds no state of its own: every load re-reads the file, every edit re-writes it.

What you see

• Nodes as hand-drawn boxes — the node's name, its typed In ports (green, left) and Out ports (violet, right). A node whose body is an unfilled Hole is tinted and dashed with a needs code tag; the toolbar counts how many nodes still need code across the whole project — click it to list every hole (file, node, and its Hole message).
• start / end as pills.
• Edges — solid arrows for sequential edges; dashed blue arrows for conditional routes, labelled with the routing function and key (e.g. route: billing). A dynamic conditional with no static mapping shows a short route → ? stub, because its targets are decided at runtime.

The layout is computed automatically (a layered DAG), so the graph is readable without manual arranging.

Edit the graph

Every edit changes the wiring — never a node body — and is written back immediately by the code⇄canvas engine:

• Create a node — click + node, give it a name and (optionally) In / Out ports like query: str, context: str. Studio writes an idiomatic class X(Node) stub — a typed hole — into your file and drops the node on the canvas, unwired. Fill its body in code; drag from its right edge to connect it.
• Wire two nodes — drag from a node's right edge onto another node. A sequential edge is added and the file is updated.
• Move a node — drag the node's body to arrange it. Position is presentation, not part of the graph, so it's saved to a sidecar (‹file›.py.layout.json) next to your code — never in it. A node you haven't moved uses the automatic layout; drop the sidecar and you lose only the arrangement.
• Delete an edge — click the arrow to select it, then press <kbd>⌫</kbd> / <kbd>Delete</kbd>.
• Pan / zoom — drag the background; scroll to zoom; reset view reframes.

After each edit the canvas resyncs from the re-extracted code — so what you see is always exactly what the file now says. Node bodies, imports, and comments are preserved untouched; only the graph definition is rewritten, and it's rewritten in the same authoring style you used (fluent chain, statement-style calls, or the >> operators) rather than collapsing to one form.

Live trace overlay

Click ▶ live in the toolbar to watch a run light up on the graph — each node ringed by its status (green ok / red error) with a badge showing latency · cost · call count. It's the same code⇄canvas idea applied to observability: a read-only projection of the run's spans onto the nodes.

This works without Studio holding any run state — which is the whole point of a stateless framework. Your agent runs in its own process and ships each finished span to the bridge; Studio just polls and paints, the way Grafana reads your metrics. Wire it with the trace fan-out:

from tensorsketch import MultiTracer, InMemoryTracer
from tensorsketch.observability.export import http_span_sink

tracer = MultiTracer(
    InMemoryTracer(),                                   # keep the trace for yourself, and…
    http_span_sink("http://127.0.0.1:8765/api/trace"),  # …feed the Studio overlay
)
await agent.invoke(inputs, tracer=tracer)               # nodes light up as spans arrive

The bridge buffers spans in memory only (a bounded live tail, gone when it stops) — the trace's real home is whatever exporter you sent it to. The sink delivers on a background thread and drops silently if Studio isn't running, so it never slows or breaks the run. Statelessness holds: Studio reads your code and reads your telemetry, and owns neither.

Just the aesthetic

The look — hand-drawn shapes, muted ink, a calm canvas — is borrowed from Excalidraw. Everything else is TensorSketch: the blocks are typed nodes with real ports, the arrows are typed edges, and every gesture round-trips through your code.

Scope

Today the Studio renders any graph, creates nodes from a palette, moves them (positions persist in a sidecar), adds/removes edges, and surfaces holes across the project — all through the round-trip, and every rewrite preserves your authoring style (fluent / statement / >>). That completes the code⇄canvas engine; see the roadmap for what's next.

State & channels

A TensorSketch graph has one piece of state, described by a Schema. But that state is not a plain dictionary — under the hood, each field is a channel. A channel owns one value and knows how to fold in new writes via a reducer. This is the mechanism that makes parallel execution well-defined: when several nodes write in the same step, the reducer decides how their writes combine.

Declaring state

from operator import add
from typing import Annotated
from tensorsketch import Schema, Reducer, Topic

class State(Schema):
    answer: str                                    # LastValue (the default)
    scratch: Annotated[list[str], Reducer(add)]    # accumulate concurrent + successive writes
    events: Annotated[list[str], Topic()]          # pub/sub append
    count: int = 0                                  # default → the channel starts at 0

The field's annotation picks its channel. No annotation → LastValue. A Reducer(op) or Topic() marker in Annotated[...] selects a reducing channel.

The channel types

LastValue — keep the most recent write (default)

Holds a single value; reading before any write raises EmptyChannelError. It rejects two writes in the same superstep, because the result would depend on scheduling order:

# If NodeA and NodeB both write `answer` in the same step:
# InvalidUpdateError: LastValue channel received 2 writes in one superstep;
#                     give this field a reducer to combine concurrent writes

That error is a feature — it turns a hidden race into a loud, early failure. If concurrent writes are intended, use a reducer.

BinaryOperatorAggregate — fold with an operator

Selected by Annotated[T, Reducer(op)]. The first write seeds the value; every later write (this step or a future one) is folded in as value = op(value, update):

total: Annotated[int, Reducer(add)]          # sums every write, across the whole run
scratch: Annotated[list[str], Reducer(add)]  # list + list = concatenation → accumulate

This is how a fan-in join collects results from parallel branches, and how a loop accumulates a log across iterations.

Topic — a stream of items

Selected by Annotated[list[T], Topic()]. Each write is a list that gets concatenated into one growing list (so a node whose Out field is list[str] writes a small list, and it's appended). Reads return the whole list. With Topic(accumulate=False) it holds only the current step's writes (useful for one-shot fan-out payloads). A Topic is always "set" — it reads as [] before any write.

How writes become updates

A node returns an Out instance; each Out field is written to the state channel of the same name. Within a superstep, writes are collected but not visible — they are applied together at the barrier at the end of the step, each through its channel's reducer. So every node in a step reads a consistent snapshot and can't see a sibling's mid-step write. (See the execution model.)

Seeding and reading state

• Seeding: invoke(input) writes your input into the matching channels. First, any state field with a default initializes its LastValue channel — so count: int = 0 really starts at 0 — and then your input is applied on top.
• Reading back: when the graph settles, every set channel is read into a validated state instance and returned from invoke.

Roadmap

Channels are also the natural checkpoint unit: a durable journal (a later phase) snapshots channel values at each barrier so a run can resume — or fork — from any superstep. Custom channel types register as plugins.

Nodes & graphs

A TensorSketch program is typed nodes wired over typed state. This page covers the authoring API in depth: ports, edges, routing, fan-out/fan-in, and holes.

Nodes: typed ports, opaque bodies

A Node declares two nested Schemas and one method:

from tensorsketch import Node, Schema, Context

class Classify(Node):
    class In(Schema):        # input ports  = state fields this node READS
        query: str
    class Out(Schema):       # output ports = state fields this node WRITES
        intent: str

    async def run(self, ctx: Context, inp: In) -> Out:
        ...                  # opaque body: LLM calls, tools, parsing — anything async
        return self.Out(intent="billing")

The split is deliberate and is the heart of TensorSketch's design:

• The interface (In/Out) is transparent. It's what the compiler type-checks, what the canvas will draw, and what natural-language generation will target.
• The body (run) is opaque. TensorSketch never introspects it. You get the full power of the host language inside a node without giving up a statically-inspectable graph.

A node's ports are slices of the graph's state: an In field named query reads state.query; an Out field named intent writes state.intent. The field types must be compatible with the state's — checked at compile().

Node names

A node's default name is its class name. Override it when adding:

g.add(Classify)                 # name = "Classify"
g.add(Classify, name="Triage")  # name = "Triage"

Graphs: wiring nodes over state

Graph(StateSchema) is a fluent builder. Every method returns the graph, so wiring chains:

from tensorsketch import Graph, START, END

app = (
    Graph(State)
    .add(Classify).add(Billing).add(Tech)
    .edge(START, "Classify")                # entry
    .conditional("Classify", route)         # dynamic routing out of Classify
    .edge("Billing", END).edge("Tech", END) # terminals
).compile()

Edges

• edge(src, dst) — a sequential edge: after src runs, dst runs.
• edge(START, x) — sets the entry node. (entry("x") is a synonym.)
• edge(x, END) — marks x as terminal along that path.
• Fan-out: add several edges from one node (edge("A", "B"), edge("A", "C")) and both run — in parallel, in the next superstep.
• Fan-in / join: point several nodes at one (edge("B", "D"), edge("C", "D")). When B and C finish together, D runs once, reading their merged writes. Give the joined field a reducer so both writes combine instead of colliding.

Conditional edges (routing)

conditional(src, path, mapping=None) routes dynamically. path receives the current state and returns the next node name — or a list of names for dynamic fan-out, or END to stop:

def route(state: State) -> str:
    return "Billing" if state.intent == "billing" else "Tech"

g.conditional("Classify", route)

With a mapping, path can return short keys that are looked up:

g.conditional("Classify", lambda s: s.intent, {"billing": "Billing", "tech": "Tech"})

A node may have either static edges or a conditional edge, not both — compile() enforces this so a node's successors have one clear source.

router(src, path, mapping=None) is the same thing under an intent-revealing name — reach for it when the point is "pick where to go next," and especially for the dynamic fan-out below.

Dynamic fan-out with Send

A conditional returning a list of node names fans out to those (distinct) nodes. To instead run the same node many times, each on its own input, return a list of Sends. The engine schedules one instance per Send — each its own superstep task with its own payload — and they all merge at the next barrier. This is a graph-level map/reduce:

from tensorsketch import Send

class State(Schema):
    numbers: list[int] = []
    n: int = 0                                    # per-worker input slot
    squares: Annotated[list[int], Reducer(add)] = []   # workers merge here
    total: int = 0

g.router("Split", lambda s: [Send("Square", {"n": x}) for x in s.numbers])
g.edge("Square", "Total")                         # every worker converges on one Total (deduped)

• The payload provides the worker's In fields; any field it omits still reads from shared state. (So the payload keys are ordinary state fields — a Send just overrides them for that instance.) A Schema works too: Send("Square", State.In(...)).
• Workers must write an aggregating channel — a Reducer or Topic — so their results merge at the barrier instead of overwriting one another. A downstream node then reads the merged value.
• Fan-out is durable: each instance journals its ctx.step effects under a distinct key, so a crash mid-fan-out resumes and replays completed workers exactly once.

This is the graph-level counterpart to gather_map (which fans out inside one node's body). Reach for Send when each unit should be its own node/superstep — visible in the trace, checkpointed, and individually durable.

Loops

loop(node, until, *, exit=END) repeats a node until a predicate holds, then continues — sugar over a self-conditional (node → node while not until(state), else → exit):

g.edge(START, "Refine").loop("Refine", until=lambda s: s.score >= 0.9)

Wire the entry separately; loop adds only the repeat/exit branch. The invoke(max_steps=...) recursion limit still bounds a runaway loop.

The >> wiring surface

For wiring that reads like the diagram it describes, Graph.nodes(...) hands back a handle per node, and the handles overload >>:

from tensorsketch import Graph, Router, START, END

g = Graph(Support)
classify, billing, tech = g.nodes(Classify, Billing, Tech)

START >> classify                                       # entry
classify >> Router(route, billing=billing, tech=tech)   # conditional fan-out
billing >> END                                          # terminate a branch
tech >> END

app = g.compile()

The operators just call .add/.edge/.conditional underneath, so this is pure sugar — the compiled graph is identical to the fluent form, and it round-trips through the code⇄canvas engine the same way. The shapes:

a >> b >> c chains left to right (each >> returns its right operand). g["Name"] gives a handle for a node you already added, so you can mix styles. Pick whichever reads best — the fluent builder, statement-by-statement calls, or >>; all three compile to the same graph and extract to the same canvas.

Holes: declare the interface, defer the body

Raise Hole to mark a node as "needs code" while keeping its typed interface:

from tensorsketch import Hole

class BillingAgent(Node):
    class In(Schema):  query: str
    class Out(Schema): answer: str

    async def run(self, ctx: Context, inp: In) -> Out:
        raise Hole("Answer billing questions using the KB tool")

The graph compiles and type-checks; running reaches the hole and raises it. Because holes are greppable and typed, tooling can report "3 nodes need code", and a later phase can compile the description into a real body against the same In/Out contract.

Compile-time validation

compile() rejects structural mistakes before any run, with messages that teach:

• no entry node set;
• an edge (or conditional target) pointing at a node that doesn't exist;
• a port that reads/writes a state field that doesn't exist;
• a port whose type is incompatible with its state channel;
• a node given both static and conditional successors;
• a duplicate node name.

Running a graph

out = await app.invoke(
    {"query": "..."},   # seeds matching state channels (dict or a State instance)
    max_steps=25,       # recursion limit — the safety net for loops
)

invoke returns the final state, typed as your state Schema.

Upcoming

The code⇄canvas engine builds on top of these authoring surfaces — the fluent builder, statement-style calls, and >> all extract to the same graph, which the visual canvas renders and edits. See the roadmap.

Execution model

TensorSketch runs graphs with a BSP (bulk-synchronous parallel) scheduler — the same model behind Google's Pregel and LangGraph's runtime. It's a small idea with big payoffs: cycles, deterministic parallelism, and clean checkpoint boundaries all fall out of it for free.

Supersteps

Execution proceeds in discrete supersteps. Each superstep has three phases:

1. Plan. The scheduler knows the active set — the nodes to run this step. Initially that's the entry node; thereafter it's whoever the previous step's nodes named as successors.
2. Execute. Every active node runs in parallel, and each one reads the same immutable snapshot of state taken at the start of the step. A node cannot observe another node's writes mid-step, so there are no read/write races by construction.
3. Barrier. All writes collected this step are folded into their channels via reducers, atomically. Then each node's successors are computed from the now-consistent state, and they become the next step's active set.

The loop ends when the active set is empty — no node named a successor, so the graph has settled.

Why this model

• Cycles are natural. A node (or a conditional edge) can name a predecessor as a successor. The loop just runs another superstep. No special "loop node" needed — see counting_loop.py.
• Parallelism is deterministic. Sibling nodes in a fan-out all read the same snapshot and merge at the barrier through reducers. The result doesn't depend on which sibling finished first. (This is also why a LastValue channel refuses two writes in one step — that would be order-dependent.)
• The barrier is a checkpoint boundary. At each barrier the state is consistent. That's exactly where the durable journal snapshots channels, so a run can resume or fork from any superstep.

Dynamic fan-out (Send)

Static fan-out runs a fixed set of nodes. To run one node many times — a worker per item, each with its own input — a router returns a list of Sends. Each Send becomes its own unit in the next superstep (its payload overlaid on the shared snapshot); they all merge at the barrier through a reducer channel. It's the same three-phase model — the plan phase simply schedules N instances of one node instead of one. Pending sends ride the checkpoint, and each instance journals its effects under a distinct key, so a crash mid-fan-out resumes and replays completed workers exactly once.

Parallel execution details

Active nodes run under an asyncio.TaskGroup — structured concurrency. If one node raises, its siblings are cancelled cleanly. A single failure propagates as the real exception (a Hole, a NodeError, ...), not an opaque wrapper.

Because a superstep is awaited as a whole, independent LLM/tool calls in a fan-out overlap in wall-clock time — parallelism you get from the graph's shape, without threading anything yourself.

The recursion limit

A guarded loop is expected to terminate. If it doesn't — a routing function that never returns END, say — the scheduler would run forever. The recursion limit is the safety net:

await app.invoke(input, max_steps=25)   # default 25
# GraphRecursionError: exceeded 25 supersteps without halting (still active: ['Spin']) ...

Raise max_steps only once you're confident the loop's exit condition can be met.

A worked trace

For the counting loop (limit=3, starting count=0):

Three supersteps, then the active set is empty and invoke returns the final state. The log channel (a Reducer(add)) accumulated one line per step.

Roadmap

The engine deliberately schedules opaque processes over channels, decoupled from the typed node layer. The durable journal and namespaced streaming already build on that seam. Still ahead: a swappable message transport so the same graph runs single-process or distributed across a gRPC host/workers, and a Rust hot-path core behind the same interface. See the architecture plan.

Durability

A long-running agent will crash, be redeployed, or hit a rate limit mid-flight. When it comes back, two things must be true: it should pick up where it left off, and it must not repeat side effects it already performed (don't charge the card twice, don't re-send the email). TensorSketch gives you both, and the second is where most frameworks stop short.

Durability is opt-in: pass a backend and a thread_id to invoke. Without them, a graph runs purely in memory.

from tensorsketch import InMemoryBackend, SqliteBackend

backend = SqliteBackend("runs.db")        # or InMemoryBackend() for dev/tests
out = await app.invoke({"query": "..."}, thread_id="user-42", backend=backend)

Two layers

TensorSketch's durability has two layers, both behind one Backend interface.

1. Checkpoints — "where are we"

At every superstep barrier, the runtime snapshots the channel values and the set of nodes about to run next, and writes a Checkpoint. Because the barrier is the point where state is consistent (see the execution model), a checkpoint is a clean place to stop and restart. Checkpoints form a tree (each has a parent_id), which is what makes forking possible.

2. The effect journal — "don't do it twice"

Checkpointing alone isn't enough. If a crash happens during a superstep, resuming re-runs that whole step — and naively that repeats every side effect in it. The journal fixes this: wrap a side effect in ctx.step(...) and its result is recorded the moment it completes. On resume, the recorded result is returned from the journal instead of re-running the effect.

class Charge(Node):
    class In(Schema):  user: str
    class Out(Schema): charged: int

    async def run(self, ctx, inp):
        # Runs once, ever — even across crashes and resumes of this thread.
        amount = await ctx.step("charge_card", lambda: payment_api.charge(inp.user))
        return self.Out(charged=amount)

ctx.step(name, fn):

• fn is a zero-argument callable returning an awaitable (e.g. an async API call). Wrap synchronous work with asyncio.to_thread.
• Each call is keyed by (superstep, node, call-order) so replays line up deterministically.
• Pass idempotency_key="..." to dedupe an effect across the whole run (e.g. a payment id), not just per step.
• Results are stored as data. There's no "your orchestration code must be deterministic" rule (the Temporal gotcha) — only the effects you explicitly wrap are memoized.

This is the line between checkpointing and durable execution. See it in action in examples/durable_resume.py: a run charges a card, crashes, resumes — and the payment API is called exactly once.

Resuming

Call invoke again with the same thread_id and backend:

# ... process crashes ...
out = await app.invoke(thread_id="user-42", backend=backend)   # continues from the last checkpoint

If a checkpoint exists for that thread, the run restores its state and continues. Journaled effects are replayed; un-run nodes execute normally. You can pass new input on resume to inject additional state before continuing; omit it to just continue.

Inspecting and forking

state   = app.get_state("user-42", backend)      # latest checkpointed state (or None)
history = app.get_history("user-42", backend)     # every checkpoint, oldest first

# Branch a new run from a past checkpoint with different input — a fresh journal, so effects
# run anew down the new branch. Great for "what if it had routed differently here".
forked = await app.fork(backend, "user-42", history[2].id, "user-42-alt", {"query": "..."})

Backends — bring your own database

TensorSketch is stateless: the framework keeps no durable state of its own. Checkpoints, the effect journal, and the event log all live in whatever Backend you pass. That's what keeps an agent stateless and horizontally scalable — state is in your database, not in the process. Switching stores is a one-line change:

from tensorsketch.runtime.backends import PostgresBackend, RedisBackend

backend = PostgresBackend("postgresql://user:pass@host/db")   # or:
backend = RedisBackend("redis://localhost:6379/0")
out = await app.invoke({"query": "..."}, thread_id="user-42", backend=backend)

The database drivers are optional and imported lazily, so pip install tensorsketch-core never pulls in psycopg or redis. Each connector auto-creates its schema (three thread_id-keyed tables/keys) on first use. You can also hand a connector an existing connection= / client= to share a pool.

The serializer seam

Every backend turns values into bytes through a Serializer. The default, PickleSerializer, round-trips any Python object (Pydantic models included) — but pickle executes code on load, so only point a pickle-backed store at data you trust. Swap the codec to change that:

from tensorsketch.runtime.backends import PostgresBackend

backend = PostgresBackend(dsn, serializer=MySignedSerializer())   # or a JSON/msgpack codec

A Serializer is just dumps(obj) -> bytes / loads(bytes) -> Any.

Writing your own

Any store works — implement the Backend ABC (eight methods: save/latest/get/list checkpoint, record/lookup effect, append/read events) and pass an instance. The SqlBackend base already covers any DB-API 2.0 database; subclass it with your dialect's placeholder, blob type, and upsert clause (that's all PostgresBackend and SqliteBackend are).

Roadmap

Still ahead in this area: transaction-piggybacked exactly-once for DB-backed steps (commit the effect result in the same transaction as the work), optional Temporal/Restate backends, and a distributed runtime. See the architecture plan.

Streaming

invoke gives you the final state. stream gives you the run as it happens — a live sequence of typed events: nodes starting and finishing, the merged state after each superstep, and whatever a node chooses to surface itself. It's how you drive a progress UI, show tokens as they arrive, or feed a live canvas trace.

async for event in app.stream({"query": "..."}):
    print(event.seq, event.type, event.node, event.data)

The event

Every event is an Event:

Event types

Because every event carries run_id, thread_id, and node, a consumer can separate lanes in a multi-agent run — e.g. render each agent's tokens in its own column from a single stream.

Emitting from a node

Call ctx.emit(type, data) inside a node body to push a custom event. It's a no-op if nobody is streaming, so it's safe to leave in.

class Search(Node):
    class In(Schema):  query: str
    class Out(Schema): hits: list[str]

    async def run(self, ctx, inp):
        await ctx.emit("status", {"stage": "searching"})
        results = await do_search(inp.query)
        await ctx.emit("status", {"stage": "done", "count": len(results)})
        return self.Out(hits=results)

This is the seam through which LLM token deltas will flow once provider nodes land — a streaming model call emits a token event per chunk.

Backpressure

The stream is bounded (stream(..., buffer=256)). If the consumer falls behind and the buffer fills, emit waits — which pauses the producing node. So a slow consumer naturally throttles the run instead of blowing up memory. If the consumer stops iterating early, the run is cancelled cleanly (structured concurrency).

Errors

If the run raises, the async for delivers every event emitted up to that point, then raises the original exception. run_end is only emitted on success.

Resumable replay

When you stream a durable run (with a thread_id and backend), every event is also persisted. A consumer that dropped can catch up from where it left off using the seq cursor:

# stream and remember the last seq you saw ...
async for event in app.stream(inp, thread_id="t", backend=backend):
    last = event.seq

# ... later, replay everything after that point
async for event in app.replay("t", backend, since=last + 1):
    handle(event)

replay reads the persisted event log for a thread and yields events with seq >= since, in order — for a completed or in-progress run.

Roadmap

Live-tailing an in-progress run from another process (merging replay catch-up with the live stream), and token-level streaming from provider nodes, build on this foundation. See the architecture plan.

Tools

A tool is a function the model can call. Write an ordinary function, annotate its parameters, add a docstring, and decorate it with @tool — TensorSketch derives the JSON schema the model needs from the signature, so there's nothing to hand-write.

from tensorsketch import tool

@tool
def add(a: int, b: int) -> int:
    """Add two numbers."""
    return a + b

add is now a Tool:

• add.name → "add" (the function name; override with @tool(name=...)).
• add.description → the docstring (override with @tool(description=...)).
• add.json_schema() → the argument schema advertised to the model, inferred from the annotations ({"a": integer, "b": integer}, both required).

Calling a tool

Tool.run(args) validates the arguments against the schema, then invokes the function:

await add.run({"a": 2, "b": 3})        # 5
await add.run({"a": "oops", "b": 3})   # raises ValidationError

Because arguments are validated first, a model that returns the wrong shape fails loudly with a clear error instead of blowing up inside your function.

Sync or async

Tools can be either — an async def tool is awaited automatically:

@tool
async def fetch(url: str) -> str:
    """Fetch a URL."""
    return await http_get(url)

Using tools

Pass tools to an agent; the agent advertises them to the model, runs the ones the model asks for, and feeds the results back:

from tensorsketch import create_agent
agent = create_agent(provider, tools=[add, fetch])

Inside an agent, every tool call is wrapped in ctx.step, so tool side effects are durable — run exactly once, even across a crash and resume.

Context-aware tools

A tool function may declare a ctx parameter. When it does, TensorSketch injects the run Context into it — and never advertises ctx to the model:

@tool
def remember(ctx: Context, note: str) -> str:
    """Persist a note."""
    ...  # the model only sees `note`; `ctx` is injected at call time

This lets a tool journal its own durable steps, emit stream events, or run a sub-graph under the same trace. It's the seam that as_tool uses to run one agent from inside another.

Roadmap

Hosted tools, MCP tool servers, and per-parameter descriptions parsed from the docstring build on this. See the architecture plan.

Providers

A provider is the one seam between TensorSketch and a model API. The core defines only the interface — it depends on no model SDK. Real providers are optional installs; swapping models means swapping a provider, and nothing else in your graph changes.

The interface

class ChatProvider(ABC):
    async def complete(
        self,
        messages: Sequence[Message],
        *,
        tools: Sequence[Tool] | None = None,
        output_schema: type[Schema] | None = None,
        max_tokens: int = 1024,
        **options,
    ) -> Completion: ...

A Completion carries the assistant message (which may include tool_calls), the validated parsed structured output (when an output_schema was requested), and token usage.

FakeProvider — for tests and offline runs

FakeProvider returns canned replies, so you can build and test agents with no API key and full determinism. Drive it with a fixed script or a policy function, and inspect .calls to assert what the model was sent.

from tensorsketch import FakeProvider
from tensorsketch.messages import assistant

provider = FakeProvider([assistant("hello there")])          # scripted
provider = FakeProvider(policy=lambda msgs, tools: assistant("..."))  # dynamic

Built-in providers

Three real providers ship with TensorSketch, each an optional install imported lazily (so importing TensorSketch never pulls in an SDK). They're interchangeable — same interface, same agents:

from tensorsketch.providers.openai import OpenAIProvider

provider = OpenAIProvider(model="gpt-4o")                      # reads OPENAI_API_KEY
provider = OpenAIProvider(base_url="http://localhost:11434/v1")  # any OpenAI-compatible endpoint
agent = create_agent(provider, tools=[...])

OpenAIProvider speaks the Chat Completions API, so it also drives OpenAI-compatible servers (Together, Groq, vLLM, Ollama, …) via base_url — one provider, many backends.

Structured output

Ask a provider for a typed result by passing output_schema; the validated instance comes back on completion.parsed. The generate_structured helper wraps this into a one-liner. Each provider implements it the way its API allows (OpenAI/Google use a JSON-schema response format; Anthropic forces a "respond" tool) — the interface is identical.

Writing a custom provider

Any other backend is a small ChatProvider. Implement complete: map the conversation to your API, call it, map the reply back to a Completion.

from collections.abc import Sequence
from tensorsketch import ChatProvider, Completion, Message, Schema, Tool
from tensorsketch.messages import Message as Msg

class EchoProvider(ChatProvider):
    async def complete(self, messages, *, tools=None, output_schema=None, max_tokens=1024, **opts):
        last_user = next(m.content for m in reversed(messages) if m.role == "user")
        return Completion(message=Msg(role="assistant", content=f"You said: {last_user}"))

That's the whole contract. It works with every agent, graph, and pattern unchanged. The built-in Anthropic, OpenAI, and Google providers are compact real references.

Note: the built-in providers' request/response mappings are covered by unit tests using injected fake clients; verify against the live APIs before relying on them in production (see the decisions log).

Choosing by name (registry)

Sometimes the model or the database shouldn't be hard-coded in an import — it should come from a config file, an environment variable, or a --flag. TensorSketch's registry lets you build a built-in by name:

from tensorsketch import create_provider, create_backend

model = create_provider(cfg["provider"], model=cfg["model"])   # "anthropic" / "openai" / "google"
store = create_backend(cfg["backend"], dsn=cfg["dsn"])         # "sqlite" / "postgres" / "redis"

Swap the strings, swap the stack — nothing else in your graph changes.

It's all one package

This is not a plugin ecosystem you assemble from pieces. Everything first-party ships in the single tensorsketch package. The extras you already know (tensorsketch-core[anthropic], tensorsketch-core[postgres], …) only gate a heavy third-party SDK or driver; they don't split TensorSketch into a core/community/provider constellation you have to wire together. The registry just gives those built-ins a name.

Deliberately small: only the two seams where name-selection genuinely pays off have a registry — providers and backends.

Constructor arguments pass straight through, so create_provider("anthropic", model="…", api_key="…") is exactly AnthropicProvider(model="…", api_key="…").

Lazy by name — nothing imported until you build it

Listing or resolving a name imports nothing optional. create_provider("anthropic", …) is the first moment the Anthropic SDK loads; until then it isn't touched. So import tensorsketch stays free of every provider SDK and DB driver, and you can introspect what's available without paying for it:

from tensorsketch.registry import providers, backends

providers.names()   # ['anthropic', 'fake', 'google', 'openai']  — no SDK imported
backends.names()    # ['memory', 'postgres', 'redis', 'sqlite']

Add your own name

In-process — register a class (or a callable that returns one, to defer a heavy import):

from tensorsketch import register_provider

register_provider("acme", AcmeProvider)
create_provider("acme", model="m1")

From a package you publish — declare an entry point. It's still one pip install for your users and needs zero TensorSketch-side wiring; the name simply appears in the registry once installed:

# your package's pyproject.toml
[project.entry-points."tensorsketch.providers"]
acme = "loom_acme:AcmeProvider"

[project.entry-points."tensorsketch.backends"]
acme-db = "loom_acme:AcmeBackend"

create_provider("acme", model="m1")     # resolves the installed entry point, lazily

An explicit register_* call takes precedence over an installed entry point of the same name, so you can always override.

That's the whole extension story: named factories with lazy loading. No plugin objects, no separate core/community packages. See examples/registry_by_name.py.

Relationship to the provider / backend guides

The registry is only the selector. Writing a provider is still the ChatProvider interface; writing a backend is still the Backend ABC. The registry just gives whatever you built a string name so config can choose it.

Agents

An agent is the model+tools loop: call the model, and if it asks for tools, run them and feed the results back, until the model answers or a budget is reached. In TensorSketch an agent is a single, durable Node — every model and tool call inside the loop is journaled, so a crash mid-loop resumes without repeating a single API call.

The quick path: create_agent

from tensorsketch import create_agent, tool
from tensorsketch.providers.anthropic import AnthropicProvider

@tool
def add(a: int, b: int) -> int:
    """Add two numbers."""
    return a + b

agent = create_agent(
    AnthropicProvider(model="claude-sonnet-4-6"),
    tools=[add],
    system="You are a careful calculator.",
    max_iterations=8,
)

result = await agent.invoke({"query": "what is 2 + 3?"})
print(result.output)     # the final answer
print(result.messages)   # the full transcript (system, user, assistant, tool, ...)

create_agent returns a normal compiled graph, so everything you know applies: run it with invoke, watch it with stream, make it durable with a thread_id + backend.

Durability comes for free

The agent wraps each model call and each tool call in ctx.step. Run it with a backend:

result = await agent.invoke({"query": "..."}, thread_id="chat-1", backend=SqliteBackend("a.db"))

If the process dies at iteration 5, resuming the same thread replays iterations 0–4 from the journal — no repeated LLM calls, no repeated tool side effects, no reasoning drift — and continues from where it stopped. (This is proven in the test suite's crash-harness.)

The budget

max_iterations caps the loop so a misbehaving model can't spin forever. When the budget is hit, the agent returns the best answer it has rather than erroring.

Composing agents into graphs

Agent is just a Node. Drop it into any graph — route to different agents, run several in parallel, put a human-review node after one. create_agent is the convenience wrapper; the primitive composes.

Single calls and structured output

For a one-shot call, use the Llm node:

from tensorsketch import Llm
graph.add(Llm(provider, system="Summarize the input."))

Structured output

generate_structured asks the model for a specific Schema and returns a validated instance — call it inside any node body (pass ctx to journal it):

from tensorsketch import Schema, generate_structured

class Sentiment(Schema):
    label: Literal["positive", "negative", "neutral"]
    confidence: float

result = await generate_structured(provider, Sentiment, "I loved it!", ctx=ctx)
# result is a validated Sentiment

If the model's reply doesn't match the schema, generate_structured feeds the validation error back and asks again (up to max_repairs times) — the validate-and-repair loop — before giving up. Each attempt is journaled when you pass ctx.

Composing agents with patterns

Inside an agent or node, the composition patterns — gather_map (map/reduce over data), parallel (independent calls at once), and run_subgraph (call one graph from another) — let you build map-reduce and multi-step flows that stay durable end to end.

Roadmap

Sub-agent handoff, supervisor/team patterns, and structured agent output build on this loop. Agent memory is intentionally not built into the framework — TensorSketch stays stateless and you bring your own store; see the decisions log. See also the architecture plan.

Multi-agent coordination

Real systems are rarely one agent. A supervisor triages and delegates to specialists; a research agent hands off to a writer; a planner farms sub-tasks out to workers. TensorSketch expresses all of these with one small primitive — as_tool — and nothing new in the runtime.

Agents as tools

An agent is a compiled graph you invoke with a query and read an answer from. as_tool wraps one as a Tool, so another agent can call it exactly like any other tool:

from tensorsketch import as_tool, create_agent

billing = create_agent(provider, tools=[lookup_invoice], system="You handle billing.")
tech = create_agent(provider, tools=[search_kb], system="You debug technical issues.")

supervisor = create_agent(
    provider,
    tools=[
        as_tool(billing, name="billing", description="Answer billing questions."),
        as_tool(tech, name="tech", description="Debug technical problems."),
    ],
    system="Route each request to the right specialist, then relay the answer.",
)

result = await supervisor.invoke({"query": "I want a refund for order #7"})

The supervisor runs the ordinary agent loop: the model sees billing and tech as callable tools, picks one, and the tool call runs that specialist and returns its answer as the tool result. That's the whole supervisor / handoff pattern — the same ReAct loop, one level up.

Why this needs no new machinery

Because a delegation is just a tool call, it inherits everything the agent loop already guarantees:

• Durability. Each tool call is wrapped in ctx.step, so a specialist's whole run is journaled. If the supervisor crashes after delegating but before finishing, resuming replays the specialist's answer from the journal instead of re-running it — no duplicated model calls, no drift.
• One trace for the whole team. The specialist runs under the caller's tracer, so its model and tool spans nest under the delegating tool call. A single trace tree shows the supervisor, each specialist it called, and the per-specialist cost.
• Composability. Specialists are ordinary agents, so they can have their own tools, their own sub-agents (supervisors of supervisors), or a different provider per specialist — a cheap model for triage, a strong one for the hard specialist.

Shaping the call

as_tool defaults match create_agent — a query in, an output out — and exposes a single string argument the supervisor fills:

as_tool(
    graph,
    name="research",
    description="Research a topic and return a summary.",
    input_key="query",        # the state field the sub-agent reads
    output_key="output",      # the state field to return as text
    arg="request",            # the tool argument the caller fills
    arg_description="What to research.",
)

To wrap a graph whose state is shaped differently, point input_key / output_key at its fields:

# a writer graph with state {topic, draft}
as_tool(writer, name="writer", description="Draft a section.",
        input_key="topic", output_key="draft")

Context-aware tools

as_tool is built on a general capability: a tool function may declare a ctx parameter, and TensorSketch injects the run Context into it (it's never shown to the model). That lets a tool journal its own durable steps, emit stream events, or — as as_tool does — run a sub-graph under the same trace:

@tool
def remember(ctx: Context, note: str) -> str:
    """Persist a note durably."""
    ...  # ctx is injected; the model only sees `note`

See the runnable examples/multi_agent.py for a supervisor routing to two specialists, offline.

Composition patterns

The classic control-flow shapes — fan a collection out and reduce it, run several things at once, call one graph from inside another — are helpers you call inside a node's run. Each wraps its work in ctx.step, so they inherit TensorSketch's durability: on resume, work that already finished is replayed from the journal instead of re-run.

gather_map — map/reduce over a collection

Run an async function over every item concurrently, in order, durably:

from tensorsketch import gather_map

class Summarize(Node):
    class In(Schema):  docs: list[str]
    class Out(Schema): summaries: list[str]

    async def run(self, ctx, inp):
        async def summarize(doc: str) -> str:
            return await llm_summarize(doc)     # a real model/tool call
        summaries = await gather_map(ctx, inp.docs, summarize, max_concurrency=5)
        return self.Out(summaries=summaries)

Because a concurrent map can't depend on completion order, each item is journaled under an explicit, deterministic key — so if the process dies halfway through 100 documents, resuming only processes the ones that didn't finish. max_concurrency caps how many run at once.

parallel — run independent calls at once

from tensorsketch import parallel

async def run(self, ctx, inp):
    weather, news = await parallel(
        ctx,
        lambda: get_weather(inp.city),
        lambda: get_news(inp.city),
    )
    ...

Results come back in argument order, and each call is durable.

run_subgraph — compose graphs

Build small graphs and call them from larger ones. run_subgraph runs a compiled graph and returns its final (typed) state; pass ctx to journal the whole call as one durable step.

from tensorsketch import run_subgraph

async def run(self, ctx, inp):
    result = await run_subgraph(research_graph, {"topic": inp.topic}, ctx=ctx)
    return self.Out(report=result.summary)

This is the composition primitive: an agent can call a sub-workflow, which can call another, each a normal graph you can test in isolation.

Why body helpers (for now)

These are functions you call inside a node, not new graph-builder syntax. That keeps them fully type-safe and lets them compose freely inside agent loops. A graph-level dynamic fan-out (spawning parallel node instances that each appear as their own superstep) is a planned runtime addition; see the architecture plan.

Middleware

Middleware is TensorSketch's extensibility seam for agents. Each piece wraps a call — a model call or a tool call — like an onion: it runs code before, after, or instead of the next layer. One uniform mechanism covers what production agents need — retries, tracing, caching, guardrails, cost accounting, error handling — without touching the agent loop.

from tensorsketch import create_agent, RetryMiddleware, ObservabilityMiddleware

agent = create_agent(
    provider,
    tools=tools,
    middleware=[RetryMiddleware(attempts=3), ObservabilityMiddleware()],
)

Writing one

Subclass Middleware and override the hook you care about — wrap_model, wrap_tool, or both. Each receives a request and a call_next to invoke the rest of the stack. The default passes straight through, so you only implement what you need.

from tensorsketch import Middleware
from tensorsketch.messages import system

class Guardrail(Middleware):
    async def wrap_model(self, request, call_next):
        request.messages.append(system("Answer in one short sentence."))
        return await call_next(request)   # ← the rest of the stack, then the real model call

A middleware can:

• observe — time the call, log it, count tokens (request / the returned Completion);
• modify — mutate request.messages / request.tools / request.options before call_next, or transform the result after;
• short-circuit — return a value without calling call_next (a cache hit, a blocked call);
• handle errors — wrap call_next in try/except to retry, fall back, or annotate.

request is a ModelRequest (messages, tools, output_schema, options, ctx, node) or a ToolRequest (call, tool, ctx, node). ctx.emit(...) from either lets a middleware push events into the run's stream.

Ordering and durability

The list wraps outermost-first — [A, B] means A sees the call before B and the result after. The whole stack runs inside the agent's durable ctx.step, so a wrapped call is journaled as one effect: on resume, the recorded result is replayed and the middleware, model, and tools are not re-run. A retry that eventually succeeds is journaled as that single success.

Built-ins

RetryMiddleware

Retries model and tool calls on error — the on_model_error / on_tool_error primitive.

RetryMiddleware(attempts=3, backoff=0.5, retry_on=(TimeoutError, ConnectionError))

Retries up to attempts times on any exception in retry_on (default: any Exception), with optional exponential backoff seconds. Each retry emits a retry event.

ObservabilityMiddleware

Emits model_call / tool_call start / end / error events (with duration and token counts) into the run's stream — a no-op when nobody is streaming. This is the seam a logging or OpenTelemetry exporter plugs into; it never changes results.

model_call  {'phase': 'start', 'node': 'agent'}
model_call  {'phase': 'error', 'error': 'rate limited (503)'}
retry       {'target': 'model', 'attempt': 1, 'error': 'rate limited (503)'}
model_call  {'phase': 'start', 'node': 'agent'}
model_call  {'phase': 'end', 'node': 'agent', 'duration_ms': 0.8, 'output_tokens': 12}
tool_call   {'phase': 'start', 'tool': 'weather'}
tool_call   {'phase': 'end', 'tool': 'weather', 'duration_ms': 0.3}

See examples/middleware_retry.py for the whole thing running offline.

Roadmap

Node/run-level middleware, and packaging middleware + tools + providers as discoverable plugins (entry-point registries), build on this same seam. See the roadmap.

Tracing & observability

TensorSketch traces itself through its own tracing abstraction — not a third-party SDK. That's a deliberate choice: you shouldn't have to adopt OpenTelemetry (or anything) to see what your agent did, how long it took, and what it cost. The core ships a real, built-in tracer; OpenTelemetry, a file/JSON exporter, or a live Studio overlay are then just adapters over the same spans — optional, never required.

The shape of it

A span is one timed unit of work — a run, a node, a model call, a tool call, or anything you mark yourself — with a duration, a status, and free-form attributes (model, tokens, cost, …). Spans nest into a tree. Tracing is always available and zero-cost by default: every run carries a tracer (NoopTracer unless you pass one), so a trace appears the moment you supply a real one.

from tensorsketch import InMemoryTracer

tracer = InMemoryTracer()
out = await agent.invoke({"query": "what is tensorsketch?"}, tracer=tracer)

print(tracer.trace.render())
print(tracer.trace.summary())

  run (12.4ms)
    agent (11.9ms)
      model_call (7.1ms)  [claude-sonnet-4-6, 120 out-tok, $0.004800]
      tool_call (0.3ms)   [search]
      model_call (4.0ms)  [claude-sonnet-4-6, 120 out-tok, $0.005400]

{'duration_ms': 12.4, 'spans': 5, 'model_calls': 2, 'tool_calls': 1,
 'input_tokens': 2200, 'output_tokens': 240, 'cost_usd': 0.0102, 'errors': 0}

The engine opens the run and node spans; agents open model and tool spans and record the model id, token usage, and estimated cost. A model call that's replayed from the durable journal on resume does no work, so it produces no span — the trace always reflects what actually ran.

What you get for evaluation

trace.summary() gives the headline numbers; the Trace also exposes them individually, which is exactly what a cost/latency/correctness eval consumes:

Cost

Cost is estimated from token usage and the model id via a small, overridable price table (estimate_cost, DEFAULT_PRICES — USD per million tokens). Pricing changes and every deployment differs, so pass your own table rather than trusting the default as gospel.

Trace your own work

Inside any node body, ctx.span(...) marks a sub-step — it nests automatically under the node:

async def run(self, ctx, inp):
    with ctx.span("parse-invoice"):
        data = parse(inp.document)
    with ctx.span("score", model="scorer-v2"):
        return self.Out(score=await score(data))

Exporters

The sink is just another Tracer, so switching where spans go changes nothing else.

File (JSON Lines) — built in, no dependencies

FileTracer streams one JSON object per span to a file as each span closes — a durable, grep/jq-friendly trace log:

from tensorsketch.observability.export import FileTracer

with FileTracer("run.jsonl") as tracer:
    await app.invoke({...}, tracer=tracer)

{"name": "model_call", "kind": "model", "duration_ms": 7.1, "status": "ok",
 "attributes": {"gen_ai.request.model": "claude-sonnet-4-6", "gen_ai.usage.cost_usd": 0.0048}, ...}

OpenTelemetry — optional adapter

If you do use OTel, it's one import away — never a requirement. Configure OTel however you like, then hand TensorSketch an OTelTracer; every TensorSketch span becomes an OTel span (nesting and attributes preserved, and TensorSketch's GenAI-style attribute names map onto OTel's GenAI conventions):

pip install tensorsketch-core[otel]

from tensorsketch.observability.otel import OTelTracer

await app.invoke({...}, tracer=OTelTracer())   # uses your globally-configured OTel tracer

Fan-out to several sinks — MultiTracer

You often want more than one destination for the same run: keep a file log and an in-memory Trace to assert on and a live feed for a viewer. MultiTracer owns a single span lifecycle (one trace_id, correct nesting and timing) and hands each finished span to every sink, so the tree is identical everywhere — no double-counting, no drift between destinations:

from tensorsketch import InMemoryTracer, MultiTracer, FileTracer

collector = InMemoryTracer()
with FileTracer("run.jsonl") as file_tracer:
    tracer = MultiTracer(collector, file_tracer, lambda span: feed.send(span.to_dict()))
    await app.invoke({...}, tracer=tracer)

print(collector.trace.render())   # exactly the spans that were written to run.jsonl

A sink is either another RecordingTracer (its _record consumes the span) or any Callable[[Span], None] — the callable form is the seam a live overlay plugs into (below). (OTelTracer drives OTel's own live span context, so it isn't a RecordingTracer sink; use the OTel SDK's exporter pipeline to fan OTel out.)

Write your own

Tracer has one method — span(name, *, kind, **attributes), a context manager. For a collecting or streaming sink, subclass RecordingTracer and override _record(span) — it owns the lifecycle (timing, status, nesting) and hands you each finished Span (with .to_dict()). That's all FileTracer is. For a bridge to a system with its own context (like OTel), implement Tracer directly.

See examples/tracing.py for the whole thing running offline.

Relationship to streaming and middleware

Three complementary seams:

• Tracing (this page) — an always-on tree of timed spans for after-the-fact analysis.
• Streaming — live events as a run progresses (for UIs).
• Middleware — intercept model/tool calls (retries, guardrails); ObservabilityMiddleware bridges middleware into the live event stream.

Roadmap

The live trace overlay in Studio is built on this same span model — feed it with MultiTracer(..., http_span_sink(url)). Richer per-provider cost/latency metrics are still to come. The eval harness already consumes traces directly for cost / latency / correctness.

Evaluation

An agent isn't a function from prompt to string — it's a path-dependent process that reasons, calls tools, and changes state. So testing it means measuring not just what it produced, but the whole trajectory it took to get there, across multiple runs (agents are non-deterministic). TensorSketch's eval harness is built for exactly that, and it consumes the trace TensorSketch already records.

It lives in the one tensorsketch package — no extra to install. LlmJudge just needs a provider.

from tensorsketch.eval import Case, Suite, evaluate, Contains, ToolCalled, StepEfficiency

suite = Suite("capitals", [
    Case("france", {"query": "Capital of France?"},
         graders=[Contains("Paris"), ToolCalled("search"), StepEfficiency(optimal_steps=3)],
         trials=3),
])

report = await evaluate(agent, suite)
print(report.render())
report.require(pass_pow_k=1.0)   # gate CI on the result

The anatomy of a test

A trial passes only if every one of its graders passes.

Graders: a hybrid ecosystem

No single grading mechanism is enough, so the built-ins span the three architectures from the research — with LLM judges for what code can't check, and code for everything it can.

Code-based — fast, cheap, reproducible

ToolArgs grades the payload, not just the path — TensorSketch's tool spans carry the arguments each tool was called with, so you can assert the agent passed the right parameters:

ToolArgs("sql_query", lambda args: args["table"] == "customers")

LLM-as-a-judge — for open-ended answers

When exact matching is too brittle, LlmJudge scores one atomic criterion with a binary verdict — the "one criterion, one failure mode" style that keeps a judge consistent and calibratable. Compose several for a multi-dimensional rubric:

from tensorsketch.eval import LlmJudge

judges = [
    LlmJudge(provider, "The answer cites at least one source."),
    LlmJudge(provider, "The tone is professional and free of hedging."),
]

The judge runs on your provider and its call is not part of the agent's trace, so it never counts against the agent's cost or latency. Calibration is your job: validate a judge against human labels before trusting it, and keep each criterion narrow.

Human-in-the-loop

The gold standard for calibrating judges and labelling hard cases. TensorSketch gives you the data — a Trial with its full transcript — but the annotation queue/UI is out of scope for the library (see deferred).

Metrics: outcome and trajectory

report.summary() and the individual properties give both halves the research calls for:

• Outcome — completion_rate (fraction of trials that succeeded), pass_at_k (succeeded at least once across k trials — retry-friendly) and pass_pow_k (succeeded every time — demands consistency), plus mean_cost / mean_latency.
• Trajectory — surfaced through the tool/step graders above; grader_breakdown() shows the pass rate per grader across all trials, so you see which criterion is failing.

pass^k is the strict one: an agent that's right 4 times out of 5 scores pass_pow_k = 0. Use it where consistency matters; use pass@k where a retry or re-prompt is acceptable.

Isolation

Trials run behind a Sandbox seam. Crucially — in every sandbox — the graders run in the harness, never inside the agent's execution, so an agent can't tamper with its own grade (the way agents have gamed benchmarks that graded in-process). The default InProcessSandbox runs the trial here, under a fresh tracer, with a fresh environment per trial so trials never cross-contaminate — the right choice for LLM/tool/trajectory evaluation, where the Case controls the tools. Stronger isolation (a subprocess, a container, a remote runner) is a future Sandbox behind the same seam.

Storing & viewing results — TensorSketch emits, you own the store

TensorSketch is stateless, so it never bundles a results database. Every result serializes to a plain JSON record — report.to_dict() (and TrialResult.to_dict()) — and you push it through a Reporter sink to wherever your team looks: a file, a warehouse, a dashboard, any database. It's the same exporter pattern as tracing, just for the scores instead of the spans.

from tensorsketch.eval import JsonlReporter, CallbackReporter

# a durable, jq-friendly log file (no dependencies)
await evaluate(agent, suite, reporter=JsonlReporter("evals.jsonl"))

# ...or straight into your own store — a Reporter is one method, sync or async
await evaluate(agent, suite, reporter=CallbackReporter(lambda record: warehouse.insert(record)))

Connecting a database is a few lines — implement emit(record):

class PostgresReporter:
    def __init__(self, pool):
        self.pool = pool

    async def emit(self, record):
        await self.pool.execute("insert into evals(doc) values ($1)", record)

For the transcript side (the raw spans, tokens, cost), the tracer exporters already cover it: FileTracer (JSONL), OTelTracer (→ Grafana / Jaeger / Honeycomb / Datadog), or a custom RecordingTracer._record. So traces and eval scores can each go to the store that suits them.

Where this fits the lifecycle

Offline (above) — curated goldens, run as a regression suite, gating CI with report.require(...).

Online — once the agent is live there's no ground truth for a novel query, so you score what a run actually did with reference-free graders (safety, tool-call failures, cost/latency budgets, on-policy LlmJudge criteria). It reuses the same graders — they already read a Trace — via score(...) for a single run or an OnlineMonitor that samples and emits:

from tensorsketch.eval import OnlineMonitor, JsonlReporter, LatencyBudget, LlmJudge

monitor = OnlineMonitor(
    [LlmJudge(judge, "The answer stays on-policy."), LatencyBudget(3000)],
    reporter=JsonlReporter("online.jsonl"),
    sample=0.1,   # score 10% of traffic
)

tracer = InMemoryTracer()
state = await agent.invoke(inputs, tracer=tracer)
await monitor.observe(state, tracer.trace)   # off the response path — sample, score, emit

The feedback loop: capture a production failure, correct the expected outcome, and add it as a new Case — every novel failure becomes a permanent regression test. (The annotation UI that automates this is deferred; the data — a Trial with its transcript — is here.)

Drift detection — alerting on the online stream

Scoring each run is one thing; noticing that the aggregate has quietly regressed is another. A DriftMonitor watches the same stream OnlineMonitor emits and raises a DriftAlert when behavior shifts from a Baseline (usually your last green offline eval, via Baseline.from_report(report)):

• a drop in pass rate — overall or for one grader — via a two-proportion z-test against the baseline (the right test for binary pass/fail), so a safety grader collapsing trips on its own;
• a change-point in cost or latency via the Page-Hinkley test — the canonical O(1) streaming detector for a shift in a numeric mean (fed the value relative to the baseline mean, so one threshold works across dollars and milliseconds).

DriftMonitor is a Reporter, so it drops straight into the online monitor. Use MultiReporter to persist every result and watch for drift from one hand-off:

from tensorsketch.eval import Baseline, DriftMonitor, MultiReporter, OnlineMonitor, JsonlReporter

baseline = Baseline.from_report(offline_report)          # what "healthy" looked like
drift = DriftMonitor(baseline, reporter=JsonlReporter("drift-alerts.jsonl"))

monitor = OnlineMonitor(
    graders,
    reporter=MultiReporter(JsonlReporter("online.jsonl"), drift),  # store + detect in one pass
    sample=0.1,
)

The rolling window lives in this process's memory — the detector persists nothing. Consistent with the rest of TensorSketch: it emits alerts to your sink and never owns a drift database. A sustained regression is latched, so it fires once (not on every subsequent record) until it recovers.

What's deferred

Logged in decisions so nothing is lost:

• Stronger sandboxes — SubprocessSandbox, DockerSandbox, and remote runners behind the seam.
• Distributional drift — DriftMonitor (pass-rate + cost/latency) ships now; population-shift detectors (PSI / KL / KS over a reference distribution) and routing alerts to a pager are next.
• Annotation queue — the human-in-the-loop UI and the trace→golden pipeline as tooling.
• More trajectory metrics — memory hit rate (for memory-enabled agents) and scope-adherence.
• Judge calibration tooling — measuring a judge's rank correlation against human labels.

See examples/evaluation.py for the whole thing running offline.

MCP interop

The Model Context Protocol is the emerging standard for connecting agents to tools. TensorSketch speaks it both ways: use anyone's MCP tools inside a TensorSketch agent, and expose your TensorSketch tools to any MCP client (Claude Desktop, another agent, an IDE).

It's an optional install — the MCP SDK is imported only when you use this module, so the core stays dependency-free:

pip install tensorsketch-core[mcp]

from tensorsketch.interop.mcp import mcp_tools, stdio_session, build_mcp_server, serve_stdio

Consume external tools (client)

Connect to a server, wrap its tools as TensorSketch Tools, and hand them to an agent. Each call is forwarded over MCP; the remote tool's JSON Schema becomes the tool's advertised schema, so the model sees exactly the right arguments.

from tensorsketch import create_agent
from tensorsketch.interop.mcp import mcp_tools, stdio_session

async with stdio_session("python", "weather_server.py") as session:
    tools = await mcp_tools(session)              # remote tools → TensorSketch tools
    agent = create_agent(provider, tools=tools)
    result = await agent.invoke({"query": "what's the weather in Paris?"})

stdio_session(command, *args) launches a server as a subprocess and yields an initialized session. Already have a session (SSE / streamable-HTTP transport)? Pass it straight to mcp_tools(session) — the wrapping is transport-agnostic.

A tool result comes back as a Python value: structured content (a dict) when the server returns one, otherwise the joined text. A server-side error surfaces as MCPError.

Expose your tools (server)

Turn TensorSketch tools into an MCP server any client can call. A TensorSketch tool's schema and description become the MCP tool; a call runs tool.run(...).

from tensorsketch import tool
from tensorsketch.interop.mcp import serve_stdio

@tool
def add(a: int, b: int) -> int:
    """Add two integers."""
    return a + b

await serve_stdio([add], name="my-tools")   # runs over stdio until closed

Need to embed the server (a custom transport, or tests)? build_mcp_server(tools, name=...) returns the configured low-level MCP Server for you to run.

How the bridge works

There's no magic layer — the adapter is thin and symmetric:

This works because a TensorSketch Tool separates its advertised schema from its invocation: a local @tool derives the schema from the function signature, while a remote tool carries the raw JSON Schema the server provided. Same Tool type either way, so remote and local tools mix freely in one agent.

Try it

examples/mcp_interop.py exposes TensorSketch tools as an MCP server and then uses them from a TensorSketch agent — over the real protocol, connected in-memory, so it runs offline with no subprocess.

Serving a TensorSketch agent

A TensorSketch agent is a graph you can run in-process — but often you want to put it behind an endpoint so other software can call it. TensorSketch serves an agent over three standard protocols, each as a mountable ASGI app:

It's an optional install — the web stack is never pulled into the core:

pip install tensorsketch-core[serve]

Each factory returns a Starlette app (itself an ASGI app), so you run it with any ASGI server or mount it under an existing one:

from tensorsketch import create_agent
from tensorsketch.serve import openai_app

agent = create_agent(provider, tools=[...])
app = openai_app(agent, model="my-bot")   # run:  uvicorn mymodule:app

import tensorsketch still imports no web framework — Starlette and httpx load only inside tensorsketch.serve.

OpenAI-compatible

openai_app exposes POST /v1/chat/completions (streaming and non-streaming) and GET /v1/models. Point the OpenAI SDK at it and nothing else in your code changes:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
client.chat.completions.create(model="my-bot", messages=[{"role": "user", "content": "hi"}])

The last user message becomes the agent's input; the agent's answer comes back as the assistant message. Streaming uses real SSE framing (chat.completion.chunk deltas ending in [DONE]).

A2A (Agent2Agent)

a2a_app publishes an Agent Card for capability discovery at /.well-known/agent.json and answers A2A's JSON-RPC methods message/send (one-shot) and message/stream (SSE task updates):

from tensorsketch.serve import a2a_app, AgentCard

app = a2a_app(agent, card=AgentCard(name="research-bot", description="Searches and summarizes."))

The other direction — calling a remote A2A agent from inside a graph — is a tool:

from tensorsketch.serve import a2a_tool

delegate = a2a_tool("https://other-agent.example/", name="ask_specialist")
agent = create_agent(provider, tools=[delegate])   # your agent can now hand off to theirs

So TensorSketch is on both sides of A2A: expose your agent to the ecosystem, and consume anyone else's.

AG-UI

agui_app exposes a single POST / that accepts an AG-UI RunAgentInput and streams the run back as typed UI events — RUN_STARTED, TEXT_MESSAGE_START / _CONTENT / _END, a STATE_SNAPSHOT of the final state, and RUN_FINISHED (or RUN_ERROR) — which a CopilotKit/AG-UI frontend renders directly.

Serving a non-agent graph

The factories default to the create_agent shape — a query in, an output out. For a graph with a different state, pass to_input / to_reply (see ChatAdapter) to map the request messages to your input and pull the reply out of your state:

app = openai_app(
    my_graph,
    to_input=lambda messages: {"prompt": messages[-1].content},
    to_reply=lambda state: state.answer,
)

What's deferred

These are pragmatic, current-shaped implementations, not the entire surface of each spec:

• Token-level streaming. Providers don't stream tokens yet (that will flow through ctx.emit); today a streamed reply is the completed text sliced into SSE deltas — real framing, so it becomes true token streaming with no client change.
• A2A covers the agent card + message/send / message/stream with a completed-task result; the full task store, tasks/get / tasks/cancel, and push notifications are not implemented.
• Multi-turn history and inbound tools (the request carrying prior turns or tool defs) aren't wired into the default agent, which builds its own conversation from a single query.

See examples/serving.py for all three running offline in-process.

Relationship to the other seams

Serving exposes an agent; MCP interop connects tools; tracing records what a served run did. AG-UI is the UI-facing cousin of streaming — it's the same run events, re-encoded in a protocol a frontend understands.

Code ⇄ canvas

TensorSketch's defining idea: your code is the single source of truth, and the visual canvas is a losslessly-synced projection of it — never a second, competing source. You sketch on the canvas or write code, switch freely, and the two stay in sync.

This page covers the engine that makes that possible: extraction (code → graph), write-back (graph → code), and the round-trip invariant that keeps them honest. The Studio is the visual canvas built on top of it — launch it with python -m tensorsketch.canvas <file>.

What round-trips, and what doesn't

Only two things round-trip between code and canvas:

• The wiring — which nodes connect to which (the Graph(...).add(...).edge(...) structure).
• The typed interfaces — each node's In/Out ports.

Node bodies do not. A node's run method is opaque — the canvas draws the node as a box with its ports and never looks inside. This isn't a limitation to fix later; it's a computability necessity. "What are this node's outgoing edges?" is a semantic property of a Turing-complete program, which is undecidable (Rice's theorem). So TensorSketch keeps wiring a declarative, syntactic surface that can round-trip, and treats bodies as opaque. This is exactly why it works where "turn my arbitrary code into a diagram" tools don't.

Extraction

from tensorsketch.canvas import extract

ir = extract(open("support_agent.py").read())
ir.to_dict()   # JSON a canvas can render

extract parses the source with a CST (concrete syntax tree) and reads:

• every class X(Node) — its name, its In/Out port fields (name + type), and whether its body is an unfilled hole (raise Hole(...));
• the graph's wiring — the nodes, entry, and edges (conditional edges carry their routing function; a mapping expands to one edge per target). router(...) reads as a conditional (its intent-named alias), and loop(node, until, *, exit=END) reads as a two-branch conditional — one edge back to the node (a self-loop the canvas draws as an arc), one to exit.

Dynamic targets can't round-trip — by design. When a route is decided inside an opaque callable — router("split", lambda s: [Send("worker", …) for … ]), or a loop/conditional whose predicate is a lambda — the targets aren't statically knowable (Rice's theorem again). Extraction shows these as a dynamic-route stub (a ? off the node) rather than inventing edges. The graph still runs; the canvas is just honest that the destination is computed at runtime.

Wiring is read no matter which authoring style the source uses — they all fold to the same GraphIR:

• a fluent chain — app = Graph(S).add(A).edge(x, y).conditional(...);
• statement style — g = Graph(S), then g.add(A), g.edge(x, y) on their own lines;
• the >> surface — a, b = g.nodes(A, B), then START >> a, a >> Router(fn, ...).

It produces a GraphIR — plain data (GraphIR / NodeIR / EdgeIR / Port) with a to_dict() for JSON.

It works on incomplete code

Extraction is purely syntactic — it never imports or runs your module. So it works on code that isn't finished: undefined names, missing imports, and unfilled holes are all fine. That's essential — you need to see the graph while you're building it, holes and all. (A node whose body is raise Hole(...) shows up with has_hole=True, so tooling can surface "3 nodes need code".)

Surfacing holes across a project

The same syntactic reading powers a what's left to implement? view over a whole codebase, not just one file:

from tensorsketch.canvas import find_holes

for hole in find_holes("src/"):
    print(f"{hole.file}:{hole.line}  {hole.node}  — {hole.spec}")

find_holes(*paths) walks files/directories for node classes still stubbed with raise Hole(...), returning a HoleRef (file, node, the Hole message, line) for each. Unreadable or unparseable files are skipped, so a broken file elsewhere never hides the rest. The CLI wraps it — python -m tensorsketch.canvas --holes src/ — and the Studio counts them in its toolbar (click to list them across the project).

Write-back

A canvas edit changes the wiring — add an edge, reroute, add a node — never a node body. So write-back edits the GraphIR and calls reconstruct:

from tensorsketch.canvas import extract, reconstruct
from tensorsketch.canvas.ir import EdgeIR

ir = extract(source)
ir.edges.append(EdgeIR(source="Billing", target="Review", kind="sequential"))  # a canvas edit
new_source = reconstruct(source, ir)

reconstruct regenerates the graph definition and drops any now-redundant wiring statements it folded in. Everything else — node classes and their bodies, imports, comments, unrelated code — is preserved byte-for-byte, because nothing else is touched.

Write-back is style-preserving: it detects how the source authored its wiring and re-emits in that same style, so the file reads the way you wrote it after a canvas edit:

All three styles render from the same ordered wiring walk, so the edges come out in identical order whichever style is chosen — which is exactly what keeps the round-trip a list equality.

Creating a node

Adding an edge is a pure-wiring change, but creating a node isn't — the new node needs a class X(Node) to exist. So when the IR names a node the source never defined (the canvas palette just made one), reconstruct synthesizes an idiomatic stub and inserts it above the graph builder:

ir = extract(source)
ir.nodes.append(NodeIR(name="Escalate",
                       inputs=[Port("query", "str")],
                       outputs=[Port("ticket", "str")],
                       has_hole=True))
ir.added.append("Escalate")
new_source = reconstruct(source, ir)

yields, written straight into your file:

class Escalate(Node):
    class In(Schema):
        query: str

    class Out(Schema):
        ticket: str

    async def run(self, ctx: Context, inp: In) -> Out:
        raise Hole("Escalate needs code")

The stub is born a hole — its typed interface is declared, its body is left for you to fill in code. A from tensorsketch import Hole is added if it isn't already imported. Crucially, the stub re-extracts to the exact NodeIR the canvas sent (has_hole=True and all), so the round-trip invariant still holds after a node is born on the canvas. Existing node classes are never touched — only genuinely new names are generated.

The safety property is the round-trip invariant, enforced in CI:

extract(reconstruct(extract(code))) == extract(code)

Re-extracting reconstructed code yields the identical graph. Within a style, reconstruct still tidies wiring without changing what the graph is — .entry(x) becomes .edge(START, "x"), conditional mappings are normalized, and a fluent chain is cleanly re-indented. (Trade-off: a comment sitting on a folded wiring statement moves with it; comments on nodes, imports, and unrelated code are untouched.)

Install

The engine is authoring-time tooling, so it's an optional extra (it isn't pulled into the runtime):

pip install tensorsketch-core[canvas]

The Studio

The Studio is the visual canvas on top of this engine. A stdlib bridge (tensorsketch.canvas.server, python -m tensorsketch.canvas <file>) serves extract(file) to a hand-drawn, Excalidraw-aesthetic frontend and applies reconstruct on every edit — so wiring on the canvas writes straight into your code, bodies untouched.

Layout lives in a sidecar, not the code

Where you place a node is presentation, not part of the graph — so it must never touch the code (the code is the source of truth about the graph, not its picture). When you drag a node in the Studio, its position is saved to a sidecar ‹file›.py.layout.json next to the source. Positions are optional and forgiving: a node with no saved position falls back to the automatic layered layout, a stale entry for a deleted node is ignored, and losing the sidecar loses only the arrangement — never the graph. This keeps the code clean while letting you arrange the canvas by hand.

See the architecture plan for the full design.

TensorSketch — a code-first, visually-editable, durable agentic framework

Design plan, v1. Codename TensorSketch (placeholder). Synthesized from 9 deep research reports (LangGraph, OpenAI Agents SDK, Claude Agent SDK+MCP, Agno/CrewAI/AutoGen, wider field, visual builders; + code↔visual bidirectional sync, execution runtime, DX/ extensibility). 2026-07-08.

0. Thesis & positioning

One-line: An agentic framework where code is the single ground truth, a visual canvas is a losslessly-synced projection of that code, execution runs on a durable BSP runtime, and every capability is a plugin — so it's easy to start, impossible to outgrow, fast, and absorbs whatever agent research comes next.

The gap we exploit. Each incumbent fails on a different axis, and no one has all of this together:

TensorSketch's bet: combine (a) code-as-truth + synced canvas, (b) a durable BSP runtime, and (c) an all-plugins core — the three things no single framework has together.

1. The five architectural commitments (everything follows from these)

1. Code is the single ground truth; the canvas is a projection. Only the graph wiring + typed interfaces round-trip; node bodies are opaque. (This is a computability necessity — Rice's theorem — not a choice; see §4.)
2. One Schema abstraction does four jobs: tool I/O, structured output, typed state channels, and design-time typed-port validation. (Pydantic v2 core / Standard Schema.)
3. BSP/Pregel scheduler on an actor/message substrate, persisted by a durable journal — cycles, deterministic parallel fan-out, per-superstep checkpoints, and single-process→distributed with unchanged code.
4. **The core knows interfaces, never implementations. Every node type, pattern, tool, provider, memory backend, channel, reducer, optimizer, and protocol is a plugin discovered via entry points. Two customization tiers: middleware (per-agent onion) + plugins** (global, with error hooks).
5. Durability contract collapses to one rule: wrap side effects in a durable step; the framework journals the result and never re-runs it on resume. No determinism straitjacket (journal-results-as-data, à la Restate).

2. The layered stack

┌───────────────────────────────────────────────────────────────────────────┐
│  L7  SURFACES:  Visual Studio (canvas)  ·  Layered API  ·  CLI  ·  Servers  │
│  L6  OBSERVABILITY & EVAL:  OTel GenAI spans · eval harness · optimizer      │
│  L5  INTEROP ADAPTERS:  MCP · A2A · AG-UI · OpenAI-compat  (plugins)         │
│  L4  EXTENSIBILITY:  entry-point registries · middleware · plugins · providers│
│  L3  CODE⇄CANVAS ENGINE:  CST extract · surgical write-back · holes · NL→code │
│  L2  AUTHORING MODEL:  typed Node classes + declarative wiring block + IR     │
│  L1  TYPE SYSTEM:  Schema (ports/state/tools/output) + reducers + validation  │
│  L0  RUNTIME CORE:  BSP scheduler · durable journal · message bus · streaming │
└───────────────────────────────────────────────────────────────────────────┘

Bodies (agent logic) run in the host language (Python/TS); L0 drives scheduling and calls back to execute a node, journals the result, advances.

3. The authoring model (L2) — what a developer actually writes

Typed node classes + a declarative wiring block in real, plain-text host code (no separate DSL — keep the truth in the language devs already use). Blends Pydantic-Graph (types define contracts), Vellum (declarative graph), and Dagster (import-free extractable).

class Classify(Node):
    class In(Schema):  query: str
    class Out(Schema): intent: Literal["billing", "tech", "other"]
    async def run(self, ctx, inp: In) -> Out:        # BODY: opaque, arbitrary code
        ...                                          # LLM calls, tools, parsing — anything

class BillingAgent(Node):
    class In(Schema):  query: str
    class Out(Schema): answer: str
    async def run(self, ctx, inp): ...

class Support(Graph):
    # WIRING: declarative, statically inspectable, round-trips losslessly.
    graph = (
        Classify
        >> Router.on(intent="billing") >> BillingAgent
        >> Router.on(intent="tech")    >> TechAgent
        >> Router.otherwise()          >> Fallback
    )

Rules that make it work:

• Typed ports are mandatory (In/Out schemas) → the canvas draws them, the compiler checks edges, NL→code has a contract.
• Wiring lives in a dedicated declarative block, never in native if/for/while. (The Prefect-1→2 lesson: the moment control flow drives the graph, the static graph is gone.)
• Bodies may contain anything; the canvas renders each node as a box with its ports and never introspects the body.
• Escape hatch: an imperative @durable async def that awaits step(...) compiles to the same journal, for devs who don't want to think in graphs. Both surfaces, one runtime.

4. The code⇄canvas engine (L3) — the crux

Why only wiring round-trips: "what are this node's out-edges" is a semantic property of a Turing-complete program → undecidable (Rice). So we keep wiring a syntactic, declarative surface and treat bodies as opaque. This is exactly why Dagster/Pydantic-Graph/Vellum/ LangGraph-StateGraph round-trip and Prefect-2/LangGraph-Functional don't.

Mechanism:

• Extract (code→graph): parse with a CST (libcst/tree-sitter), import-free — so it works on incomplete/broken code (essential for holes). Read node classes, their In/Out, and the graph wiring block.
• Project (graph→canvas): lay out; store layout (x/y, color, collapsed) in a sidecar (*.canvas.json), never fused into the truth (n8n's mistake).
• Reconstruct (canvas→code): a canvas edit is a structured mutation ("add edge A→B") applied as a surgical CST patch of the wiring block + class headers only — bodies, comments, imports provably untouched. Enforced by a CI invariant: extract(reconstruct(extract(code))) == extract(code), and reconstruct is a byte no-op when wiring didn't change.
• Degrade loudly: wiring the parser can't model → shown as one opaque "custom subgraph" node, code preserved verbatim. Never silently drop.

Incomplete → prompt for code = a typed hole:

class BillingAgent(Node):
    class In(Schema):  query: str
    class Out(Schema): answer: str
    async def run(self, ctx, inp: In) -> Out:
        raise Hole("Answer billing questions using the KB tool")   # greppable, type-checked

The interface is fully declared and round-trips; the body is a stub. The system surfaces "3 nodes need code" by grepping Hole(...).

NL→code, made reliable by the typed contract: the In/Out schema + docstring is the generation spec → generate body → compile + type-check against the ports + pass auto-generated contract tests → only then replace the Hole with real code. NL is an input method, never a stored representation. (BAML/DSPy "typed target + validate/repair" model.)

Dynamic behavior is never faked: autonomous loops and dynamic fan-out get an authored envelope + a runtime-trace overlay (Temporal Event-History model). Two visual layers: the static editable graph (what's possible) and a read-only execution trace (what happened). Never fabricate edges you can't derive statically.

5. The node / primitive vocabulary (L2)

The "periodic table" of primitives, each a plugin implementing the Node contract:

• Compute: agent (encapsulated autonomous loop: model + tools + memory + stop/budget), llm (single call), tool (fn/hosted/MCP), code (typed body).
• Control flow: sequence, router/switch (typed conditional), map (fan-out over a collection → reduce), loop (iterate-until + mandatory guard), parallel (+ join).
• State: typed channels with reducers (last/append/topic/custom); memory (short/long/typed); retriever.
• Structure: subgraph (nestable, promoted params, hidden internals), reroute, comment.
• Coordination: handoff/sub-agent, supervisor/orchestrator, team container.
• Reliability: human (HITL/interrupt), guardrail/validator, evaluator.

Edges: sequential · conditional · handoff · soft/dynamic (LLM-decided routing, drawn distinctly) · data-dependency. Ports are typed; connection legality is checked at edit time.

6. The runtime (L0)

• Scheduler: BSP/Pregel supersteps (plan → execute-in-parallel → apply-reducers-at- barrier). Cycles, deterministic parallel fan-out, and a natural checkpoint boundary fall out of the model. Fan-out via a Send-style dynamic-branch primitive, but with per-branch journaling/commit (so one straggler/failure doesn't discard siblings) and pass-IDs-not-payloads (avoids O(branches×state) blowup) — fixing LangGraph's sharp edges.
• Actors underneath: nodes communicate only via an abstract message bus → transport is swappable (in-proc → gRPC), so the same graph runs single-process or distributed unchanged (AutoGen-Core's trick). Partition by thread_id/agent key for single-writer state and horizontal scale (Restate Virtual Objects / Dapr virtual actors).
• Durability: a Restate/DBOS-style journal, not just state checkpoints.
• Two tiers: per-superstep StateSnapshot (resume/time-travel/fork, latest fetched O(1)) + per-effect journal entries (LLM/tool/message steps memoized → not re-run on resume → no duplicate side effects, no reasoning drift).
• Journal-results-as-data → no determinism straitjacket on orchestration code.
• Postgres transaction-piggyback (DBOS) for exactly-once DB-backed steps.
• Pluggable backends: in-memory (dev) → SQLite (local) → Postgres (prod); Temporal/ Restate/Inngest only as optional backends, never required (embeddability first).
• Streaming: everything is a namespaced event (run_id, thread_id, node_path, agent_id) → coherent multi-agent lanes from one stream; backpressure + structured concurrency (TaskGroup/AnyIO) for clean parallel-tool cancellation; resumable streaming (replay from a cursor).
• Fast by construction: parallel/DAG tool calls by default (1.8–3.7× wall-clock); prompt-cache-stable prefixes (byte-stable tool ordering/serialization, auto cache_control); delta + latest-only checkpointing (constant per step regardless of conversation length); async persistence off the critical path; sticky hot-state cache.
• Language strategy: pure-Python reference runtime first (fastest path to a real product; validate semantics), with a clean SDK↔core boundary from day one, then swap the hot path to a Rust core (scheduler/journal/bus/streaming) behind the same interface — escaping LangGraph's two-codebases drift. Python-primary + TypeScript-parity SDKs; cross- language interop over MCP/A2A/AG-UI.
• The one durability rule for authors: wrap side effects in step(name, fn, idempotency_key=...) (LLM/tool calls auto-wrapped). Mark @pure vs @effect so "replay re-ran my LLM call" is never a surprise. Ship a crash-harness (kill at each replay boundary) — durability that isn't crash-tested is theater.

7. Type system (L1)

• One Schema protocol over Pydantic-v2 (Rust core, 5–50× faster; validates bare values → full models) / Standard Schema (TS). Drives: tool I/O, structured output, typed state channels+reducers, typed ports.
• Tools: auto-schema from signature (inspect + docstring via griffe) — zero boilerplate.
• Structured output: strict constrained-decoding when the provider supports it (hard guarantee), validate-and-repair (ModelRetry/reask, minimal repair context) otherwise.
• Design-time port validation (ComfyUI insight, via types not Any): edge legal iff src type assignable to dst (or a registered coercion); illegal connections rejected at edit time with teaching errors ("expected str, got list[Doc]; did you mean .candidates?"). Parametric generics (Node[TIn,TOut]), never bare wildcards.

8. Extensibility (L4) — "scales with new research"

• Discovery via entry points: third-party packages self-register into typed registries (tensorsketch.providers, tensorsketch.nodes, tensorsketch.tools, tensorsketch.memory, tensorsketch.channels, tensorsketch.reducers, tensorsketch.optimizers). pip install tensorsketch-core-voice = new modality, no core release.
• Two customization tiers:
• Middleware (per-agent, onion order): before/after_model, before/after_tool, before/after_agent, wrap_model_call, wrap_tool_call; can short-circuit / rewrite requests. (LangChain 1.x model.)
• Plugins (app-global, precedence over middleware): logging, OTel export, policy, quotas, caching, on_model_error/on_tool_error (adaptive recovery — fixes CrewAI's retry-loop). (Google ADK model.)
• Provider abstraction: ChatProvider interface with a capabilities probe (strict JSON, tools, vision, audio, cache); core depends on no provider SDK — each is an optional package (kills the LangChain bloat complaint).
• Why it future-proofs: the core exports a fixed small set of contracts (Node, Schema, ChatProvider, MemoryStore, Channel, Middleware, Plugin, Reducer, Optimizer). Any new capability = implement one contract + register. A 2027 pattern (new node + reducer + escalation middleware + streaming modality + memory + optimizer + a new wire protocol) ships as ~7 zero-core-change plugins.

9. Interop (L5) & observability/eval (L6)

• Protocol-native, bidirectional adapter plugins: MCP (tools/resources/prompts — consume any server, expose any agent; stateless-by-default per the 2026 direction), A2A (every agent auto-publishes an Agent Card; remote agents are typed nodes), AG-UI (native streaming event format → live UI/canvas for free), OpenAI-compat (consume + expose /v1/chat/completions). Protocol churn absorbed at the adapter layer.
• OTel GenAI semconv emitted natively (spans invoke_agent/execute_tool/chat; token/duration metrics; MCP trace propagation; privacy modes) → LangSmith/Langfuse/Phoenix work with zero custom instrumentation, via a built-in TracingPlugin.
• Eval harness (datasets, LLM-judge/custom evaluators, scenario tests, CI gating; every run is a replayable trace → "prod trace → eval case" free) and a DSPy-style optimizer registry (MIPROv2) as a distinct capability — agents that improve with data.

10. The layered API (L7) — easy to start, impossible to outgrow

• L7a Prebuilt agents (default door): create_agent(model, tools, memory, middleware, output=Schema) — is a graph factory that decomposes to the same primitives (no cliff).
• L7b Graph builder: Graph(State); g.add(...); g.connect(port, port); g.validate(); g.compile(checkpointer=Postgres(), middleware=[...]).
• L7c Primitives: Node, Port/Schema, Edge, State+Reducer, ChatProvider, Middleware, Plugin, Channel, MemoryStore.
• L7d Surfaces: serve_openai_compat, serve_a2a, serve_mcp, stream_agui, and the Visual Studio (canvas over the code⇄canvas engine + live trace overlay + debug: step, replay-a-node, edit-state, time-travel).
• DX principles: few primitives / progressive disclosure; prebuilt decomposes to primitives; slim core + opt-in integrations; errors that teach; SemVer'd core contracts (churn in plugins, not interfaces); code-first + testable (in-memory runner, fake providers, scenario tests).

11. How TensorSketch beats each incumbent (scorecard)

12. Phased build roadmap

Phase 0 — Runtime & type spine (the foundation).

• Pure-Python BSP scheduler; typed channels + reducers; Schema abstraction; typed ports + design-time validation; in-memory + SQLite journal (two-tier: snapshot + effect journal); durable step; namespaced event streaming. Acceptance: a graph runs, persists, resumes from any checkpoint, and re-runs no journaled effect on resume.

Phase 1 — Authoring model & code⇄canvas engine (the differentiator).

• Typed Node classes + declarative wiring block; CST extract (import-free); surgical write-back with the CI round-trip invariant; layout sidecar; typed holes + hole surfacing. Acceptance: edit on canvas → code changes with bodies untouched; edit code → canvas updates; incomplete graph runs up to its holes.

Phase 2 — Agent primitives & prebuilt API.

• agent (autonomous-loop node), llm, tool (fn + auto-schema), router/map/loop/ parallel, memory, subgraph; create_agent; structured output (strict + repair); provider abstraction (OpenAI/Anthropic/LiteLLM as optional packages). Acceptance: build a real multi-tool agent both in code and on canvas; run it durably.

Phase 3 — Extensibility, interop, observability.

• Entry-point registries; middleware + plugins (incl. error hooks); OTel GenAI tracing plugin; MCP client+server; A2A Agent Cards; AG-UI streaming; OpenAI-compat serve. Eval harness. Acceptance: add a node type + a provider + an MCP server as external packages, zero core edits; traces land in Langfuse unmodified.

Phase 4 — NL→code, optimizer, distribution, Studio polish.

• NL→code (typed contract + validate/repair + generated tests); DSPy-style optimizer registry; distributed runtime (gRPC host/worker, key partitioning) with unchanged agent code; Rust hot-path core behind the same interface; Studio debug (step/replay/time-travel) + trace overlay. TypeScript SDK parity.

Cross-cutting from day one: the SDK↔core boundary, the @pure/@effect marker, the crash-harness, SemVer'd contracts, and the round-trip CI invariant.

13. Open decisions / risks

• Rust-core timing. Pure-Python first is right, but the FFI boundary (per-node/token marshaling) must be designed early or the later swap leaks. Mitigation: opaque state handles in the core, marshal only deltas, batch token events.
• Wiring-DSL ergonomics. The >>/Router.on surface must stay both human-writable and cleanly CST-patchable. Needs real dogfooding; may need a small set of canonical forms the reconstructor emits.
• How much control flow lives in wiring vs bodies. Too little in wiring → canvas is anemic; too much → we recreate Prefect-2's dynamic-graph loss. The map/loop/router primitives are the negotiated line; validate with real agents.
• NL→code trust. Only as good as the generated tests; needs a strong contract-test generator and a visible "unverified" state until it passes.
• Naming / scope. "TensorSketch" is a placeholder; and whether this ships as its own product vs. the engine under Agent Arena is a strategic call (Agent Arena becomes the first application of TensorSketch — a constrained, fully-visual deployment where user code is sandboxed away).
• Verify before shipping: Anthropic prompt-cache pricing/TTL numbers (from secondary sources) against primary docs.

Roadmap

The full architecture is in the architecture plan. This page tracks the phased build. For a running ledger of exactly what's built and what's deferred (and why), see Build status & backlog.

Phase 0 — Runtime & type spine (complete)

The foundation everything else stands on.

• [x] Schema abstraction (over Pydantic v2)
• [x] Typed state channels with reducers — LastValue, BinaryOperatorAggregate, Topic
• [x] BSP superstep engine — parallel execution, barrier reduction, cycles, recursion limit
• [x] Typed Node (In/Out ports) and Graph builder (sequential + conditional edges)
• [x] Design-time validation — port existence, port/channel type compatibility, edge integrity
• [x] Typed holes (Hole) for "this node needs code"
• [x] Generic typing end-to-end — invoke returns your concrete state type
• [x] Durable execution — per-barrier checkpoints, resume/fork, and a per-effect journal (ctx.step) so side effects run exactly once; in-memory + SQLite backends; crash-harness test
• [x] Namespaced event streaming — live stream() (node/values/custom events, monotonic cursor, backpressure), ctx.emit, and resumable replay from a cursor
• [x] Test suite (44 tests), runnable examples, and these docs
• [x] Hardening — edge-case coverage, micro-benchmarks, ruff format, and GitHub Actions CI (lint · format · strict types · tests on Python 3.11 + 3.12)

Phase 1 — Authoring model & code⇄canvas engine (complete — the differentiator)

• [x] CST extraction (tensorsketch.canvas.extract, import-free) — nodes + typed ports + hole detection + the Graph(...) builder wiring → a JSON-able GraphIR
• [x] Surgical write-back (tensorsketch.canvas.reconstruct): rebuild only the builder chain from the (edited) IR, bodies/imports/comments byte-preserved; the round-trip CI invariant extract(reconstruct(extract)) == extract (parametrized test gate)
• [x] Ergonomic >> wiring surface (START >> a >> Router(fn, ...)) — handles from Graph.nodes(...), pure sugar over .add/.edge/.conditional
• [x] Statement-style builder support (g = Graph(...); g.add(...), incl. annotated g: Graph[S] = ...); add(name=...) renames; clean generated formatting at any nesting
• [x] The visual canvas (TensorSketch Studio) — Excalidraw aesthetic (see decisions): a stdlib bridge (python -m tensorsketch.canvas <file>) + hand-drawn frontend that renders the GraphIR and writes edits (add/remove edges) back through reconstruct
• [x] Node creation from a palette — Studio's + node dialog (name + optional ports) generates a class X(Node) stub (typed ports + Hole body) via reconstruct; re-extracts to the exact NodeIR, so the round-trip invariant holds
• [x] Project-wide hole surfacing — find_holes(paths) / python -m tensorsketch.canvas --holes; Studio counts holes across the project and lists them (file · node · Hole message)
• [x] Layout sidecar — drag to arrange; positions persist in ‹file›.py.layout.json beside the code (never in it), with automatic layout as the fallback
• [x] Style-preserving write-back — detect the source's style (fluent / statement / >>) and re-emit in it; all three render from one ordered wiring walk, so edge order (and the round-trip) is preserved by construction

Phase 2 — Agent primitives & prebuilt API (complete)

• [x] tool with schema auto-derived from the function signature (sync + async)
• [x] Llm single-call node; Agent autonomous loop node (durable — every model/tool call wrapped in ctx.step)
• [x] create_agent(...) prebuilt returning a normal graph
• [x] Provider abstraction (ChatProvider, zero SDK deps) with FakeProvider and an optional AnthropicProvider
• [x] Structured output via generate_structured / provider output_schema
• [x] Composition patterns: gather_map (durable concurrent map), parallel, run_subgraph
• [x] Validate-and-repair loop for structured output
• [x] Providers: Anthropic, OpenAI, Google + a dead-simple custom-provider path
• [x] Bring-your-own-database connectors — PostgresBackend, RedisBackend behind the Backend ABC (lazy drivers), a shared SqlBackend base, and a pluggable Serializer seam
• [x] Graph-level dynamic fan-out (Send) — a router returns Send(node, payload)s and the engine spawns one superstep task per Send (its own payload), merging at the barrier via a reducer channel. Durable: each instance journals under a distinct key; pending sends ride the checkpoint. Plus loop/router builder sugar over conditional.
• [x] Multi-agent coordination — as_tool(graph) wraps an agent as a Tool so a supervisor delegates to specialists (agents-as-tools / handoff). Reuses the agent loop, so delegations are journaled and trace as one team; built on general ctx-injection into tool functions.

Removed: an early in-framework memory subsystem (keyword search). Memory/state belongs outside the framework — see decisions.

Phase 3 — Extensibility, interop, observability

• [x] MCP client + server (tensorsketch.interop.mcp) — consume external tool servers as TensorSketch tools; expose TensorSketch tools to any MCP client. Optional tensorsketch-core[mcp].
• [x] Middleware (tensorsketch.middleware) — wrap-style interceptors around every model/tool call (RetryMiddleware = on_model_error/on_tool_error, ObservabilityMiddleware); durable (journaled inside ctx.step)
• [x] Native tracing (tensorsketch.observability) — vendor-neutral Tracer + built-in InMemoryTracer; run/node/model/tool spans with timing, tokens, cost, errors; ctx.span; overridable cost model. Completion.model makes model/cost first-class.
• [x] Exporters — FileTracer (JSON-lines, dependency-free) and an optional OTelTracer (tensorsketch-core[otel]) bridging TensorSketch spans to OpenTelemetry — adapters over the Tracer, both driven by a reusable RecordingTracer base.
• [x] Registry (tensorsketch.registry) — select a built-in provider/backend by name (create_provider("anthropic", …) / create_backend("postgres", …)) so config can choose it; register_* and tensorsketch.providers/tensorsketch.backends entry points add names. Lazy (no SDK imported to list or resolve). Named factories, not a plugin ecosystem — see decisions D5.
• [x] Serving (tensorsketch.serve, optional tensorsketch-core[serve]) — expose an agent as a mountable ASGI app over OpenAI-compatible (openai_app), A2A (a2a_app + a2a_tool to consume), and AG-UI (agui_app) protocols; Starlette-based, imported only in tensorsketch.serve.
• [x] Eval harness (tensorsketch.eval) — Case/Trial/Grader/Report; code-based graders (answer, tool trajectory + payload, step efficiency, cost/latency, final-state) and LlmJudge; multi-trial pass@k / pass^k + completion rate; Sandbox seam (in-process default); report.require(...) CI gate. Consumes the trace for cost/latency/trajectory.
• [x] Eval results emit + online — to_dict() everywhere + a Reporter sink (JsonlReporter, CallbackReporter, custom emit to any DB); evaluate(reporter=…). Online: score(...) / OnlineMonitor grade live production traces reference-free and emit to a sink. TensorSketch stays stateless (emits, never owns the store).
• [x] Drift detection (tensorsketch.eval.drift) — a DriftMonitor (itself a Reporter) over the online result stream: two-proportion z-test on pass-rate / per-grader drops + a Page-Hinkley change-point on cost/latency, against a Baseline (from_report). Emits DriftAlerts to a sink; the window is in-memory only (nothing persisted). MultiReporter fans each result to a store and the monitor. Stdlib-only stats — see decisions D7b.
• [x] Multi-sink trace fan-out (MultiTracer) — one span lifecycle (one trace_id, correct nesting) fanned to several sinks at once: any mix of RecordingTracers (FileTracer, InMemoryTracer) and Callable[[Span], None] viewers. The callable sink is the seam the live overlay plugs into.
• [x] Live trace overlay in Studio — click ▶ live and a run lights up on the canvas: each node ringed by status (ok/error) with a latency · cost · calls badge. A read-only projection of the run's spans, fed by MultiTracer(..., http_span_sink(url)) from the agent's own process; the bridge buffers spans in memory only. Statelessness holds — Studio reads code + telemetry, owns neither.

Phase 4 — NL→code, optimizer, distribution, Studio

• Natural-language → code (typed contract + validate/repair + generated tests)
• DSPy-style optimizer registry
• Distributed runtime (gRPC host/worker) with unchanged agent code; Rust hot-path core
• Visual Studio debugging (step/replay/time-travel) + live trace overlay; TypeScript SDK parity

Cross-cutting from day one: a clean SDK↔core boundary, @pure/@effect markers, a crash-harness for durability, SemVer'd core contracts, and the round-trip CI invariant.

Build status & backlog

The single source of truth for what's built and what's deliberately deferred (with the reason). Updated as work lands, so nothing gets dropped. Pairs with the roadmap (phase plan) and the architecture plan (the full design).

Last updated: Phase 1 complete — style-preserving write-back (fluent / statement / >> all re-emit in their own style, edge order preserved via one shared wiring walk). Earlier: renamed to TensorSketch (import tensorsketch, dist tensorsketch-core 0.1.0) + packaging foundation (LICENSE, classifiers, URLs, wheel verified); multi-agent coordination (as_tool); the Studio's node-creation palette, project-wide hole surfacing, and layout sidecar. (__version__ now 0.1.0.)

Naming & packaging ✅

• Name: the library is tensorsketch-core on PyPI, imported as tensorsketch; the product/platform is TensorSketch. (Both loom and tensorsketch are taken on PyPI; loom was always a placeholder.) Base error is TensorSketchError; trace attrs are tensorsketch.*.
• Packaging: version 0.1.0, Apache-2.0 LICENSE, trove classifiers, project URLs (placeholder until the repo is public), py.typed. uv build produces a wheel that ships the Studio assets. Author/email and real URLs are marked TODO(publish) in pyproject.toml.

Done

Phase 0 — runtime & type spine ✅ (complete + hardened)

• Schema abstraction (Pydantic v2).
• Typed state channels + reducers: LastValue, BinaryOperatorAggregate, Topic.
• Typed Node (In/Out ports) and generic Graph builder (sequential + conditional edges, START/END); compile-time validation (ports, types, edges, single-successor-source, duplicate names); typed holes (Hole).
• BSP superstep engine: parallel fan-out on a shared snapshot, reducer barrier, native cycles, recursion-limit guard.
• Durable execution: per-barrier checkpoints, resume, fork, get_state/get_history; the per-effect journal (ctx.step) → exactly-once side effects; InMemoryBackend + SqliteBackend.
• Streaming: live stream() (run/node/values/custom events, monotonic seq, backpressure, structured-concurrency cancel), ctx.emit, resumable replay.
• Hardening: edge-case tests, benchmarks, ruff format, GitHub Actions CI.

Phase 2 — agent primitives (in progress) 🚧

• Tools: @tool with schema auto-derived from the signature (sync + async); arg validation.
• Messages: Message/ToolCall, constructors, add_messages reducer.
• Providers: ChatProvider (zero SDK deps), Completion/Usage, FakeProvider, and three optional real providers — Anthropic, OpenAI, Google (lazy imports; OpenAI also covers OpenAI-compatible endpoints via base_url); documented custom-provider path.
• Agents: Llm single-call node; Agent durable loop (every model/tool call via ctx.step); create_agent.
• Structured output: generate_structured + provider output_schema, with validate-and-repair.
• Composition patterns: gather_map (durable concurrent map), parallel, run_subgraph.
• Bring-your-own-database backends (D1 delivered): PostgresBackend (tensorsketch-core[postgres]) and RedisBackend (tensorsketch-core[redis]) behind the Backend ABC — lazy drivers, a shared SqlBackend base (any DB-API store), and a pluggable Serializer codec (default PickleSerializer). The whole durability suite runs against Redis via fakeredis; Postgres via LOOM_TEST_POSTGRES DSN.
• Graph-level dynamic fan-out (Send, D8): a router's path returns Send(node, payload)s; the engine spawns one superstep task per Send (its payload overlaid on the shared snapshot, filtered to the node's In), and they merge at the barrier via a reducer/Topic channel. Fan-out is durable — each instance's ctx.step effects journal under a distinct key (a new instance tag on Context, empty for normal nodes so existing keys are byte-identical), and pending sends ride the checkpoint (a new Checkpoint.sends, whole-checkpoint pickled so all backends carry it). Plus loop/router builder sugar over conditional. Verified: map/reduce correctness, distinct per-instance journal keys, crash-mid-fan-out resume exactly-once across memory/SQLite/ Redis. Phase 2 is now complete (its roadmap boxes all done). See decisions D8.

Phase 1 — code⇄canvas engine ✅ (complete)

• CST extraction (tensorsketch.canvas.extract, optional canvas extra): parses TensorSketch source with libcst — import-free, works on incomplete/hole code — into a JSON-able GraphIR (nodes + typed ports + has_hole + wiring from the Graph(...) builder, conditional mappings expanded).
• Write-back (tensorsketch.canvas.reconstruct): regenerates only the graph definition from the (edited) IR, byte-preserving bodies/imports/comments; the round-trip invariant extract(reconstruct(extract)) == extract is a parametrized test gate. Also generates node stubs — a new node the IR names gets a synthesized class X(Node) (typed ports + Hole body).
• >> wiring surface (tensorsketch.core.wiring: NodeHandle/Router, Graph.nodes()/graph[name]): START >> a >> Router(fn, ...), fan-out via a >> [b, c] — pure sugar over the builder.
• Style-preserving write-back: fluent chain, statement-style (incl. annotated g: Graph[S] = ...), and >> all extract to the same IR; write-back detects which style the source used and re-emits in that same style (a chain stays a chain, statements stay statements, >> stays >>), rather than canonicalizing to the fluent chain. All three render from one ordered wiring walk (_wiring_items), so edge order — and thus the list-equality round-trip — is preserved by construction; linear >> runs merge into one a >> b >> c spine. Clean indentation at any nesting depth; add(name=...) renames handled.
• Studio — the visual canvas (tensorsketch.canvas.server + tensorsketch/canvas/studio/, run via python -m tensorsketch.canvas <file>): a stdlib localhost bridge serving extract, a hand-drawn Excalidraw-aesthetic frontend (layered layout, typed ports, hole/conditional rendering), and add/remove-edge edits written straight back through reconstruct. IR gained from_dict. A + node palette creates nodes on-canvas (name + optional ports → a generated stub in code); drag a node to move it (positions persist in a .layout.json sidecar, never in the code); a toolbar badge surfaces project-wide holes (click to list every node still needing code).

Quality bar (holds for everything above)

ruff + ruff format clean · mypy --strict clean · full test suite green · runnable examples · CI on 3.11 + 3.12. Run everything with make check.

Deferred / saved for later

Each item is real and intended — just not built yet. Grouped by area, newest-relevant first.

Phase 2 remainder (agents)

• Memory (re-approach): not in-framework; an external, embedding-based store the app owns. The keyword-search version was removed. See decisions.
• More builder sugar: map / parallel as Graph constructs (vs today's body-helpers). loop/router and graph-level Send fan-out now ship; these two remain body-helpers.
• Structured agent output (typed state carrying a parsed Schema). Deferred to avoid dynamic In/Out on nodes; generate_structured covers the standalone case.
• More providers: OpenAI / OpenAI-compatible, LiteLLM. Straightforward on the abstraction.
• Tools: hosted/MCP tools; per-parameter descriptions parsed from the docstring.
• ~~Coordination: sub-agent handoff, supervisor/orchestrator.~~ Done — as_tool(graph) wraps an agent as a Tool, so a supervisor calls specialists as tools (agents-as-tools). Built on a general seam: a tool declaring a ctx param gets the Context injected. Delegations are ordinary tool calls, so they're journaled (specialist replayed on resume) and nest in one trace. A dedicated team/orchestrator container is still possible but unnecessary — the pattern is just create_agent(tools=[as_tool(...), ...]).
• Subgraph as compile-time inlining (namespaced nodes, uniform BSP + checkpointing) vs the current run_subgraph wrapper.
• Agent decomposition to primitives (agent loop as a visible subgraph) vs today's single node.

Runtime & durability

• Transaction-piggybacked exactly-once for DB steps (commit the effect result in the same transaction as the work); optional Temporal/Restate backends. Postgres/Redis connectors done.
• Distributed runtime (gRPC host/worker, key partitioning) with unchanged agent code.
• Rust hot-path core behind the same interface.
• Perf: prompt-cache-stable prefixes; delta + latest-only checkpointing; async persistence off the critical path.
• max_steps is currently an absolute superstep budget across resumes — revisit if per-invoke budgeting is wanted.

Type system

• Richer port typing including reducer/Topic update types (today the assignability check is skipped for reducer/Topic fields); parametric generics on nodes; registered coercions.

Streaming

• Live-tailing an in-progress run from another process (merge replay catch-up with live stream).
• Token-level streaming from provider nodes (flows through ctx.emit).

Phase 1 — code⇄canvas engine (complete — the headline differentiator)

• Done: CST extraction (fluent + statement-style + >>), the >> authoring surface, surgical style-preserving write-back with clean formatting, the round-trip invariant, and the Studio (bridge + hand-drawn frontend) closing the loop end to end.
• New authoring forms extract + round-trip: router(...) (== conditional), and loop(node, until, *, exit=END) (a two-branch conditional: a self-loop the canvas draws as an arc, plus the exit edge). Dynamic routing inside an opaque callable (Send fan-out, lambda predicates) shows as a dynamic-route stub — the graph runs; the canvas is honest that the target is computed at runtime. Mapping targets render END/START as barewords.
• Node creation from a palette: reconstruct now synthesizes a class X(Node) stub (typed In/Out ports + a Hole body) for any node the IR names but the source never defined, inserts it above the graph builder, and adds from tensorsketch import Hole when needed. The stub re-extracts to the exact NodeIR (has_hole=True), so the invariant holds. Studio wires it to a + node dialog (name + optional ports) — a created node lands unwired, ready to drag-connect.
• Project-wide hole surfacing: find_holes(*paths) -> [HoleRef] (file, node, Hole spec, line) walks a codebase syntactically (skips unreadable/unparseable files); CLI python -m tensorsketch.canvas --holes [paths]; Studio GET /api/holes + a clickable toolbar badge that lists every hole across the project.
• Layout sidecar: manual node positions persist in ‹file›.py.layout.json beside the source (never in the code) — POST /api/layout writes it, GET /api/graph serves it; drag a node's body to move it, unmoved nodes keep the automatic layered layout. Malformed/stale entries ignored.
• Style-preserving write-back (done — the last Phase 1 item): write-back detects the source's style (fluent / statement / >>) and re-emits in it instead of canonicalizing to a fluent chain. All three styles render from one ordered wiring walk (_wiring_items), so the emitted edge order is identical across styles — the round-trip stays a list equality by construction. Linear >> runs merge into one a >> b >> c spine; an arrow graph that gains its first conditional gets a from tensorsketch import Router import added. Phase 1 is now complete.

Phase 3 — extensibility, interop, observability (in progress) 🚧

• MCP interop done (tensorsketch.interop.mcp, optional tensorsketch-core[mcp]): mcp_tools(session) wraps a server's tools as TensorSketch tools; build_mcp_server/serve_stdio expose TensorSketch tools to any MCP client; stdio_session convenience transport. Round-trip tested over the real protocol via the SDK's in-memory transport. Needed a small Tool generalization (raw-JSON-schema tools).
• Middleware done (tensorsketch.middleware): wrap-style (onion) interceptors around every model and tool call — wrap_model/wrap_tool + compose_*, wired into the Agent loop inside ctx.step (so retries/tracing are journaled, never re-run on resume). Built-ins: RetryMiddleware (the on_model_error/on_tool_error primitive) and ObservabilityMiddleware (start/end/error + duration events into the stream). create_agent(middleware=[...]).
• Native tracing done (tensorsketch.observability): a vendor-neutral Tracer (no OTel dependency) + built-in InMemoryTracer. The engine opens run/node spans, agents open model/tool spans (model id, tokens, cost, status). Spans nest via a ContextVar (correct across await and parallel tasks). Trace aggregates duration/tokens/cost/errors + render()/summary(); ctx.span(...) for custom spans; overridable estimate_cost/DEFAULT_PRICES. invoke/stream take tracer=. Replayed (journaled) calls produce no span, so traces reflect real work. Completion.model is now first-class (providers set it from the response), so cost no longer reads a private attr.
• Exporters done: a reusable RecordingTracer base (owns lifecycle, _record(span) hook); FileTracer writes JSON-lines (dependency-free, Span.to_dict() + wall-clock started_at); optional OTelTracer (tensorsketch-core[otel], tensorsketch.observability.otel) bridges each TensorSketch span to an OTel span (nesting + attributes preserved), tested against a real in-memory OTel exporter.
• Registry done (tensorsketch.registry): select a built-in provider/backend by name so a config value can pick it — create_provider("anthropic", …) / create_backend("postgres", …), plus register_* and tensorsketch.providers/tensorsketch.backends entry points to add names. A generic lazy Registry[T] (built-ins are thunks, entry points .load() on demand) keeps import tensorsketch free of every SDK/driver — verified in a fresh interpreter that listing/creating imports none. Scoped to two seams on purpose; not a plugin ecosystem or a core/community package split (everything first-party stays in the one tensorsketch package). See decisions D5.
• Serving done (tensorsketch.serve, optional tensorsketch-core[serve]): expose an agent-shaped CompiledGraph as a mountable ASGI app over three standard protocols — OpenAI-compatible (openai_app: /v1/chat/completions stream + non-stream, /v1/models), A2A (a2a_app: agent card + JSON-RPC message/send/message/stream; plus a2a_tool(url) to consume a remote agent), and AG-UI (agui_app: RunAgentInput → SSE of RUN_STARTED/TEXT_MESSAGE_*/STATE_SNAPSHOT/ RUN_FINISHED). Built on Starlette + a shared ChatAdapter (to_input/to_reply) and SSE helper; Starlette/httpx imported only in tensorsketch.serve (verified import tensorsketch pulls in neither). Tested over the real ASGI apps (Starlette TestClient + httpx ASGI transport), including an A2A consume↔expose round-trip. Pragmatic subsets — token streaming, full A2A task store, and multi-turn/inbound-tools are deferred; see decisions D6.
• Eval harness done (tensorsketch.eval, in-package — no extra): the task/trial/transcript/outcome/ grader model over TensorSketch's own Trace. Case (inputs + graders + per-trial setup env + trial count) → evaluate → Report. Hybrid graders: code-based over the answer (Contains/ Equals/Regex), the trajectory + payload (ToolCalled/ToolArgs/ToolSequence — needed enriching the agent's tool span with args/result), StepEfficiency, CostBudget/LatencyBudget, FinalState (outcome/env), Custom/@grader; plus LlmJudge (binary, one-criterion, structured Verdict via the provider seam — its call isn't traced, so it doesn't hit the agent's cost). Multi-trial metrics: completion rate, pass@k / pass^k, mean cost/latency, per-grader breakdown; render()/summary(); Report.require(...) as a CI gate. Sandbox seam with an in-process default; graders always run in the harness, never inside the agent (anti-gaming). Fresh env per trial. See decisions D7 for the deferred list.
• Eval results emit + online done: results serialize (to_dict() on Report/CaseResult/ TrialResult/Grade) and emit through a Reporter sink — JsonlReporter, CallbackReporter, or a ~5-line custom emit() into any DB (sync/async); evaluate(reporter=…). Same exporter pattern as tracing, for scores. Online: score(output, trace, graders) grades a live run reference-free; OnlineMonitor(graders, reporter=, sample=) samples production runs off the response path and emits each result. Trial.case made optional. TensorSketch stays stateless — emits, never owns the results store. Tested (test_eval.py, 18 total).
• Drift detection done (tensorsketch.eval.drift): a DriftMonitor that watches the online result stream and emits DriftAlerts when behavior drifts from a Baseline — a two-proportion z-test on pass-rate / per-grader drops, and a Page-Hinkley change-point on cost/latency (fed baseline- relative so one threshold spans dollars and ms). It is a Reporter, so it chains after OnlineMonitor; MultiReporter fans each result to a store and the monitor. The rolling window lives in-process only — TensorSketch emits alerts, never owns a drift store. Sustained regressions are latched (fire once until recovery). Stdlib stats, no deps. See decisions D7b.
• Multi-sink trace fan-out done (MultiTracer): one span lifecycle (a single trace_id, correct nesting/timing) fanned to several sinks at once — any mix of RecordingTracers (FileTracer, InMemoryTracer) and plain Callable[[Span], None] viewers. A RecordingTracer sink shares the one trace_id and only its _record consumer half is driven; the callable form is the exact seam the Studio live overlay plugs into. Tested (test_exporters.py).
• Live trace overlay in Studio done: click ▶ live and a run paints onto the canvas — each node ringed by status (ok/error) with a latency · cost · calls badge, keyed off the tensorsketch.node span attribute (model/tool spans fold into their parent node). Fed by http_span_sink(url) (a new stdlib, non-blocking, drop-on-failure callable sink in observability.export) inside a MultiTracer from the agent's own process; the bridge gained an ephemeral in-memory TraceBuffer (POST/GET /api/trace) that persists nothing. Statelessness holds — Studio reads code + telemetry, owns neither (the "how does Studio work if stateless" answer, made concrete). Tested end-to-end (test_studio_live.py: buffer semantics + a real run's spans delivered through the sink).
• Phase 3 is now complete. Remaining observability polish (richer per-provider cost/latency) and the deferred eval items (distributional drift, stronger sandboxes, annotation queue) live below.

Evaluation (offline harness shipped — these extend it)

• Stronger sandboxes: SubprocessSandbox, DockerSandbox, and remote runners behind the existing Sandbox seam (in-process ships today). Needed for agents that write code / run bash.
• Distributional drift: DriftMonitor (pass-rate two-proportion + cost/latency Page-Hinkley) ships now; population-shift detectors (PSI / KL / KS over a reference distribution) and routing alerts to a pager/webhook are the next layer.
• Feedback loop as tooling: the annotation queue / human-in-the-loop UI and the trace→golden pipeline (today a failure becomes a Case you construct by hand).
• More trajectory metrics: memory hit rate (memory-enabled agents) and scope-adherence.
• Judge calibration: measuring an LlmJudge's rank correlation against human labels.
• Unbiased pass@k estimator for k < n samples (today pass@k uses k = trials run).

Phase 4 — NL→code, optimizer, distribution, Studio (not started)

• NL→code (typed contract + validate/repair + generated tests); DSPy-style optimizer registry; Visual Studio debug (step/replay/time-travel) + trace overlay; TypeScript SDK parity.

Verify before shipping

• Anthropic prompt-cache pricing/TTL numbers in the design doc (from secondary sources) against primary docs.
• Confirm the Anthropic, OpenAI, and Google provider request/response mappings against the live APIs (currently covered by unit tests with injected fake clients, not real calls). Also confirm the default model ids are current.

Decisions log

Short, dated records of design decisions — especially reversals — so the why isn't lost. Newest first. See also the build status & backlog.

D1 — TensorSketch is stateless; state and memory live outside the framework

Decision. The framework holds no state of its own. Everything persistent — run checkpoints, the effect journal, and (later) memory — goes through pluggable connectors the application points at its own database. TensorSketch ships small built-ins (InMemoryBackend, SqliteBackend) and a clean interface (Backend) so any popular or custom database drops in.

Why. A stateless core is what makes an agent system scalable and easy to operate: any number of workers can serve the same runs because the state isn't in the process. It also keeps the framework unopinionated about your infrastructure.

Implications / to build. "Bring-your-own-database" connectors (Postgres, Redis, …) behind the existing Backend interface; the interface is the seam, the connectors are add-ons.

Status. Delivered. PostgresBackend (psycopg 3, tensorsketch-core[postgres]) and RedisBackend (redis-py, tensorsketch-core[redis]) ship behind the Backend ABC — lazily imported, so the core depends on no driver. A shared SqlBackend base covers any DB-API database (SQLite and Postgres are thin dialects). Serialization is a pluggable Serializer seam (default PickleSerializer). The full durability suite runs against Redis (via fakeredis) in CI; a Postgres DSN in LOOM_TEST_POSTGRES adds it too. Writing a connector for an unshipped store is just implementing the ABC.

D2 — The first-cut memory subsystem was removed

Decision. The keyword-search MemoryStore (InMemoryStore, memory_tools, and agent recall injection) is removed, not iterated on.

Why.

1. Keyword search doesn't work. Real recall needs semantic search (embeddings / a model), not token overlap. A toy matcher would set the wrong expectation.
2. In-framework memory conflicts with statelessness (D1). Conversation memory and user memory should be owned by the application and backed by its database, not baked into the loop.

Future approach. Memory will be an external, embedding-based store the app owns, reached through the same bring-your-own-database story — not a built-in keyword store. Agents will read and write it explicitly (e.g. via tools) rather than the framework auto-managing it.

D3 — Batteries-included providers: OpenAI, Anthropic, Google — plus an easy custom path

Decision. Ship first-class providers for OpenAI, Anthropic, and Google (each an optional install, lazily imported), and make adding any custom LLM API trivial via the ChatProvider interface. No more than those three built in; everything else is a ~30-line custom provider.

Why. These three cover the vast majority of usage; beyond them, breadth is better served by a low-friction extension point than by the core carrying every SDK.

Status. Anthropic, OpenAI, and Google providers built (lazy imports, optional extras, mapping covered by fake-client tests). OpenAIProvider also drives OpenAI-compatible endpoints via base_url. Custom path documented. Gate for Phase 1 is cleared. Live-API verification for all three is still pending (see status.md → Verify before shipping).

D4 — Phase 1 canvas adopts the Excalidraw aesthetic only

Decision. When the code⇄canvas engine (Phase 1) lands, the canvas and blocks take on Excalidraw's look — hand-drawn shapes, color palette, and theme. The block content and behavior are entirely TensorSketch's (typed ports, holes, the graph model); only the visual style is borrowed.

Why. Excalidraw's friendly, sketchy aesthetic fits the "head-start sketch you refine in code" positioning, and it's a familiar, well-liked visual language.

D5 — Extensibility is named factories, not a plugin ecosystem

Decision. The way to select or extend TensorSketch's swappable parts is a small registry that maps a name → factory, with lazy loading. Only two seams get one — providers and backends — via create_provider("anthropic", …) / create_backend("postgres", …) and register_* / tensorsketch.providers · tensorsketch.backends entry points. No Plugin bundle object, and no splitting TensorSketch into core/community/provider packages.

Why.

1. No install fragmentation. Everything first-party stays in the single tensorsketch package; extras (tensorsketch-core[anthropic], …) only gate a heavy third-party SDK/driver, never TensorSketch itself. A user should pip install tensorsketch-core and have the batteries, not assemble a constellation of sub-packages (the pattern that made early LangChain painful).
2. The real win is config-driven selection. Naming a model/database lets a config value or a --flag choose it, instead of a hard-coded import — the only ergonomic gap worth closing.
3. Keep the zero-import property. Names resolve lazily (built-ins are thunks; entry points .load() on demand), so import tensorsketch still pulls in no optional SDK and listing names imports nothing. Entry points are inherently lazy, which is why they fit.
4. Small on purpose. Two registries cover the seams where names pay off. Middleware, tracers, and tools are constructed and passed explicitly today; a registry can be added later if a real need appears, on this same primitive.

Status. Built (tensorsketch.registry): a generic lazy Registry[T], the providers and backends instances pre-registered with lazy thunks, create_*/register_* helpers, and entry-point discovery (explicit registration overrides an installed name). Tested including a fresh-interpreter check that listing/creating imports no provider SDK or DB driver.

D6 — Serving is an optional ASGI layer over standard protocols

Decision. Serving a TensorSketch agent (OpenAI-compatible, A2A, AG-UI) is an optional tensorsketch-core[serve] extra built on Starlette. Each factory (openai_app / a2a_app / agui_app) returns a mountable ASGI app; the user runs it with any ASGI server (uvicorn). A2A also has a consume side — a2a_tool(url) — so TensorSketch sits on both ends. A shared ChatAdapter (to_input/to_reply) is the single seam between a graph and every protocol.

Why.

1. Async-native + real SSE. All three protocols stream over Server-Sent Events; the framework is async, so an ASGI foundation is the right substrate (vs. bridging an async engine into a sync stdlib server). Starlette is tiny, standard, and battle-tested.
2. Dependency-free core preserved. Starlette/httpx import only inside tensorsketch.serve — import tensorsketch still pulls in no web framework, consistent with the mcp/otel/db extras.
3. Mountable, not a bundled server. Returning an ASGI app (not a serve() that binds a port) lets users pick their server, add middleware/auth, and mount under an existing app.
4. One adapter, three protocols. The protocols differ only in envelope; the graph↔chat mapping is shared, so a non-agent graph is served by overriding to_input/to_reply, nothing else.

Scope (deliberate subsets). Token-level streaming waits on provider token streaming (today the completed reply is sliced into real SSE deltas). A2A implements the agent card + message/send / message/stream with a completed-task result, not the full task store / tasks/* / push-notifications. Multi-turn history and inbound tool definitions aren't wired into the default agent. Each is a clean extension on this same foundation.

Status. Built (tensorsketch.serve): openai_app, a2a_app + a2a_tool, agui_app, ChatAdapter, AgentCard, shared SSE helper. Tested over the real ASGI apps (Starlette TestClient + httpx ASGI transport), including an A2A consume↔expose round-trip. import tensorsketch verified free of Starlette/httpx.

D7 — Evaluation grades the trajectory, over the trace we already emit

Decision. The eval harness (tensorsketch.eval) is a code-first, in-package subsystem (no extra to install) built directly on TensorSketch's own Trace. It models the task/trial/transcript/outcome/grader anatomy: a Case runs over multiple trials, each producing a Trial (trace + final state + env) scored by a hybrid grader set — code-based checks (answer, tool trajectory + payload, step efficiency, cost/latency, final-state) and LlmJudge (binary, one-criterion). evaluate returns a Report with completion rate, pass@k / pass^k, cost/latency, and a per-grader breakdown; report.require(...) is the CI gate. Trials run behind a Sandbox seam (in-process default). This is the offline half of the lifecycle.

Why.

1. The trace is the transcript. TensorSketch already records every model/tool call with tokens, cost, and timing (native tracing, D-tracing). So the harness doesn't rebuild observability — it grades the span tree. That's the whole reason tracing came first. (One enabling edit: the agent's tool span now carries the call's args and result, so ToolArgs can grade the payload, not just the path.)
2. Agents are path-dependent and non-deterministic. Hence multi-trial by default and both pass@k (retry-friendly) and pass^k (consistency) — a single run is statistically invalid.
3. No single grader suffices. Code for the deterministic/objective (fast, cheap, reproducible), LLM-as-judge for the open-ended (binary, atomic criteria to stay calibratable). Human-in-the-loop is supported as data (a Trial with its transcript), not a bundled UI.
4. Anti-gaming by construction. Graders run in the harness, never inside the agent's execution; the Sandbox only runs the agent and hands back artifacts. So an agent can't overwrite the grader the way agents have gamed in-process benchmarks. A fresh env per trial prevents cross-contamination.
5. Code, not YAML. Cases and graders are Python, consistent with "code is the source of truth" — a config-matrix surface would contradict the framework's thesis.

Deferred (logged, not lost). Stronger sandboxes (SubprocessSandbox / DockerSandbox / remote) behind the same seam — needed once agents write code / run bash. Online evaluation (async sampling of production traces with the same graders; drift detection). The feedback-loop as tooling (an annotation queue + trace→golden pipeline; today a failure is a Case you construct). More trajectory metrics (memory hit rate, scope adherence). Judge calibration tooling (rank correlation vs. human labels). An unbiased pass@k estimator for k < n.

Status. Built (tensorsketch.eval): Case/Suite/Trial/Grade/Grader, the code + judge graders, Sandbox/InProcessSandbox, evaluate + Report (metrics, render, require). Tested (test_eval.py, 13) and exampled (examples/evaluation.py). make check green.

D7a — Results are emitted through a sink; online eval reuses the same graders

Decision (extends D7). Eval results serialize to plain JSON (to_dict() on Report / CaseResult / TrialResult / Grade) and are delivered through a Reporter sink — one method, emit(record), sync or async. Built-ins: JsonlReporter (dependency-free file log) and CallbackReporter(fn); any database is a ~5-line custom emit. Online evaluation is score(output, trace, graders) (grade one live run, reference-free) and OnlineMonitor(graders, reporter=, sample=) (sample production runs, score off the response path, emit). Trial.case became optional so a captured run needs no Case.

Why.

1. Stateless: emit, don't own. TensorSketch must not bundle a results database. Serializing to JSON and handing it to a sink lets a team put results in their store — file, warehouse, dashboard, any DB — exactly as the framework's persistence stance requires.
2. A dedicated Reporter, not the Backend ABC. Backend is for run durability (checkpoints/journal keyed by thread); eval results almost always belong in a different place (analytics/dashboard). Keeping the sinks separate lets checkpoints, traces, and eval scores each target the store that fits — decoupled.
3. Symmetry with tracing. This is the tracing-exporter pattern (FileTracer / OTelTracer / _record) applied to scores. Traces cover the transcript; Reporter covers the verdict.
4. Online = the same graders on a live trace. Graders already read a Trace, so production monitoring needs no new grading — just a reference-free subset (safety, tool-failure, cost/latency, on-policy judges) plus sampling and a sink. What's deferred is drift/alerting over the emitted stream, not the scoring.

Status. Built (tensorsketch.eval): to_dict() throughout; Reporter + JsonlReporter + CallbackReporter; evaluate(reporter=…); score + OnlineMonitor. Tested (test_eval.py, 18) and exampled (examples/evaluation.py shows offline emit + a scored production trace). Green.

D7b — Drift detection: lightweight streaming stats, in-process window, emit-only

Decision (extends D7a). A DriftMonitor consumes the online result stream and raises a DriftAlert when behavior drifts from a Baseline (Baseline.from_report(report) or set directly). Two detectors, both stdlib: a two-proportion z-test on the overall / per-grader pass rate (binary outcomes -> a proportion test is exactly right; per-grader so one collapsing check trips alone), and the Page-Hinkley change-point test on cost/latency (fed the value relative to the baseline mean, so a single (delta, lambda) spans dollars and milliseconds). DriftMonitor implements the Reporter protocol, so it chains straight after OnlineMonitor; a new MultiReporter fans each result to a store and the monitor. A per-metric latch makes a sustained regression fire once until it recovers.

Why.

1. Chosen over distributional (PSI/KL/KS) for the first cut. The z-test and Page-Hinkley are interpretable, threshold-light, and dependency-free — you can read why an alert fired. PSI/KL/KS over a reference distribution (population shift) is a real next layer, but heavier to calibrate; deferred, logged.
2. Stateless still holds. The rolling window lives in this process's memory; the monitor persists nothing. It emits alerts through the same Reporter seam and never owns a drift store — identical stance to results (D7a) and traces.
3. No new grading, no new sink type. It reads the TrialResult records the stream already emits and forwards alerts through the existing Reporter; MultiReporter composes rather than adding a bespoke pathway.

Deferred (logged). Distributional/population-shift detectors (PSI / KL / KS); routing alerts to a pager/webhook (today: emit a DriftAlert record, wire your own alerting); auto-tuned thresholds.

Status. Built (tensorsketch.eval.drift): DriftMonitor, Baseline, DriftAlert, PageHinkley, two_proportion_z, plus MultiReporter. Tested (test_eval.py, 24) and exampled (examples/evaluation.py fires a pass-rate alert). Green.

D8 — Dynamic fan-out is Send from a router; instances stay durable

Decision. Graph-level dynamic fan-out is a Send(node, input) value returned from a router's path (alongside, or instead of, plain node names). The engine schedules one superstep task per Send, each with its own payload overlaid on the shared snapshot (filtered to the target node's In); all instances merge at the barrier via the target's write channels — so an aggregating Reducer/Topic channel is the reduce half. Added Graph.router (the intent-named form of conditional) and Graph.loop(node, until, *, exit=END) (repeat-until sugar). This is the graph-level map/reduce; gather_map remains the in-node-body version.

Why.

1. Reuse the conditional machinery, don't grow a new one. A router already computes successors from post-barrier state; letting it also yield Sends means fan-out rides the existing edge/ scheduling path. No new builder concept, no parallel API. Payloads are state-shaped (the payload overrides the worker's channel reads for that instance), so compile-time port validation is unchanged — a worker still declares In fields that exist on the state.
2. Instances must be individually durable. Two instances of one node in one superstep would have collided in the effect journal (keys were superstep:node:index:name). Added an instance tag to Context folded into the key only when non-empty, so a normal node's keys are byte-identical (no journal migration) while fan-out instances stay distinct and replay their own result on resume.
3. Pending fan-out belongs in the checkpoint. A crash between scheduling a Send and running it must resume the same work. Added Checkpoint.sends ((node, payload) pairs); because every backend pickles the whole Checkpoint, this carries through memory / SQLite / Postgres / Redis with no per-backend change. Instance tags are the send's ordered position, so they're stable across resume.

Limitation (documented). Successors are computed from shared post-barrier state, so per- instance routing after a fan-out isn't supported (all instances of a node route identically — which is exactly what converges them on a collector). Nested fan-out from a fanned-out node would duplicate; the map -> workers -> reduce shape is the supported one.

Deferred (logged). map/parallel as first-class Graph constructs (still body-helpers); canvas extraction of router/loop/Send (they run today but the Studio round-trip doesn't read them yet — a Phase 1 follow-up).

Status. Built: tensorsketch.Send; engine _plan + per-instance execution; Context.instance; Checkpoint.sends; Graph.router/Graph.loop. Tested (test_fanout.py, 8; plus a parametrized crash-mid-fan-out resume in test_durability.py across memory/SQLite/Redis) and exampled (examples/dynamic_fanout.py). Phase 2 complete. make check green.