Metadata-Version: 2.4
Name: aid-runtime-python
Version: 0.1.39
Summary: The AID Framework Python runtime: model gateway with pinning and cost budgets, prompt registry, tool executor, OTel GenAI tracing, plain-async pipeline, and an eval harness that gates on statistical regression against a stored baseline.
Author: The AID Framework authors
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/aid-framework/aid-framework
Project-URL: Documentation, https://github.com/aid-framework/aid-framework/blob/main/docs/design.md
Project-URL: Issues, https://github.com/aid-framework/aid-framework/issues
Keywords: ai,codegen,llm,evals,opentelemetry,pydantic
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Typing :: Typed
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: httpx<1,>=0.27
Requires-Dist: opentelemetry-api<2,>=1.24
Requires-Dist: pydantic<3,>=2.7
Requires-Dist: pydantic-ai-slim<3,>=2
Provides-Extra: pgvector
Requires-Dist: psycopg[binary]<4,>=3.1; extra == "pgvector"
Provides-Extra: deepagents
Requires-Dist: deepagents==0.7.32; extra == "deepagents"
Provides-Extra: dev
Requires-Dist: mypy<3,>=1.11; extra == "dev"
Requires-Dist: opentelemetry-sdk<2,>=1.24; extra == "dev"
Requires-Dist: pytest<10,>=8; extra == "dev"
Requires-Dist: pytest-asyncio<2,>=0.24; extra == "dev"
Requires-Dist: ruff<1,>=0.6; extra == "dev"

# AID runtime — Python

The Python runtime library: the behavior layer imported by apps generated by the
[`py-fastapi`](../../packages/generators/py-fastapi/) generator. Generated code is thin glue; this is where
the behavior lives ([`docs/design.md` §4](../../docs/design.md), principle 1).

Distribution name **`aid-runtime-python`**, import package **`aid_runtime`**. The two differ deliberately: a
distribution name is conventionally hyphenated, and a Python import package cannot be. That asymmetry is
asserted in `tests/test_package.py` so it cannot drift.

Requires Python 3.11+.

```sh
pip install -e ".[dev]"        # the runtime plus its test/lint/type tooling
pytest -q
pip install -e ".[pgvector]"   # only if you want the pgvector store adapter
```

## What it provides

| Module | Provides |
| --- | --- |
| `gateway.py` | `ModelGateway` — alias resolution, capability negotiation, ordered fallback, cost accounting, GenAI spans |
| `providers/` | The `Provider` protocol, one HTTP adapter (`OpenAICompatibleProvider`), and one deterministic fake (`FakeProvider`) |
| `prompts.py` | `PromptTemplate` / `PromptRegistry` — versioned prompts, typed rendering, few-shot assembly |
| `structured.py` | `build_agent` — structured output via Pydantic AI only |
| `tools.py` | `Tool` / `ToolExecutor` — declared definitions, argument validation, structured results |
| `pipeline.py` | `Pipeline` / `PipelineContext` — plain-async named steps over a frozen context |
| `agents/` | The native agent runtime: `AgentRuntime` — the loop, transfer vs. delegate, step/token/cost budgets with bounded degrade, memory, and a checkpointed human-in-the-loop pause |
| `retrieval/` | Retrieval: connectors, chunkers, the `Embedder` / `Reranker` / `VectorStore` protocols, hybrid fusion, citations, and retrieval metrics |
| `evals/` | Datasets, metrics, and the statistical baseline gate |
| `tracing.py` | OTel GenAI spans: `llm_span`, `agent_span`, `record_usage`, `record_cost`, `record_attributes`, `record_hitl`, `record_exception` |
| `cost.py` | `Budget`, `CostLedger`, `estimate_cost_usd`, `estimate_tokens` |
| `types.py` | The request/response/usage/spec models everything else is typed against |

194 names are exported from `aid_runtime`. The layering is acyclic and one-directional: `errors` and `types` import
nothing else in the package, `gateway` is imported only by the package `__init__` and the eval runner, and no
core module imports a higher layer back.

## Model pinning

An **alias resolves to an exact provider model id**, including a pinned version. That is not cosmetic: a
baseline recorded against `gpt-4o` means nothing if the alias floats to `gpt-4o-2026-…` on the next run, so an
**unknown alias is an error** (`UnknownModelAliasError`) rather than a silent fall-through to a provider
default.

```python
from aid_runtime import (
    Budget,
    CompletionRequest,
    CostProfile,
    ModelGateway,
    ModelSpec,
    OpenAICompatibleProvider,
    PinnedModel,
)

spec = ModelSpec(
    name="gpt-4o-2024-08-06",
    provider="openai-compatible",
    cost=CostProfile(input_per_1k_usd=0.0025, output_per_1k_usd=0.01),
    context_window=128_000,
    max_output_tokens=4096,
)

gateway = ModelGateway(
    providers={"openai-compatible": OpenAICompatibleProvider(api_key=...)},
    aliases={"small": PinnedModel(alias="small", spec=spec)},
    budget=Budget(max_usd=5.0, scope="nightly"),
)

result = await gateway.generate("small", CompletionRequest.prompt("hello"))
result.text, result.usage, result.cost_usd, result.alias, result.attempts, result.fell_back
```

- `resolve(alias)` returns the `PinnedModel` (alias + `spec`); `provider_for(alias)` its provider. `require(alias,
  capabilities)` asserts the capabilities *and* returns the pinned model, checking **both** sides — a model that
  supports tool calling behind an adapter that does not serialise tool definitions cannot tool-call — and raises
  `CapabilityMismatchError` naming what is missing.
- `generate(alias, request, *, fallbacks=(), budget=None)` tries the alias then each fallback **in the order
  given**, and reports the aliases actually attempted on the result. Fallbacks are ordered, not conditional.
- Every charged call opens a span and updates the ledger. `CostLedger` tracks per-call `LedgerEntry` records
  and yields a `SpendSummary`; `Budget` (`max_usd`, `max_total_tokens`, `scope`) is enforced before the call,
  and `estimate_usage` / `estimate_cost_usd` price a request from a model's `CostProfile` when a caller wants
  to check before spending.

## The eval harness gates on statistics, not equality

Phase 0 ratifies sampling-based gating: a model is **expected** to vary between runs, so a metric that
compares two runs for equality is either always red or measuring nothing. Non-determinism here is not a bug to
be fixed; it is the property the gate has to be robust against.

- `Dataset` — versioned cases with inputs, expectations, tags and metadata; `from_records` / `from_json` /
  `from_path`, `to_json`, and lookups by id (`case`) and by tag (`by_tag`).
- Metrics — `ExactMatch`, `NormalizedContains`, `JsonSchemaValid`, and `LlmJudge`. `metric_names` and
  `default_metrics` cover the common set.
- `run_evaluation(dataset, runner, metrics, runs=5)` samples the runner **N times** and produces an `EvalRun`
  of per-run metric means, so a single unlucky run does not decide the verdict. `evaluate_once` is the
  single-pass primitive; a `DatasetRun` gives per-case outcomes and a rendered `summary()`.
- `EvalBaseline.from_run(run, model=..., note=...)` stores a baseline as JSON: per metric the sample count,
  mean and spread. `runtimes/python/tests/fixtures/capitals-baseline.json` is a committed example.
- `BaselineGate(thresholds).evaluate(run, baseline)` compares the current sample against the baseline using a
  one-sided normal test, with each `Threshold` carrying `metric`, an optional absolute `min_mean`, a
  `max_regression` allowance, and an `alpha` significance level. A metric passes by clearing `min_mean`, by
  being within `max_regression` of the baseline, or by regressing only within sampling noise.

Two deliberate limits:

- **`LlmJudge` is implemented and never CI-required.** A judge is itself a model call, and a gate that
  depends on one non-deterministic call inside the gate you are trying to trust is the problem this design
  avoids. `gateway_judge(gateway, alias)` builds the judge call when a caller wants it.
- **`GateConfigurationError` refuses a threshold on a model-derived metric**, because an N-run sample of a
  probabilistic judge cannot carry a fixed threshold honestly.

`cost` is the enforced budget; a latency p95 budget belongs with Phase 2 dashboards, not here.

## Retrieval

`retrieval/` is an IR-driven layer: `binding_from_spec(retriever, embedding=..., store=...)` folds the IR's
`retrievers`, `embeddings` and `stores` entries into one `RetrieverBinding`, which a `Retriever` then ingests
and searches through. The runtime consumes the names the IR emits, field for field.

- Three protocols, in the same style as `providers.Provider`: `Embedder`, `Reranker`, `VectorStore`. Phase 1
  ships the implementations listed below and no others.
- `FakeDeterministicEmbedder` maps text to a stable unit vector with a hash — keyless and offline, which is
  what lets a retrieval gate run in CI. `OpenAICompatibleEmbedder` is the one hosted adapter: a single POST to
  `/embeddings` over `httpx`, with the retry, timeout and error-chaining behaviour of the HTTP provider.
- `LexicalReranker` is the only reranker. It is deterministic and lexical; there is no hosted rerank adapter.
- Chunk ids are **content hashes**, so re-ingesting the same input produces the same ids and a no-op diff.
  Ingest is therefore held to idempotence rather than to the byte-identical codegen guarantee.
- **Pinning.** `embedModel` names an IR `embeddings` entry and resolves at codegen time to an exact `modelId`
  plus its recorded width; the resolved id, width and store `kind` travel into the generated code. An alias is
  never resolved at runtime, because a baseline recorded against a floating model means nothing.
- **Cost.** Embedding calls charge the existing `CostLedger` through the existing `Budget`, using the catalog's
  `CostProfile` for the pinned model. There is no retrieval-specific budgeting machinery.

`RetrievalMetric` (`RecallAtK`, `MeanReciprocalRank`, `CitationGrounding`) is deliberately a **separate
protocol** from the shipped `Metric`, which scores a single string prediction. Retrieval scores a *retrieved
set*: recall@k and MRR need the ranked ids, and citation-grounding needs the passages behind the citations.
Widening `Metric` would have changed an interface the existing gate already threshold-checks, for every
existing implementation — so retrieval is sampled through `RetrievalGate.sample(...)` instead, producing a
standard `EvalRun`, and then judged by the **unchanged** `BaselineGate`. Retrieval metrics are deterministic,
so they pass the gate's own eligibility check.

### pgvector

`PgVectorStore` is a thin adapter over `psycopg` and the `vector` extension. Two consequences worth stating:

- It is **locally provisioned and not covered by any gate.** No CI job starts a database, and no required
  check touches one; the in-memory adapter carries the store gates, and the two adapters share the same
  `VectorStore` protocol and the same `similarity` semantics. `compose.pgvector.yml` beside this README brings
  up a database for local use.
- The `[pgvector]` extra is **import-safe when absent**: importing `aid_runtime` or `aid_runtime.retrieval`
  never imports the driver, and the adapter loading it raises a `ConfigurationError` naming the extra.

## Control flow is plain async

`Pipeline` is a list of named async (or plain) steps over a `PipelineContext` that is **frozen**, so a step
cannot mutate what an earlier step observed — it returns a new context, which keeps a failed run's state
inspectable. A step may return a context, a mapping, or `None`. Each step runs in its own span, a step raising
anything aborts the run with the original exception chained (`PipelineError.__cause__`), and `StopPipeline` is
the supported early exit.

**Pydantic AI is used for structured output and nothing else** (`structured.build_agent`). The orchestration
library stays out of the control-flow path deliberately, so it remains swappable, and **no graph/orchestration
dependency is taken at all**: the agent loop in `agents/` is this package's own, and escalation routing and
third-party orchestration frameworks remain out of scope ([`docs/design.md` §14](../../docs/design.md)).

Tracing follows the OTel GenAI semantic conventions: `gen_ai.*` attributes for the operation, model, provider,
request parameters and usage, with AID-specific values namespaced `aid.*` so the two never collide. A failing
call records its exception exactly once.

## The agent runtime

`AgentRuntime.from_ir(document, providers=..., ...)` reads the emitted IR and runs an agent to a stop
condition: model call, tool calls, observations, until a terminal answer, `maxSteps`, or a budget breach.
Model calls go through `ModelGateway`, so fallback, cost accounting and tracing apply without a second
implementation.

**Transfer and delegation are different, and the API says which.** `_transfer` (`handoffs`) hands the
conversation to the target and never resumes the caller, so `RunResult.agent_id` is the target and `path`
grows. `_delegate` (`delegates`) runs the target in an isolated `[system, user]` context and returns its answer
as a single tool observation, so `RunResult.agent_id` is the caller and `path` is unchanged. Both refuse to
re-enter an agent already on the path.

**A budget breach fails hard.** `BudgetExceededError` stops the run; the gateway preflight and the ledger both
enforce the ceiling. An agent declaring `onBudgetExhausted: 'degrade'` (v0.2) instead steps exactly one rung
down its declared alias ladder — no retry loop, and the build-time gate rejects the declaration when no
strictly cheaper alias exists. Each degraded step is recorded in `StepRecord.degraded_from`, and a checkpoint
carries its current alias so a resumed run re-routes from where it stopped.

**Memory** is in-process plus one store-backed adapter. `thread` scope is the checkpointer unit: trimmed to its
`tokenBudget`, never written to a store. `user` and `global` scopes read and write the store and render as one
extra system message, so injection is visible in the request rather than hidden.

**Human-in-the-loop** pauses by returning a checkpoint (completed steps, pending call, current alias), and
`resume(checkpoint, decision)` continues it. The pause is recorded as an `aid.hitl` span event, not an
attribute, because a run can pause more than once.

## Development

Four gates, all run by the `python` job in [`.github/workflows/ci.yml`](../../.github/workflows/ci.yml) over
real content:

```sh
pytest
ruff check .
ruff format --check .
mypy
```

`mypy` runs strict over `src` **and** `tests`, so the tests are type-checked too. The `python` job is a real
gate but is **advisory**: the `main` ruleset requires the `node`, `workflows` and `licenses` contexts only.

`tests/fixtures/capitals-baseline.json` is regenerated from `FakeProvider(accuracy=0.6, seed=0)` over 10 runs
— the fake is seeded and deterministic per call, so the committed baseline reproduces exactly while a fresh
sample of it still varies, which is exactly the property the gate has to tolerate.

**Non-responsibility:** no app-specific prompts, tools, or rubrics. Those belong to the generated app's
`business/` tier ([`docs/design.md` §8.4](../../docs/design.md), the seam).

**Status:** Phase 0 scope implemented, plus the Phase 1 retrieval runtime. One thing deferred and
undeclared, on purpose: `ModelGateway.stream` raises `FeatureDeferredError`, and no provider advertises
`Capability.STREAMING` or target declares the capability, because the transport is request/response-only.
Retrieval likewise fails loudly rather than
approximating: non-file sources, `semantic` chunking, `incremental` refresh, retriever `filters`, a hosted
reranker, and the `qdrant` / `pinecone` / `weaviate` / `custom` store kinds all raise `FeatureDeferredError`.
Also deferred (§14): escalation routing, memory, guardrails, prompt/semantic caching, rate limiting, token
buckets, and adapters beyond the one HTTP provider and the fake.

Emitting retrieval into a generated app and indexing it from the CLI are later layers, so this package adds no
entry point for either.

Reference: [`docs/design.md` §9, §12, §14, §17 #2](../../docs/design.md).
