Metadata-Version: 2.4
Name: cortex-bus
Version: 0.4.1
Summary: Reference implementation of the Cortex Bus spec: semantic pub/sub with tiered (topic / embedding / LLM) routing
License: Apache-2.0
Project-URL: Homepage, https://github.com/cortex-bus/cortex-bus
Project-URL: Specification, https://github.com/cortex-bus/cortex-bus/blob/main/spec/SPEC.md
Project-URL: Issues, https://github.com/cortex-bus/cortex-bus/issues
Keywords: pubsub,message-bus,ai-agents,mcp,a2a,kafka,semantic-routing,llm
Classifier: Development Status :: 3 - Alpha
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: System :: Distributed Computing
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: kafka
Requires-Dist: confluent-kafka>=2.3; extra == "kafka"
Dynamic: license-file

# Cortex Bus

**A bus that is an agent, on any transport.**

Cortex Bus is an open specification and reference implementation for an *agentic* publish/subscribe layer. It sits on top of the message transport you already run (Kafka, NATS, Redis Streams, SQS) and works with whichever model you choose (Claude, OpenAI, Gemini, Ollama, vLLM). On that base it adds what a conventional queue can't do: it understands what a message means.

| Conventional broker | Cortex Bus |
|---|---|
| String topics | Semantic subscriptions: subscribe by meaning |
| Payload delivered as-is | Schema mediation: each subscriber gets its own contract |
| Dedupe by message ID | Semantic dedupe: 40 alerts about one incident become 1 |
| Static priority | Reasoned urgency, with a recorded rationale |
| Slow consumer → lag or drop | Backlog summarization: a digest instead of a loss |
| Dead-letter queue | Self-healing DLQ: diagnose, repair, retry or escalate |
| Replay the raw log | Briefings: "catch me up on order 123 since Monday" |
| One reply | Consensus fan-in: ask N agents, reconcile the answers |
| Per-topic ACLs | Per-subscriber redaction by clearance |
| Opaque routing | Every decision audited and replayable |

## Design principles

1. **Deterministic first.** Exact topic match runs in microseconds, embeddings in milliseconds, and the LLM runs only when the lower tiers are unsure.
2. **Agnostic by contract.** Transport and model are adapters behind small interfaces.
3. **Explainable.** Every routing, transform or drop decision writes an audit record with its reason, model, prompt hash and input hash.
4. **Safe by default.** Redaction and clearance are checked before any model sees a payload. Any topic can be declared `strict`, which means no model in its path.

## Repository layout

```
spec/SPEC.md                     The specification (v0.1, draft)
schemas/                         JSON Schemas: envelope, subscription, audit record
cortexbus/                       Reference implementation (Python ≥3.10, standard library only)
  core.py                        The bus: tiered routing, delivery pipeline, briefings, fan-in, audit + replay
  guard.py                       Tier 1 injection guard: phrase patterns and a model screen
  mediation.py                   Schema mediation, transform promotion, DLQ repair proposals
  schema.py                      Small JSON Schema validator (stdlib)
  mcp.py                         MCP server over stdio (SPEC §9.1)
  http_api.py                    HTTP tool API, A2A agent (SPEC §9.2), SSE / WebSocket endpoints, live viewer
  streams.py                     Server-Sent Events and WebSocket (RFC 6455) delivery, stdlib
  store.py                       Durable state in SQLite
  connectors.py                  Sources and sinks: JSON Lines files, webhooks, Kafka (optional client)
  manifest.py                    Manifest loading: JSON or YAML (stdlib YAML subset), validated
  runtime.py                     Shared server start-up; in-process, HTTP and A2A fan-in responders
  adapters/                      Embedder / reasoner adapters: local, Ollama, Claude
conformance/                     Routing conformance suite: fixtures, runner, reports, RESULTS.md
tests/                           Unit tests (mocked HTTP) and an end-to-end test over local sockets
examples/subscriptions/          Subscription manifests (YAML)
examples/envelopes/              Sample message envelopes
examples/python/quickstart.py    Single-file offline demo (the original spec sketch)
examples/python/capabilities.py  Tour of the package's capabilities (offline, or --real)
```

## Install

```bash
pip install cortex-bus            # the package; import it as cortexbus
pip install "cortex-bus[kafka]"   # adds the Kafka connectors
```

The spec is [spec/SPEC.md](spec/SPEC.md).

To run it as an MCP server (stdio), point your MCP client at `uvx cortex-bus`. Add `--store bus.db` to keep state between runs. It is listed in the [MCP Registry](https://registry.modelcontextprotocol.io) as `io.github.cortex-bus/cortex-bus`.

<!-- mcp-name: io.github.cortex-bus/cortex-bus --> The package needs Python 3.10 or later and has no required dependencies.

## Quickstart

```bash
python3 examples/python/quickstart.py      # single-file offline demo
python3 examples/python/capabilities.py    # capabilities tour on the package (add --real for Ollama + Claude)
python3 -m conformance.run --calibrate     # offline routing benchmark (no dependencies)
python3 -m unittest discover -s tests      # tests, no network needed
```

The demo runs fully offline. It uses an in-memory transport, a bag-of-words embedder and a rule-based stand-in for the LLM, and shows tiered routing, schema mediation, redaction, semantic dedupe, backlog digests and the audit log.

## With real models

The embedder (Tier 1) and the reasoner (Tier 2 judge) are separate roles, so any embedder pairs with any reasoner. Ollama provides embeddings and an optional local judge. Claude (Anthropic Messages API) provides the judge, schema mediation and summaries. Models only ever see the `pii:none` view of a payload, wrapped in `<message>` tags as data.

```python
from cortexbus import CortexBus, Subscription
from cortexbus.adapters import build_embedder, build_reasoner

bus = CortexBus(build_embedder("ollama"), build_reasoner("claude"), delta=0.05)
bus.subscribe(Subscription("fraud-agent", intent="possible account takeover or fraudulent payment activity", threshold=0.58))
print(bus.publish({"id": "m1", "topic": "payments.events", "data": {"text": "New device, password reset, then a $4,800 transfer"}}))
```

Setup: `ollama pull nomic-embed-text` (and `ollama pull llama3.2` for a local judge), and `export ANTHROPIC_API_KEY=...` for Claude. Optional settings: `OLLAMA_HOST`, `CORTEX_CLAUDE_MODEL` (default `claude-haiku-5-5`), `CORTEX_OLLAMA_MODEL`, `CORTEX_OLLAMA_EMBED_MODEL`. Thresholds depend on the embedder's similarity scale, so calibrate them with `python -m conformance.run --embedder ollama --calibrate`. Measured routing quality per model combination is in [conformance/RESULTS.md](conformance/RESULTS.md).

## Capabilities

Everything below is in `cortexbus` and covered by offline tests; `examples/python/capabilities.py --real` runs it against Ollama + Claude.

| SPEC | Capability | API |
|---|---|---|
| §5 | Tiered routing (topic → embedding → LLM judge), Tier 1 injection guard, per-subscriber and per-publisher token budgets, `max_tier_1` policy | `publish(env)` |
| §6.2 | Schema mediation: model transform on the `pii:none` view, PII re-attached per clearance, JSON Schema validation, transform promotion | `Subscription(schema=…, language=…, detail=…)` |
| §6.3 | Semantic dedupe within a window, per trace root unless `cross_trace` | `dedupe_threshold`, `dedupe_window` |
| §6.4 | Digests (`mode: digest`, `digest_window`, `max_lag`) and overflow `summarize` / `drop_oldest` / `block` | `consume(sub)`, `lag(sub)`, `flush_digests()` |
| §6.5 | Self-healing DLQ: classify, propose a repair, retry once, escalate; strict messages never repaired | `dlq_list()`, `dlq_resolve(id, action)` |
| §6.6 | Briefings with citations | `brief(query, since, scope)` |
| §6.7 | Consensus fan-in with quorum, timeout, model or majority reconciliation | `register_responder()`, `ask()` |
| §8 | Audit record per decision, explain and replay | `explain(id)`, `replay(id)` |
| §9.1 | MCP server (stdio) | `python -m cortexbus.mcp --embedder ollama --reasoner claude --manifest subs.yaml` |
| §9.2 | A2A agent and HTTP tool API; skills mirror the MCP tools | `python -m cortexbus.http_api --port 8765 --manifest subs.yaml` |

## Running it as a service

Both servers take the same flags:

- **Manifests:** `--manifest` (repeatable) reads `.yaml`, `.yml` or `.json`: one manifest, a list, or several YAML documents. YAML is parsed by a small standard-library reader that covers what manifests use; anchors, aliases and tags are rejected with a line number. Every manifest is validated against `schemas/subscription.schema.json`, and a `contract.schema` path is resolved next to the manifest file.
- **Durable state:** `--store bus.db` keeps subscriptions, message history, the audit log, the DLQ, undelivered queues and open digests in SQLite, written in one transaction per operation. After a restart, `explain`, `replay` and briefings still work on old messages. Consumer handlers (code) are not stored, so re-attach them.
- **Fan-in across processes:** `--responder` (repeatable) takes `name=module:function` (in-process), `name=http(s)://…` (POST `{question, request}` and get back `{answer}`) or `name=a2a+http(s)://…` (an A2A agent, called with `message/send`). With `--allow-register`, clients can add remote responders at runtime through the `register_responder` tool. The bus then POSTs to URLs those clients choose, so this is off by default.

```bash
python -m cortexbus.http_api --embedder ollama --reasoner claude --store bus.db \
  --manifest examples/subscriptions/ops-inbox.yaml --responder risk=a2a+http://127.0.0.1:9001/a2a
curl -s localhost:8765/.well-known/agent-card.json             # A2A agent card
curl -s localhost:8765/publish -d '{"topic":"ops.checkout","data":{"text":"checkout 500s in eu-west"}}'
curl -s localhost:8765/tools/consume -d '{"subscriber":"ops-inbox"}'
```

### Live delivery: Server-Sent Events and WebSockets

A subscriber can take its deliveries as a live stream instead of polling `consume`. Routed messages, digests and mediated payloads arrive as they happen. Anything delivered while nobody was connected waits in the queue (and in the store, with `--store`) and is sent on connect.

- `GET /stream/<subscriber>`: Server-Sent Events. Each delivery is an `event: message` or `event: digest` whose data is the envelope as JSON. Any browser's `EventSource` works.
- `GET /ws`: a WebSocket that works both ways. Send `{"op":"subscribe","manifest":{…},"ephemeral":true}` to create a subscription for this socket only, removed when it closes, or `{"op":"subscribe","subscriber":"name"}` to attach to an existing one. Deliveries are pushed as `{"type":"message"|"digest","subscriber":…,"envelope":…}`. The same socket can `publish` and `call` any tool, such as `brief`.
- `GET /ui`: a small page that watches any subscriber's SSE stream. It works on a phone too.

```bash
curl -N localhost:8765/stream/ops-inbox            # SSE in a terminal
```
```js
const ws = new WebSocket("ws://localhost:8765/ws");
ws.onopen = () => ws.send(JSON.stringify({op: "subscribe", ephemeral: true,
  manifest: {subscriber: "my-tab", match: {intent: "customer-facing outages", threshold: 0.6}}}));
ws.onmessage = e => console.log(JSON.parse(e.data));
```

Both endpoints implement the SSE and WebSocket bindings in [SPEC §9.3](spec/SPEC.md) (message shapes: `schemas/stream.schema.json`; WebSocket subprotocol `cortex.v0.2`). Streams are event-driven: the bus announces each delivery, so an idle stream does no work, and a write that fails puts the unsent items back at the front of the queue. A connected stream is that subscriber's consumer, so each subscriber allows one live connection at a time; a second one gets 409. For fan-out to many viewers, give each viewer its own (ephemeral) subscription. Each one then also gets its own contract, clearance and digest settings. Both stream types use only the standard library.

The HTTP server listens on 127.0.0.1. To bind any other address you must set `--token` (or `CORTEX_BUS_TOKEN`), and every POST and stream then needs `Authorization: Bearer <token>`. Streams also accept `?token=`, because browsers can't set headers on `EventSource` or `WebSocket`. Requests whose `Origin` is another site are refused, so a web page you visit can't drive a bus running on your machine. A2A clients send `message/send`, name the skill in `message.metadata.skill` or in a data part `{skill, arguments}`, and get back a message with one data part. A text-only message runs a briefing.

**Connectors** (`cortexbus.connectors`) connect the bus to message systems. A source is any iterable of envelopes, and `pump(source, bus)` publishes them. A sink is a callable you attach as a subscription's `handler`, so a failed hand-off goes to the self-healing DLQ like any consumer error.

| Connector | Needs | Notes |
|---|---|---|
| `JsonlSource(path, follow=True)` / `JsonlSink(path)` | stdlib | tails a file like `tail -f`; bad lines are skipped and reported |
| `WebhookSink(url)` | stdlib | non-2xx or network error → DLQ `consumer_error`; timeout → `timeout` |
| HTTP `POST /publish` | stdlib | for producers and webhooks; any JSON value is wrapped as `data` |
| `KafkaSource(topics, …)` / `KafkaSink(topic, …)` | `confluent-kafka` or `kafka-python` | JSON records; offsets committed after each publish (at-least-once); produce waits for the broker's ack |

```python
from cortexbus.connectors import KafkaSource, KafkaSink, pump
bus.subscribe(Subscription("fraud-out", intent="account takeover", threshold=0.6, handler=KafkaSink("fraud.alerts")))
pump(KafkaSource(["payments.events"], bootstrap_servers="localhost:9092"), bus)
```

The Kafka connectors are unit-tested against an in-memory fake of the `confluent-kafka` API. `tests/test_kafka_live.py` also runs them against a real broker when credentials are present (it passed on Aiven's free tier, and CI skips it). `examples/python/kafka_live.py` streams live Wikipedia edits through Kafka into the bus, and `serve` shows the routed feed in a browser. Results are in [conformance/RESULTS.md](conformance/RESULTS.md#live-wikipedia-edits-through-a-managed-kafka-2026-11-19).

## Conformance levels

| Level | Name | Requires |
|---|---|---|
| L0 | Classic | Envelope, exact topics, audit |
| L1 | Semantic | L0 + embedding tier, semantic subscriptions, dedupe |
| L2 | Agentic | L1 + LLM tier, schema mediation, summarization, self-healing DLQ, briefings, fan-in |

## Status

Draft spec v0.1. The reference implementation covers every capability in L0–L2 (table above), plus the optional A2A interface (§9.2). State is in memory, or in SQLite with `--store`. One bus process owns a store: there is no clustering or replication. Connectors cover files, webhooks, HTTP and Kafka. NATS, Redis Streams and SQS fit the same source/sink shape but are not written yet. Feedback is welcome through issues and RFC pull requests against `spec/`.

## License

Apache-2.0
