Metadata-Version: 2.4
Name: windsock
Version: 0.1.0
Summary: Product analytics for AI agents — aggregate action-graph analytics (tool sequences, decision paths, completion funnels) across runs. Open-core.
License-Expression: MIT
License-File: LICENSE
Requires-Python: >=3.12
Requires-Dist: click>=8.1
Requires-Dist: opentelemetry-api>=1.27.0
Requires-Dist: opentelemetry-sdk>=1.27.0
Provides-Extra: radar
Requires-Dist: beautifulsoup4>=4.12.0; extra == 'radar'
Requires-Dist: ddgs>=6.0; extra == 'radar'
Requires-Dist: httpx>=0.27.0; extra == 'radar'
Requires-Dist: langchain-ollama>=0.2.0; extra == 'radar'
Requires-Dist: langchain>=0.3.0; extra == 'radar'
Requires-Dist: langgraph>=0.2.0; extra == 'radar'
Requires-Dist: openinference-instrumentation-langchain>=0.1.0; extra == 'radar'
Requires-Dist: python-dotenv>=1.0.0; extra == 'radar'
Provides-Extra: server
Requires-Dist: clickhouse-connect>=0.8.0; extra == 'server'
Requires-Dist: django>=5.0; extra == 'server'
Requires-Dist: opentelemetry-proto>=1.27.0; extra == 'server'
Description-Content-Type: text/markdown

# Windsock

**Product analytics for AI agents.** Aggregate *behavior* analytics — tool-call sequences, decision-path funnels, completion rates — across thousands of agent runs. Open-core.

> Per-trace tools (Langfuse, LangSmith, PostHog AI Analytics) let you debug *one* run. Windsock answers the questions you can only ask across *all* runs: *which tool sequences lead to failure? where do agents get stuck? did my last prompt change improve completion rate?* The unit of analysis is the agent **action graph**, not a single conversation.

Status: **early build.** Shipped so far: `radar/` — the dogfood agent (below), the [span/ingestion contract](docs/span-schema.md), the shared parser, and the `windsock report` CLI. Ingestion (ClickHouse), dashboard (React + Kea), and MCP server follow. See tracker: [`#1`](https://github.com/Ref34t/windsock/issues/1) (epic) · [`#2`](https://github.com/Ref34t/windsock/issues/2) (Radar).

---

## Install

```bash
pip install windsock              # SDK + CLI — deps: opentelemetry + click, nothing else
pip install "windsock[server]"    # + the ingest server (Django, ClickHouse)
pip install "windsock[radar]"     # + the Radar demo agent (LangGraph, Ollama)
```

Instrument an agent in five lines:

```python
import windsock.sdk as ws

ws.configure("spans.jsonl")                          # once at startup
with ws.session("research:acme", channel="web"):     # one session per run
    with ws.llm("draft", model="gpt-5.6-terra"):
        ...                                          # your provider call
        ws.usage(prompt=812, completion=344)
    ws.gate("relevance", "keep")                     # each decision, as made
    ws.outcome("complete")
```

Then `windsock report spans.jsonl` answers "is it working?" across every run.

## Quickstart — `windsock report`

Point it at an OpenInference/OTel span file (JSONL, one span per line — see [the contract](docs/span-schema.md)) and see in one screen whether your agent is working. No new instrumentation, no server.

```bash
uvx --from git+https://github.com/Ref34t/windsock windsock report spans.jsonl

# or in a clone:
uv run windsock report data/spans.jsonl --by channel
```

Real output over Radar's own 1,911 runs:

![windsock report output](docs/img/report.svg)

Flags:

| Flag | Meaning |
|------|---------|
| `--window 7d` | Rolling window for rates + trend (`24h`, `7d`, `2w`, …). Rates are windowed, never all-time-only; the all-time line is printed alongside. |
| `--by channel` | Slice the class mix by any dimension your spans carry (`channel`, `task`, …) |
| `--by version` | Release-over-release drift: class mix per producer-stamped `version` — did the outcome mix move after a deploy? Unversioned runs group under `(none)` |
| `--json` | Machine-readable output — the stable report schema below |
| `--session <run_id>` | Single-run drill-down: outcome, gates (with judged item titles), tokens |
| `--cost-per-mtok 2.50` | Add an estimated cost line (USD per 1M total tokens) |
| `--fail-over 'failure_rate>20'` | CI/agent gate: exit **3** when the windowed rate breaches (`failure_rate`/`completion_rate`, `>`/`<`). Output unchanged; breach reason on stderr. Exit codes: 0 ok · 1 error · 2 usage · 3 breach |
| `--min-runs 5` | Liveness gate: exit **3** when the window holds fewer than N runs. A dead agent (zero runs) passes every rate gate — this catches it. Pair with `--fail-over` |

There's also `windsock audit spans.jsonl` — a second-opinion pass over the agent's gate decisions: re-judges every unique titled keep/discard with an adversarial prompt (configurable LLM, default local Ollama) and prints flagged disagreements as a copy-paste-ready gold-candidate block. Report-only: it never writes anything — agents nominate, humans canonize. Cost: one LLM call per unique item in `--window` (default 24h); cap with `--limit`.

`--json` emits one object, stable under `report_schema_version` (currently `0`): `{report_schema_version, runs, window, window_end, windowed, all_time, trend, gates, tokens[, by, dimensions]}` — `windowed`/`all_time`/each `dimensions[]` entry share one rates shape (`runs, classified, unclassified, class_mix, completion_rate, failure_rate`). Rates follow the contract's §5 definition: unclassified runs are excluded from numerator and denominator and reported separately. Exit code is `0` on success.

---

## Instrument your own agent — the `trace()` SDK

No framework required. `windsock.sdk` emits the same `windsock.*` span schema Radar produces natively, so `windsock report`/`audit` and ingestion read it through the same parser. Two equivalent styles:

```python
import windsock.sdk as ws
ws.configure("spans.jsonl")

# A) explicit — you own the session boundary
def my_agent(question: str) -> dict:
    with ws.session(intent=question, channel="cli"):
        with ws.tool("search"):
            hits = search(question)
        ws.gate("relevance", "keep" if hits else "discard", title=hits[0] if hits else None)
        ws.outcome("complete" if hits else "no_results")
        return {"answer": summarize(hits) if hits else None}

# B) one-line wrap — for an UN-instrumented body
agent = ws.trace(my_agent_without_session)   # each call == one windsock.run
```

**`session()` vs `trace()` — pick one, don't stack them.** `trace()` opens the run's root span for you; use it when the wrapped body has *no* `session()` of its own. If the body already calls `session()`, call it directly — you don't need `trace()`. Wrapping an already-instrumented body still produces exactly **one run per call** (the inner `session()` folds into the outer root, and the inner `intent`/dimensions win so the outcome row reflects the real intent, not the wrapper's function name), but it emits a one-time `WindsockUsageWarning` nudging you to drop the redundant wrap. Nesting two `session()`s directly behaves the same way: one root, inner intent wins.

---

## `radar/` — the dogfood agent

Radar is a research agent that tracks the AI agent-tooling / observability space. It exists to generate **real, messy agent-behavior data** to build Windsock against — and its OpenInference span schema **is** Windsock's ingestion contract. It's also genuinely useful: its output is ongoing competitive intel.

- **Stack**: LangGraph + `openinference-instrumentation-langchain` → OpenInference/OTel spans → JSONL (`data/`).
- **Inference**: self-hosted **Ollama + Qwen2.5-3B-Instruct** on a 4-core/8GB/no-GPU VPS — zero-cost, self-host-first. See [`docs/agdr/AgDR-0001`](docs/agdr/AgDR-0001-radar-model-and-serving.md).
- **7 tools**: `web_search`, `search_hn`, `search_reddit`, `fetch_url`, `classify_relevance`, `dedupe`, `summarize`, `store`.
- **2 decision gates**: relevance (keep/discard), dedupe (new/seen) — tagged on spans for the decision-tree view.
- **Failures are real, not simulated**: `fetch_url` fails naturally on JS/paywalled/timeout pages; junk searches get discarded — producing the 20–40% failure rate that makes "where agents get stuck" meaningful.

### Run it (on the VPS)

```bash
# 1. Ollama with the model (AgDR-0001)
ollama pull qwen2.5:3b-instruct-q4_K_M

# 2. Deps + config
uv sync --all-extras
cp .env.example .env        # set OLLAMA_BASE_URL if not localhost

# 3. Single run
uv run python -m radar.run --topic "Voker agent analytics" --channel web

# 4. Batch backfill toward ≥1,000 runs
uv run python -m radar.run --backfill            # every (target × channel) seed
uv run python -m radar.run --backfill --limit 50 # first 50
```

Spans land in `data/spans.jsonl` — one trace per run, tool/gate/LLM spans nested. That file is the input the ingestion layer (build-order step 3) will consume.

### Definition of done (#2)

- [ ] Runs end-to-end on the real space, on a schedule
- [ ] Emits OpenInference spans covering all 7 tools + both gates + session boundary
- [ ] Inference against the self-hosted OSS model (no external LLM API)
- [ ] ≥1,000 runs with an observed 20–40% failure rate
- [ ] Span schema documented → proposed as Windsock's ingestion contract
- [ ] Raw material for blog post #1 (*"where my agent kept failing"*)

## Layout

```text
radar/
  config.py           # env-driven config
  instrumentation.py  # OpenInference/OTel → JSONL exporter (the ingestion contract)
  tools.py            # 7 tools + 2 gates
  agent.py            # LangGraph action graph
  seeds.py            # ~30 targets × 3 channels
  run.py              # single run + batch backfill
docs/agdr/            # decision records
data/                 # span JSONL (gitignored)
```
