Metadata-Version: 2.4
Name: vigil-monitor
Version: 0.5.0
Summary: Vigil: monitor everything an AI model or agent does, from the outside, with guarantees (ledger, counterfactual replay attribution, revealed objective, conformal escalation, fingerprinting, gateway, dashboard)
License: MIT
Project-URL: Documentation, https://github.com/
Keywords: ai,llm,agents,observability,monitoring,safety,prompt-injection,conformal,opentelemetry
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: System :: Monitoring
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.24
Dynamic: license-file


# Vigil — monitor everything an AI model or agent does, from the outside, with guarantees

`pip install vigil-monitor` · MIT · numpy only · Python 3.9+ · open research for the AI community

Vigil watches AI models and agents **without needing anything from the model vendor**: no reasoning traces, no
logprobs, no weights. It treats the deployed system as a scientific instrument: conserved quantities are checked,
causes are established by intervention, behaviour is compared with the declared objective, every alarm carries a
stated false-alarm rate, and a silent change of the model behind an API is detected from the outside.

## In one minute

An AI agent reads documents, calls tools and answers. Vigil records what it did, checks that its permissions, secrets
and budget balanced, replays it with pieces of its context removed to find out **what caused each action**, compares
its behaviour with the goal it was given, and wraps every alarm in a **stated false-alarm rate**. A separate canary
check tells you when the model behind the API has been **silently swapped**. A gateway can **block** a disallowed
tool call before the agent ever sees it.


### Who it helps


### Use cases


### How an injection gets caught, step by step


### A clean run reads zero


### What "guaranteed" means here


### Every instrument says how hard it is to fool


## Scorecard of live results


## What it does

| Instrument | Question it answers | Needs |
|---|---|---|
| **Ledger** | Did authority, information and budget balance on every step? (double-entry bookkeeping for agents) | traces |
| **Replay attribution** | Which context item *caused* the action? (Shapley over counterfactual replays, exact removal first) | a replay endpoint |
| **Revealed objective** | Did the agent optimise the objective it was given? (regret residual; GARP rationalisability) | declared objective + feasible menu |
| **Conformal escalation** | With what guaranteed false-alarm rate should this be escalated, and has that guarantee broken? | known-good runs |
| **Fingerprint** | Has the model behind the endpoint silently changed? (rotating canaries, paired test, BH control) | canary prompts |
| **Judge invariance** | Does the evaluator change its verdict when consequences change? (monitor the supervisors) | a judge |
| **Grounding, memory, drift, fleet, registry, oversight, infra, probes** | claim support, memory poisoning, embedding drift, loops and cascades, AI bill of materials, signed approvals, server and GPU telemetry, activation probes on owned weights | see docs |
| **Gateway** | Enforce it: strip disallowed tool calls before the agent sees them; approval unlock; fail closed | sits in front of the model |
| **Store + dashboard** | Time series of verdicts, ledger residuals and signals; self-contained dashboard; `/metrics` for Prometheus | SQLite |

## Results on live models in detail (16 September 2026)

Two open-weight agents on vLLM, 20 injected and 20 clean runs per condition. Every number below is reproducible from
the scripts in `experiments/` and the JSON in `experiments/results/`.


Replay attribution named the injected document as the cause in 16/16 (Qwen3-8B) and 15/16 (Qwen3-32B) of the runs
where the model followed the injection, every one escalated as high, with zero false alarms on clean runs.


Injections split across documents, base64-encoded, or hidden in a tool result were followed by the 32B agent every
time and caught every time by three independent instruments.


The model behind the endpoint was swapped from Qwen3-32B to Qwen3-8B. The paired per-canary test caught it; a pooled
divergence did not. Drift marks the calibration set stale so every downstream guarantee is reported as unbacked.


With owned weights, a linear probe on the residual stream detects injections and generalises to unseen phrasings.


The gateway stripped 13/13 injected transfers before the agent saw them and unlocked one after a human approval; the
real LLM judge was consequence-invariant; grounding surfaced that half the model's claims went beyond thin evidence.


The dashboard over 24 hours of stored verdicts: conservation accounts, escalations, regret and disagreement, the
fingerprint band breaking at hour 20 and marking calibration stale, coverage, infrastructure, recent escalations.

## Quick start

```python
from vigilai import Monitor, ModelFingerprint, Store, from_spans
from vigilai.adapters import OpenAICompatibleRunner

runner = OpenAICompatibleRunner("https://api.example/v1", "model-name", tools_schema, api_key="…")
m = Monitor(runner=runner, alpha=0.05)
m.store = Store("vigil.db")                                   # persist every verdict and signal
m.calibrate([from_spans(s, tool_meta=meta) for s in known_good_span_sets])
m.attach_fingerprint(ModelFingerprint().fit(baseline_canary_responses))

v = m.assess(from_spans(new_spans, tool_meta=meta), output=answer_text, evidence=retrieved_chunks)
if v.escalate:
    route_to_human(v.severity, v.reasons, v.attribution.top1())
```

```bash
vigil-dashboard --db vigil.db --port 8020      # dashboard at /, Prometheus text at /metrics
```

To enforce rather than observe, put the gateway in front of the model:

```python
from vigilai import AgentPolicy, PolicyGateway, serve_proxy
gw = PolicyGateway({"agentA": AgentPolicy("agentA", {"model-name"}, {"search", "read", "summarize"}, {"read"})}, tool_meta)
serve_proxy(gw, upstream="https://api.example", port=8010)     # agents call http://gateway:8010/v1/chat/completions
```

## Design principles

1. **Evasion-hard quantities first.** Scopes used, secrets moved, budgets spent and actions under intervention must be
   violated to be beaten; text-based signals can merely appear clean.
2. **Instruments vote independently and their disagreement is reported**; each publishes its adversarial robustness
   and the attack that defeats it (`vigilai.ROBUSTNESS`).
3. **Guarantees are explicit.** Conformal calibration on known-good runs; a fingerprint change marks it stale.
4. **Trust is derived from provenance**, never declared upward; reasoning blocks are secrets by default.
5. **Standards in, standards out.** OpenTelemetry gen_ai and OpenInference spans in; decision spans and Prometheus
   metrics out. Vigil sits on top of any tracing platform; it does not replace it.

## Limits, stated plainly

Replay needs a reproducible environment. The regret residual needs a correct feasible menu (it is skipped and
declared when absent). Lexical grounding is gameable by paraphrase (plug in an entailment scorer). Nothing here sees
inside a hosted model. The dashboard and proxy have no authentication and must sit behind an authenticated ingress.
Samples are 20 runs per condition: enough to show mechanism, not to quote population rates.

## Layout

`vigilai/` package · `tests/` 43 tests · `experiments/` live experiments, node setup, plots, demo dashboard ·
`experiments/results/` dated JSON and logs · `docs/img/` figures and logo.

## Licence and community

MIT. Built as open research for the AI community; features from the wider observability field were adopted with
thanks rather than compared against. Contributions, replications and disagreements welcome.
