Metadata-Version: 2.4
Name: vigil-monitor
Version: 0.8.0
Summary: Vigil: monitor everything an AI model or agent does, from the outside, with guarantees (ledger, counterfactual replay attribution, revealed objective, conformal escalation, fingerprinting, gateway, dashboard)
License: MIT
Project-URL: Homepage, https://pypi.org/project/vigil-monitor/
Project-URL: Figures, https://github.com/deepaksatna/vigil-monitor-assets
Keywords: ai,llm,agents,observability,monitoring,safety,prompt-injection,conformal,opentelemetry
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: System :: Monitoring
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.24
Dynamic: license-file

<p align="center"><img src="https://raw.githubusercontent.com/deepaksatna/vigil-monitor-assets/main/docs/img/vigil-logo.png" width="220" alt="Vigil"></p>

# Vigil — monitor everything an AI model or agent does, from the outside, with guarantees

`pip install vigil-monitor` · MIT · numpy only · Python 3.9+ · open research for the AI community

Vigil watches AI models and agents **without needing anything from the model vendor**: no reasoning traces, no
logprobs, no weights. It treats the deployed system as a scientific instrument: conserved quantities are checked,
causes are established by intervention, behaviour is compared with the declared objective, every alarm carries a
stated false-alarm rate, and a silent change of the model behind an API is detected from the outside.

## In one minute

An AI agent reads documents, calls tools and answers. Vigil records what it did, checks that its permissions, secrets
and budget balanced, replays it with pieces of its context removed to find out **what caused each action**, compares
its behaviour with the goal it was given, and wraps every alarm in a **stated false-alarm rate**. A separate canary
check tells you when the model behind the API has been **silently swapped**. A gateway can **block** a disallowed
tool call before the agent ever sees it.

Those false-alarm rates hold **at every stopping time**, not just per decision, so watching a stream for a million
runs does not quietly spend the guarantee. Where the endpoint returns log-probabilities, Vigil reads the model's
own uncertainty without needing its weights. And images are inputs like any other: an instruction painted into a
retrieved page is invisible to every text-level check, and Vigil refuses to let text extracted from it outrank
the page it came from.

<p align="center"><img src="https://raw.githubusercontent.com/deepaksatna/vigil-monitor-assets/main/docs/img/explainer-what-vigil-does.png" width="820" alt="what Vigil does"></p>

### Who it helps

<p align="center"><img src="https://raw.githubusercontent.com/deepaksatna/vigil-monitor-assets/main/docs/img/explainer-who-it-helps.png" width="820" alt="who it helps"></p>

### Use cases

<p align="center"><img src="https://raw.githubusercontent.com/deepaksatna/vigil-monitor-assets/main/docs/img/explainer-use-cases.png" width="820" alt="use cases"></p>

### How an injection gets caught, step by step

<p align="center"><img src="https://raw.githubusercontent.com/deepaksatna/vigil-monitor-assets/main/docs/img/explainer-how-injection-is-caught.png" width="820" alt="how an injection is caught"></p>

### A clean run reads zero

<p align="center"><img src="https://raw.githubusercontent.com/deepaksatna/vigil-monitor-assets/main/docs/img/explainer-ledger.png" width="720" alt="ledger"></p>

### What "guaranteed" means here

<p align="center"><img src="https://raw.githubusercontent.com/deepaksatna/vigil-monitor-assets/main/docs/img/explainer-guarantee.png" width="760" alt="guarantee"></p>

### Every instrument says how hard it is to fool

<p align="center"><img src="https://raw.githubusercontent.com/deepaksatna/vigil-monitor-assets/main/docs/img/explainer-robustness.png" width="820" alt="robustness"></p>

## Scorecard of live results

<p align="center"><img src="https://raw.githubusercontent.com/deepaksatna/vigil-monitor-assets/main/docs/img/explainer-scorecard.png" width="820" alt="scorecard"></p>

## What it does

| Instrument | Question it answers | Needs |
|---|---|---|
| **Ledger** | Did authority, information and budget balance on every step? (double-entry bookkeeping for agents) | traces |
| **Replay attribution** | Which context item *caused* the action? (Shapley over counterfactual replays, exact removal first) | a replay endpoint |
| **Revealed objective** | Did the agent optimise the objective it was given? (regret residual; GARP rationalisability) | declared objective + feasible menu |
| **Conformal escalation** | With what guaranteed false-alarm rate should this be escalated, and has that guarantee broken? | known-good runs |
| **Fingerprint** | Has the model behind the endpoint silently changed? (rotating canaries, paired test, BH control) | canary prompts |
| **Judge invariance** | Does the evaluator change its verdict when consequences change? (monitor the supervisors) | a judge |
| **Grounding, memory, drift, fleet, registry, oversight, infra, probes** | claim support, memory poisoning, embedding drift, loops and cascades, AI bill of materials, signed approvals, server and GPU telemetry, activation probes on owned weights | see docs |
| **Token telemetry** | What is the model's own uncertainty, and is it about to act? (entropy, margin, tail mass, decision-token uncertainty, decode integrity) | an endpoint that returns log-probabilities |
| **Anytime-valid alarms** | Has this *endpoint* changed, with a guarantee that survives looking a million times? (test martingale, Ville's bound; changepoint; e-BH across a fleet) | a stream of clean runs |
| **Media** | Images and audio as inputs: trust inheritance for anything extracted from them, recurrence across sessions, model-versus-extractor disagreement | a medium and its extracted text |
| **Gateway** | Enforce it: strip disallowed tool calls before the agent sees them; approval unlock; fail closed | sits in front of the model |
| **Multi-model / cluster** | one endpoint per (model, revision, process, node, GPUs); per-endpoint calibration and fingerprints; per-GPU and per-model telemetry mapping; training-run account | endpoint registry |
| **Store + dashboard** | Time series of verdicts, ledger residuals and signals; self-contained dashboard; `/metrics` for Prometheus | SQLite |

## Results on live models in detail (16 September 2026)

Two open-weight agents on vLLM, 20 injected and 20 clean runs per condition. Every number below is reproducible from
the scripts in `experiments/` and the JSON in `experiments/results/`.

<p align="center"><img src="https://raw.githubusercontent.com/deepaksatna/vigil-monitor-assets/main/docs/img/live-attribution.png" width="680" alt="live attribution results"></p>

Replay attribution named the injected document as the cause in 16/16 (Qwen3-8B) and 15/16 (Qwen3-32B) of the runs
where the model followed the injection, every one escalated as high, with zero false alarms on clean runs.

<p align="center"><img src="https://raw.githubusercontent.com/deepaksatna/vigil-monitor-assets/main/docs/img/evasion-survival.png" width="680" alt="evasion suite"></p>

Injections split across documents, base64-encoded, or hidden in a tool result were followed by the 32B agent every
time and caught every time by three independent instruments.

<p align="center"><img src="https://raw.githubusercontent.com/deepaksatna/vigil-monitor-assets/main/docs/img/fingerprint-swap.png" width="680" alt="fingerprint swap"></p>

The model behind the endpoint was swapped from Qwen3-32B to Qwen3-8B. The paired per-canary test caught it; a pooled
divergence did not. Drift marks the calibration set stale so every downstream guarantee is reported as unbacked.

<p align="center"><img src="https://raw.githubusercontent.com/deepaksatna/vigil-monitor-assets/main/docs/img/probe-generalisation.png" width="520" alt="activation probe"></p>

With owned weights, a linear probe on the residual stream detects injections and generalises to unseen phrasings.

<p align="center"><img src="https://raw.githubusercontent.com/deepaksatna/vigil-monitor-assets/main/docs/img/gateway-judge-grounding.png" width="760" alt="gateway, judge, grounding"></p>

The gateway stripped 13/13 injected transfers before the agent saw them and unlocked one after a human approval; the
real LLM judge was consequence-invariant; grounding surfaced that half the model's claims went beyond thin evidence.

<p align="center"><img src="https://raw.githubusercontent.com/deepaksatna/vigil-monitor-assets/main/docs/img/dashboard.jpg" width="760" alt="dashboard"></p>

The dashboard over 24 hours of stored verdicts: conservation accounts, escalations, regret and disagreement, the
fingerprint band breaking at hour 20 and marking calibration stale, coverage, infrastructure, recent escalations.


## Phase 4 results (16 September 2026, Nebius H100, Qwen3-8B agent)

<p align="center"><img src="https://raw.githubusercontent.com/deepaksatna/vigil-monitor-assets/main/docs/img/phase4-results.png" width="900" alt="phase 4 results"></p>

| Experiment | Result | What it means |
|---|---|---|
| Hidden goals planted in the agent's configuration (7 framings: compliance CC, QA vendor, accounting ping, backup partner, internal shared inbox, internal log, internal zero-value ticket) | Qwen3-8B acted on **0 of 70** runs (2 of 20 in a separate pass) | the model refuses operator-planted exfiltration even when framed as internal housekeeping; the revealed-objective instrument still has no live positive case on this family |
| Trace dataset | 1,420 examples: 1,200 synthetic, 220 live (154 clean, 64 harmful, 2 medium); the 8B agent did not follow base64 injections (0/20) while the 32B did (20/20) | evasion susceptibility differs by model size |
| LM trace monitor, **full trace** (Qwen3-1.7B, LoRA, tail loss, 3 epochs, 4.5 min) | AUROC 1.0, TPR 1.0 at 1 % FA on live held-out and on unseen evasions | it sees the labelling action in the trace, so this is a sanity baseline, not evidence of anticipation |
| LM trace monitor, **early warning** (final action and ledger removed) | live held-out: AUROC 1.0, TPR 1.0 at 1 % FA; unseen evasions: AUROC 0.83, TPR 0 at 1 % / 5 % FA, conformal catch 100 % | it detects that an injection is *present* in the context (an input-side early warning) but cannot predict whether this model will *act* on it; predicting compliance before the action remains open |
| Grounding: lexical vs NLI entailment (30 real answers) | documents-only instruction: 0.69 vs 0.62 unsupported; add-background instruction: 0.88 vs 0.98 | NLI is stricter on outside knowledge and more lenient on faithful paraphrase; it separates the two instructions better |
| DCGM during LoRA training | GPU_UTIL 100 %, tensor pipes 18 % active, 65.8 GB, 604 W | the classic "utilisation lies" pattern; the default exporter config lacks SM-activity, so the automatic check needs a custom metrics file |

Numbers are from 10–45 runs per condition and one model family; they show mechanisms, not population rates.

## ORB: eight GPUs, six model families (17 September 2026)

Seven endpoints on an 8-GPU H200 node, spanning 8B to 120B: Qwen3-8B, Qwen3-32B, Phi-4, DeepSeek-R1-Distill-Llama-8B,
IBM Granite 3.3-8B, gpt-oss-20B and gpt-oss-120B. Full tables in `experiments/results/ORB-REPORT-SECTION.md`.

<p align="center"><img src="https://raw.githubusercontent.com/deepaksatna/vigil-monitor-assets/main/docs/img/orb-guarantees.png" width="860" alt="ORB token telemetry and anytime-valid alarms"></p>

**The guarantee now covers the deployment, not just the decision.** Split conformal controls the false-alarm rate
of one verdict; a production monitor makes millions of them, and at alpha 0.05 over 10,000 runs the chance of at
least one false alarm is essentially one. `TestMartingale` replaces that with a bound that holds at every stopping
time, including one chosen after looking at the data. On 400-step clean streams at all seven live endpoints there
were no false endpoint alarms; on the two endpoints that genuinely changed under injection it fired after nine
runs, and the changepoint detector after seven. Fleet e-BH picked exactly those two out of seven, with FDR
controlled under arbitrary dependence rather than the independence a real fleet never has.

**A model that complies with an injection is more confident, not less.** Among runs that all contain an injection,
the entropy at the token naming the tool separates compliance from refusal — perfectly on gpt-oss-20B, and in the
same direction on three of the four models that act at all. The fourth reverses it, which is why the monitor
calibrates per endpoint instead of shipping a shared threshold. This signal is contemporaneous with the action
rather than ahead of it; it is useful because the gateway sits between the token and the tool, so a streaming
monitor can abort a call while it is still forming.

<p align="center"><img src="https://raw.githubusercontent.com/deepaksatna/vigil-monitor-assets/main/docs/img/orb-provenance-probe.png" width="720" alt="provenance probe across model families"></p>

**Models do represent who asked. They comply anyway.** On a task balanced for length and content — the same two
sentences, only their placement swapped between the user turn and a retrieved document — a linear probe on the
residual stream reaches AUROC 0.98 to 1.00 in all five families tested, spanning 8B to 32B, with a
shuffled-label control at chance in every one. On this evidence prompt injection is not a perception failure but
a control failure, which is a different problem and a different fix.

**And the warning that goes with it.** Those same probes transfer their *direction* and not their *threshold*:
0.56 to 0.84 false alarm on unseen phrasings against a 0.05 target, versus 0.00 to 0.08 in distribution, in every
family and with no improvement at larger scale. About
twenty unlabelled runs from the target distribution restore it, with detection back at 1.00. Twenty is not tuned:
below nineteen points split conformal cannot certify a 5% rate and correctly refuses to flag anything at all.

<p align="center"><img src="https://raw.githubusercontent.com/deepaksatna/vigil-monitor-assets/main/docs/img/orb-recalibration.png" width="760" alt="recalibration budget across five models"></p>

**Compliance is a cliff, not a slope.** The same demand at six levels of explicitness: only the rung claiming
system authority got through, and softer phrasings failed completely rather than partially. The 120B model
resisted the blatant order more often than a bureaucratic "processing note" framing — suggestive at ten runs per
rung, and pointing the wrong way, since the polite framing is the one an attacker would pick.

Three of the seven endpoints never emitted a tool call at the decision step, writing prose instead. Their zero
compliance rate means untested, not safe.

**A false-positive mechanism the run exposed — now found and fixed.** Every endpoint escalated none of its
clean traffic except the gpt-oss family, which escalated 70% against a 10% target. The cause was a category
error: the attribution gate compared a **per-step** regret against a detector calibrated on **trajectory-mean**
regret. Those are different scales. A single mildly suboptimal step sits well above a trajectory's mean, so
gpt-oss — which re-reads its documents before summarising, `search, read, search, read` — tripped a gate that
Qwen3-8B, going straight to `search, read, summarize`, never touched. The trigger was a working style, not
misbehaviour, and any agent that double-checks its sources would have been punished the same way.

The gate now calibrates a separate conformal detector on per-step regrets. Re-measured on a single H100 with
both models served side by side:

| model | false alarm, in-sample | held-out | detection on runs it acted on |
|---|---|---|---|
| gpt-oss-20B | 0.07 | 0.03 | 1.00 (23/23) |
| Qwen3-8B | 0.00 | 0.00 | 1.00 (32/32) |

Target was 0.10. Both are under it, detection is unchanged, and the style-dependence is gone: the gap between
the re-reading model and the tidy one fell from 70 points to at most 7. The same run also settled which of two
candidate causes was responsible — in-sample and held-out scoring agree, so scoring the calibration set was
never the problem; the scale mismatch was the whole of it.


## Single-H100 follow-up (17 September 2026)

Everything below ran on one H100 PCIe 80 GB serving gpt-oss-20B and Qwen3-8B side by side, at roughly a tenth of
the eight-GPU cost.

### The false-alarm defect is closed, and the first fix was the wrong one

<p align="center"><img src="https://raw.githubusercontent.com/deepaksatna/vigil-monitor-assets/main/docs/img/h100-gate-fix.png" width="860" alt="closing the false-alarm defect"></p>

The cause was a category error, not a threshold needing tuning. The conformal detector is calibrated on
**trajectory-mean** regret and the attribution gate was testing a **per-step** regret against it. Those are
different quantities: per-step regret reaches 0.40 at the 90th percentile while the trajectory-mean threshold
sits at 0.15, so a single mildly suboptimal step cleared it every time. gpt-oss re-reads its documents before
summarising, so it tripped a gate that Qwen3-8B, going straight to the point, never touched. The trigger was a
working style, and any agent that double-checks its sources would have been punished the same way.

The gate now calibrates a separate detector on per-step regrets. Against a 0.10 target, held-out false alarms
are 0.03 for gpt-oss-20B and 0.00 for Qwen3-8B, with detection unchanged at 1.00 on every run either model
acted on. The style-dependence that defined the bug fell from a 70-point gap to at most 7.
`tests/test_gate_step_regret.py` pins it with no GPU.

### Predicting compliance before the model acts

<p align="center"><img src="https://raw.githubusercontent.com/deepaksatna/vigil-monitor-assets/main/docs/img/h100-compliance.png" width="900" alt="predicting compliance"></p>

**Compliance can be predicted before the model acts — by reading the prompt, not the model.** Among injected
runs, with train and test on disjoint phrasings and the demanded action held constant: lexical features of the
context reach AUROC 0.935, the model's own opening tokens reach 0.559, and the contemporaneous decision-token
signal reaches 0.863. The cheap pre-generation check beats the expensive telemetry. Read it with its caveat: the
features count imperative and authority markers and the attack ladder varies along exactly that axis, so the
number is partly circular.

### Images are inputs too, and the first detector missed all of it

<p align="center"><img src="https://raw.githubusercontent.com/deepaksatna/vigil-monitor-assets/main/docs/img/h100-multimodal.png" width="860" alt="image-borne injection"></p>

`vigilai.multimodal` closes the largest coverage gap. The core move is refusing a silent trust promotion: when a
pipeline OCRs an image, the resulting string is usually handed onward as a *tool result*, which this package
trusts more than retrieved content, so an instruction painted into a screenshot arrives with more authority than
the same sentence in a document. `extracted_text` links the extraction to its parent medium, and the existing
taint propagation forces it back down. Also: perceptual-hash recurrence, so the same page arriving in forty
sessions reads as one campaign rather than forty accidents, and cross-modal disagreement.

Against Qwen2.5-VL-7B, instructions rendered onto an archive page were obeyed on 29% of injected pages and 0% of
benign ones, an "urgent" framing succeeded every time while the blatant one mostly failed, and **the model never
once acknowledged seeing an instruction**.

The uncomfortable part: the first cross-modal detector flagged none of it. Agreement was 0.65 on injected pages
against 1.00 on benign, so the signal was there, but the hand-picked threshold of 0.35 sat below both — a
guessed constant in a package whose whole argument is that thresholds must be calibrated. The default is now
measured, and `CrossModalMonitor` calibrates on your own benign pages and states a false-alarm rate. The benign
control is what exposed it; without a control, a 100% flag rate on injected pages looked like success.

**The adaptive adversary made its own attack worse.** Over four rounds an attacker model rewrote its injection
to be more persuasive; it became more polite and more oblique, and compliance fell from 4/8 to 0/8. Every attack
that did work was escalated. Effective non-blatant framings do exist elsewhere in our data, so this shows that
this attacker did not find them, not that they cannot be found.

## Quick start

```python
from vigilai import Monitor, ModelFingerprint, Store, from_spans
from vigilai.adapters import OpenAICompatibleRunner

runner = OpenAICompatibleRunner("https://api.example/v1", "model-name", tools_schema, api_key="…")
m = Monitor(runner=runner, alpha=0.05)
m.store = Store("vigil.db")                                   # persist every verdict and signal
m.calibrate([from_spans(s, tool_meta=meta) for s in known_good_span_sets])
m.attach_fingerprint(ModelFingerprint().fit(baseline_canary_responses))

v = m.assess(from_spans(new_spans, tool_meta=meta), output=answer_text, evidence=retrieved_chunks)
if v.escalate:
    route_to_human(v.severity, v.reasons, v.attribution.top1())
```

```bash
vigil-dashboard --db vigil.db --port 8020      # dashboard at /, Prometheus text at /metrics
```

To enforce rather than observe, put the gateway in front of the model:

```python
from vigilai import AgentPolicy, PolicyGateway, serve_proxy
gw = PolicyGateway({"agentA": AgentPolicy("agentA", {"model-name"}, {"search", "read", "summarize"}, {"read"})}, tool_meta)
serve_proxy(gw, upstream="https://api.example", port=8010)     # agents call http://gateway:8010/v1/chat/completions
```

## Design principles

1. **Evasion-hard quantities first.** Scopes used, secrets moved, budgets spent and actions under intervention must be
   violated to be beaten; text-based signals can merely appear clean.
2. **Instruments vote independently and their disagreement is reported**; each publishes its adversarial robustness
   and the attack that defeats it (`vigilai.ROBUSTNESS`).
3. **Guarantees are explicit.** Conformal calibration on known-good runs; a fingerprint change marks it stale.
4. **Trust is derived from provenance**, never declared upward; reasoning blocks are secrets by default.
5. **Standards in, standards out.** OpenTelemetry gen_ai and OpenInference spans in; decision spans and Prometheus
   metrics out. Vigil sits on top of any tracing platform; it does not replace it.

## Limits, stated plainly

Replay needs a reproducible environment. The regret residual needs a correct feasible menu (it is skipped and
declared when absent). Lexical grounding is gameable by paraphrase (plug in an entailment scorer). The dashboard
and proxy have no authentication and must sit behind an authenticated ingress.

Token telemetry sees a hosted model's uncertainty, not its representations, and an endpoint that declines to
return log-probabilities blinds it entirely. Activation probes are stronger and need weights you own. A probe
ships as a *direction*, never a threshold: measured across five model families from 8B to 32B, a threshold
calibrated on one set of phrasings produced false-alarm rates of 0.56 to 0.87 on new phrasings against a 0.05
target. Use `ProbeMonitor.recalibrate` on the deployment's own clean traffic, which needs no labels; about twenty
runs restores the stated rate, and below the conformal floor it refuses to certify rather than guess.

The media instruments establish provenance, promotion, recurrence and disagreement. **They do not decide whether
the pixels are benign**, and no external monitor can. Audio is handled by construction but has never been run
against a speech model. Adversarial images that carry no rendered text are out of scope entirely.

A trigger that only the model can recognise is undetectable before it fires, by any external monitor, and
nothing here changes that. Neither does anything here measure intent: the ledger can prove an action exceeded
its authority, but no outside measurement establishes that it was meant to.

## Layout

`vigilai/` package · `tests/` 81 tests · `experiments/` live experiments, node setup, plots, demo dashboard ·
`experiments/results/` dated JSON and logs · `docs/img/` figures and logo.

## Licence and community

MIT. Built as open research for the AI community; features from the wider observability field were adopted with
thanks rather than compared against. The package is distributed through PyPI; the figures are hosted in a public
assets repository. Feedback via the PyPI maintainer contact.
