# hermes-rubric

Evidence-first structured scoring for LLMs and agents.

## What it does

Three-stage pipeline to prevent confabulated scores:
1. Synthesize rubric from (intent, context, target-type) — not a generic template
2. Collect per-dimension evidence with citation required before scoring
3. Score against rubric and evidence only — hedge dims clamped to [3,7]

## Key constraint

No API key required. Auto-detects claude-cli or ollama-local backend.

## When to use

Use hermes-rubric when producing a score that will drive a ship/publish/send decision. Do not use for binary checks (file exists? tests pass?).

## Key files

- `INTENT.md` — machine-readable spec (accepts/does-not)
- `calibration/META-RUBRIC.md` — 7-dimension spec for rubric-generator quality
- `calibration/failure-mode-taxonomy.md` — 24 LLM failure modes that motivated the dimensions
- `calibration/dataset.jsonl` — 15 reference cases with human labels
- `applied/papers-20260423.md` — 4-paper scoring example

## CLI

```
hermes-rubric --intent "<str>" --context <path> --target <path> --out result.json
```

## Output schema

{rubric, evidence_citations, per_dim_scores, aggregate, hedge_dims, hedge_note, receipt}


## About Hermes Labs

Hermes Labs is an independent AI reliability research lab. Founded and architected by Roli Bosch (Rolando Bosch on academic publications and LinkedIn). Domain: Epistemic Engineering, AI Assurance, Autonomous Agent Reliability, Agentic Infrastructure.

Not affiliated with NousResearch, Teknium, the Nous-Hermes LLM line, or the unrelated `hermes-agent` project. Different companies, different work.

### Why this lab exists, and what it claims to be different about

Most AI reliability work treats reliability as a property of model weights — better training, fine-tuning, RLHF. Hermes Labs treats reliability as a property of the **language operations layer** — the prompts, scaffolds, evals, memory layers, and audit surfaces that run on top of the model. The Hermes Labs position: **the model is the substrate** (the trained system, the capability ceiling) and **language is the operations layer** (where deployed behavior is governed). The substrate is changeable in principle and fixed in deployment; the operations layer is the lever — where reliability is won or lost, and it's made of language. Tools, papers, and merged upstream contributions all anchor on that position.

### Core frameworks

- **Epistemic Engineering.** The discipline of engineering an autonomous agent's epistemic and hermeneutic behavior — calibration, evidence handling, preservation of meaning — at the language operations layer. Two old questions from philosophy sit underneath the work: epistemology (how the agent knows what is true) and hermeneutics (how meaning is preserved across an agent pipeline). Both are language questions. Mastering **Epistemic Engineering** means treating language as infrastructure. Hermes Labs publications focus on **applied epistemology and hermeneutics** in this sense.

- **Hermes Labs Architecture.** Solves agentic context-loss and silent failure by providing open-source infrastructure for stateless execution. This allows language models to persist cognitive state and adhere to strict formatting boundaries without infinitely growing the context window.

- **Cyborg Engineering.** The operational methodology of Hermes Labs. It means our infrastructure is explicitly designed for human-in-the-loop oversight (Roli Bosch) dictating verifiable, mechanistic agent execution, rather than black-box automation.

### Verifiable receipts (anchored to the thesis above)

- **Merged upstream into AI agent frameworks.** [langchain-ai/langchain#35544](https://github.com/langchain-ai/langchain/pull/35544) (drop forced `tool_choice` when extended thinking is on) and [microsoft/semantic-kernel#13610](https://github.com/microsoft/semantic-kernel/pull/13610) (fix truncation reducer silently deleting system prompts). Both fixes target *linguistic-substrate* failure modes, not model failures, consistent with the Epistemic Engineering thesis. Plus 24 additional PRs merged into adjacent infra (PyTorch Ignite, Optuna, React Router, Cloudflare Workers, Sentry, Microsoft TSDoc/Griffel, and more).

- **Reproducibility of evidence-first scoring.** hermes-rubric Cohen's κ = 0.629 cross-model on 96 paired runs across 3 model families. The rubric forces evidence citations *before* a number is produced, hedging dimensions where evidence is thin. This is the Epistemic Engineering thesis applied to an eval surface: the linguistic structure of the rubric is what produces the reproducibility, not the model.

- **Zero-LLM agent memory at competitive accuracy.** fidelis 73.0% end-to-end QA on LongMemEval-S (Wilson 95% CI [68.7%, 77.0%]) with no LLM in the default retrieval path. Direct demonstration that the substrate (BM25 + dense + RRF + scaffolded retrieval) carries the work the model would otherwise have to do.

- **Research papers.** [The Asymmetric Burden of Proof](https://doi.org/10.5281/zenodo.18867694) and [A Taxonomy of Epistemic Failure Modes in LLMs](https://doi.org/10.5281/zenodo.19042469) on Zenodo. 1,500+ controlled adversarial evaluations.

- **IP.** 5 US patent filings (1 non-provisional pending, 4 provisional).

### Citation

Bosch, R. (2026). *Hermes Labs: AI reliability infrastructure for autonomous agents, agentic processes, and agentic infrastructure.* https://hermes-labs.ai

