Metadata-Version: 2.3
Name: anchor-eval
Version: 0.2.2
Summary: Retrieval evaluation harness for technical corpora — span-anchored ground truth, archetype-conditioned diagnosis.
Author: Prasant Mishra
Author-email: Prasant Mishra <mishraprasant73@gmail.com>
License: BSL-1.1
Requires-Dist: pydantic>=2.7
Requires-Dist: typer>=0.12
Requires-Dist: rich>=13.7
Requires-Dist: pynacl>=1.5
Requires-Dist: opensearch-py>=2.4
Requires-Dist: jsonpath-ng>=1.8.0
Requires-Dist: tree-sitter>=0.22 ; extra == 'ast'
Requires-Dist: tree-sitter-python>=0.23 ; extra == 'ast'
Requires-Dist: tree-sitter-typescript>=0.23 ; extra == 'ast'
Requires-Dist: tree-sitter-go>=0.23 ; extra == 'ast'
Requires-Dist: tree-sitter-rust>=0.23 ; extra == 'ast'
Requires-Dist: pypdf>=3.0 ; extra == 'docs'
Requires-Dist: python-docx>=1.0 ; extra == 'docs'
Requires-Python: >=3.12
Project-URL: Homepage, https://github.com/pmishra73/anchor-eval
Project-URL: Documentation, https://github.com/pmishra73/anchor-eval/tree/main/docs
Project-URL: Repository, https://github.com/pmishra73/anchor-eval
Project-URL: Bug Tracker, https://github.com/pmishra73/anchor-eval/issues
Project-URL: License, https://github.com/pmishra73/anchor-eval/blob/main/LICENSE
Project-URL: Licensing, https://github.com/pmishra73/anchor-eval/blob/main/docs/reference/licensing.md
Project-URL: Changelog, https://github.com/pmishra73/anchor-eval/blob/main/CHANGELOG.md
Provides-Extra: ast
Provides-Extra: docs
Description-Content-Type: text/markdown

# anchor-eval

**Span-anchored retrieval evaluation for any text corpus.**

anchor-eval measures how well your retrieval pipeline (chunking, embedding, reranking) finds
the *exact source spans* that answer questions about your corpus — source code, documentation,
legal contracts, support tickets, or any collection of text files.
Unlike chunk-overlap metrics, span-anchored evaluation catches regressions that look fine
in aggregate but silently break specific retrieval patterns.

## Quickstart

```bash
pip install anchor-eval
# or: uv add anchor-eval
# For PDF and DOCX support:       pip install 'anchor-eval[docs]'
# For AST drift detection:        pip install 'anchor-eval[ast]'

# Scaffold a config, then generate and run CI
anchor init --domain code
anchor generate --corpus ./my-repo --domain code --output question_set.json
anchor ci --question-set question_set.json --corpus ./my-repo --baseline anchor-baseline.json

# Document corpus (Markdown, HTML, plaintext, RST)
anchor generate --corpus ./docs --domain docs --output doc_qs.json
anchor score doc_qs.json --corpus ./docs --chunking-strategy document_structure

# Try locally with Ollama (no API key required)
anchor generate --corpus ./docs --domain docs --llm-provider ollama --model llama3.2:3b

# Try with no setup at all
anchor demo
anchor demo --domain docs
```

`--domain` selects the correct pack automatically. `--pack` overrides it when you want a custom pack.

See [docs/guides/quickstart.md](docs/guides/quickstart.md) for the full walkthrough.

## LLM providers

`anchor generate` supports three providers via `--llm-provider`:

| Provider | Flag | Key required | Notes |
|---|---|---|---|
| `fake` | `--llm-provider fake` | No | Deterministic, instant. Good for CI smoke tests. |
| `openai` | `--llm-provider openai` | `OPENAI_API_KEY` | Default model: `gpt-4o-mini`. Override with `--model`. |
| `ollama` | `--llm-provider ollama` | No | Runs against a local Ollama server. Default model: `qwen2.5-coder:7b`. Override with `--model` and `--llm-base-url`. |

```bash
# Ollama on a remote host
anchor generate --corpus ./docs --domain docs \
  --llm-provider ollama --model llama3.2:3b \
  --llm-base-url http://gpu-host:11434/v1
```

## The problem, explained from scratch

**RAG in one sentence:** Split documents into chunks → embed them → when a user asks a question, find the most similar chunks → give those chunks to an LLM.

The "documents" can be anything — PDFs, web pages, Slack messages, or code files. A `.py` file is a document. A `.go` file is a document. A code repository is just a folder of text files. The pipeline is identical.

**Who builds RAG over text?**

Almost everyone with a knowledge base. A few common shapes:

- **Code corpora** — "Ask our codebase anything": "Where does authentication happen?", "Which function validates JWT tokens?", "What config controls the timeout?" Developer assistants (Cursor, Copilot workspace) and onboarding bots fall here too.
- **Documentation & wikis** — internal runbooks, product docs, API references. Users ask in natural language; retrieval has to find the right section.
- **Legal & compliance** — contracts, policies, regulations. Questions have exact answers buried in specific clauses; a wrong retrieval is a liability.
- **Support & ticketing** — historical tickets, knowledge-base articles. Retrieval drives deflection; a missed span means a human handles it instead.

The retriever's job in all of these: given a natural-language question, find the correct chunk of text that answers it.

**The evaluation problem**

Say you want to test if your retriever is working. The obvious approach:

1. Take a chunk — say, lines 45–90 of `auth.py`
2. Generate a question from it: "How does this codebase validate API keys?"
3. Run retrieval and check: did you get that chunk back?

This works. Until you change your chunking strategy.

If you go from 512-token chunks to 1024-token chunks, `lines 45–90` no longer exists as a chunk — it got merged into `lines 1–120`. Your benchmark just broke. Every question is now tied to a chunk that doesn't exist. You either throw away the benchmark or you never experiment with chunking at all.

Most teams never change chunking because they can't measure the impact. They're flying blind.

**What anchor-eval does differently**

Instead of anchoring a question to a chunk ID, it anchors it to a character span in the *raw source file*:

```json
{
  "question": "How does this codebase validate API keys?",
  "anchor": { "file": "auth.py", "char_start": 1820, "char_end": 2140 }
}
```

`auth.py` characters 1820–2140 exist forever, regardless of how you chunk. When you run retrieval, a chunk is a "hit" if it covers that span. Change your chunking from 512 to 1024 tokens: the span is still there, the check still works, the benchmark still runs.

You can now run two configs side by side and get a real score for each. You know which one is better and *why*.

**The diagnosis**

Questions are tagged by failure type — `cross_file_causality`, `lexical_collision`, `clause_cross_reference`, etc. When one archetype scores 33% and another scores 90%, that pattern names the broken component. anchor-eval says "your chunker is splitting conditional clauses from their consequences" rather than "retrieval is 61% overall." It then proposes the targeted A/Bs that would confirm the cause.

## What are spans?

A *span* is a `(doc_id, char_start, char_end)` triple that points to an exact range of text in a
source document. Question answers in anchor-eval are defined as one or more anchor spans —
the minimal text ranges that contain the answer. Hit-rate@k is computed by checking whether
the retriever's top-k chunks cover those spans at a configurable IOU threshold.

Document spans also carry a `section_id` (e.g. `"doc#section/page:3/p:2"`) for human-readable
location display in reports.

## Commands

| Command | Description |
|---|---|
| `anchor init` | Scaffold an `anchor.json` config with domain-appropriate defaults. |
| `anchor generate` | Generate a graded QuestionSet from a corpus. Requires a license. |
| `anchor score` | Score a QuestionSet against a corpus; print hit-rate@k by archetype. |
| `anchor run` | Run a grid of retrieval configs, checkpointed to SQLite. Safe to interrupt. |
| `anchor resume` | Resume an interrupted `anchor run` from its checkpoint store. |
| `anchor ci` | CI gate: compare current scores against a committed baseline. Exits 0/1/2. |
| `anchor demo` | Run a 5-question demo against a built-in micro corpus. No license required. |
| `anchor license install` | Install a license token to `~/.config/anchor/license.token`. |
| `anchor license verify` | Verify the installed license token and print its status. |

### Grid runs (`anchor run` / `anchor resume`)

`anchor run` executes a matrix of retrieval configs over a QuestionSet and writes a `RunReport`. Each config is evaluated independently so you can compare chunking strategies, retrieval modes, and k-values side by side.

```bash
# Single default config (BM25, fixed-token chunking)
anchor run --question-set qs.json --corpus ./my-repo --output report.json

# Custom grid from a JSON config file
anchor run --question-set qs.json --corpus . --grid-config grid.json --output report.json

# Estimate cost without running
anchor run --question-set qs.json --corpus . --estimate-only

# Resume after Ctrl-C
anchor resume --question-set qs.json --corpus . --store run.db --output report.json
```

The checkpoint store (`run.db`) is a WAL-mode SQLite file. Each RunUnit (question × config) is claimed and completed atomically, so interrupted runs restart from exactly where they left off.

### AST spans and drift detection (`--ast`)

Pass `--ast` to `anchor generate` to enrich spans with tree-sitter AST node IDs. This enables structural drift detection: if a function is renamed or moved, `anchor drift` detects it and flags the affected questions before a CI run wastes time on stale anchors.

```bash
# Requires: pip install 'anchor-eval[ast]'  and a 'generate:ast' license entitlement
anchor generate --corpus ./my-repo --domain code --ast --output qs.json
```

AST enrichment is off by default because it adds tree-sitter parsing overhead and requires the `ast` optional dependency.

## Key concepts

- **QuestionSet**: A committed JSON file of verified questions with anchor spans and difficulty scores.
- **Archetype**: A failure pattern category tagged on each question. Regressions appear per-archetype, not just in aggregate.
- **Pack**: A JSON bundle of archetype definitions and generator prompts for a specific domain.
- **`anchor ci`**: The CI gate. Compares current scores against a committed baseline JSON, exits 1 on per-archetype regression, exits 2 on corpus drift.
- **`anchor run`**: A checkpoint-based grid runner. Evaluates multiple retrieval configs in one pass and writes a structured RunReport.
- **RunStore**: SQLite-backed checkpoint store. Each unit (question × config) is claimed atomically so concurrent or resumed runs are safe.

## Built-in packs

| Pack | Domain | Archetypes |
|---|---|---|
| `code-oss-v0` | Python, TS, Go, Rust | `identifier_free_intent`, `cross_file_causality`, `negative_existence`, `shadow_identifier`, `lexical_collision`, `parameter_level` |
| `jvm-oss-v0` | Java, Kotlin | `identifier_free_intent`, `cross_file_causality`, `parameter_level`, `overload_disambiguation`, `annotation_sensitivity`, `generic_type_boundary` |
| `systems-oss-v0` | Go, Rust | `identifier_free_intent`, `cross_file_causality`, `ownership_borrow_context`, `unsafe_block_scope`, `trait_impl_dispatch`, `cgo_ffi_boundary` |
| `docs-general-v0` | Markdown, HTML, RST, plaintext | `section_cross_reference`, `implicit_negative`, `term_definition_lookup`, `conditional_answer`, `table_cell_lookup`, `procedural_step` |
| `docs-legal-v0` | Legal contracts, regulations | `clause_cross_reference`, `effective_date_shadowed`, `defined_term_collision`, `implicit_obligation`, `negative_obligation`, `jurisdiction_qualifier` |
| `docs-support-v0` | Support tickets, runbooks | `symptom_cause_link`, `workaround_vs_fix`, `version_specific_answer`, `escalation_path`, `product_name_alias` |

Custom packs can be written for any domain — define archetypes, write prompts with `{text_excerpt}` / `{archetype_description}` / `{output_format}`, and point `--pack` at the directory.

## `anchor-corpus.json` manifest

Drop an `anchor-corpus.json` at the root of any corpus directory to configure domain defaults:

```json
{
  "domain": "legal",
  "version": "2024-q4",
  "description": "ACME Corp master service agreement corpus, 2024 edition"
}
```

The loader merges `domain`, `version`, and `description` into every `SourceDocument.metadata` and activates domain-specific chunking (e.g. numbered-section splitting for `"domain": "legal"`).

> **Note (v0.2.2):** only the three keys above are forwarded into document metadata. Custom fields are ignored to prevent unintended data leakage.

## Supported file types (v0.2.2)

| Extension | Content type | Notes |
|---|---|---|
| `.py` | `code_python` | |
| `.ts`, `.tsx` | `code_typescript` | |
| `.js`, `.jsx`, `.mjs` | `code_javascript` | |
| `.java` | `code_java` | |
| `.kt`, `.kts` | `code_kotlin` | |
| `.cs` | `code_csharp` | |
| `.go` | `code_golang` | |
| `.rs` | `code_rust` | |
| `.md` | `docs_markdown` | |
| `.txt`, `.log` | `docs_plaintext` | |
| `.html`, `.htm` | `docs_html` | Tags stripped; plain text stored |
| `.rst` | `docs_rst` | |
| `.pdf` | `docs_pdf` | Requires `anchor-eval[docs]` |
| `.docx` | `docs_docx` | Requires `anchor-eval[docs]` |
| `.yaml`, `.yml`, `.json` | `docs_openapi` | |

## Changelog

### v0.2.2

Security:
- `anchor generate --ast` now uses full ed25519 verification for the `generate:ast` entitlement — previously the second gate fell into placeholder-key mode and accepted any non-empty token
- License server: email plain-text body strips newlines from the customer org name, preventing RFC 2822 header injection
- License server: `ANCHOR_SIGNING_KEY_HEX` is validated to be exactly 64 hex characters before being passed to nacl

Performance:
- Dense retrieval (`_cosine_top_k`) uses `numpy.dot` when available — ~300–1000× faster than the previous Python loop for real embedding dimensions (768–3072)
- `UniquenessVerifier` default `top_k` raised from 5 → 20, catching ambiguous questions in larger corpora that were previously missed

Correctness:
- `EmbeddingCache.get_or_compute` is now thread-safe (double-checked lock pattern)
- `engine_version` in generated `QuestionSet` files now reflects the actual installed package version, fixing false engine-mismatch exits from `anchor ci`
- OpenSearch `_execute_search` skips malformed hits (missing `_source` fields) with a warning instead of crashing the scoring run
- PDF and DOCX extraction failures (corrupt files, bad ZIP, etc.) are now logged and skipped rather than aborting the entire corpus load
- `LocalTelemetryEmitter` handles read-only filesystems gracefully; telemetry file rotates at 10 MB

Configuration:
- `OpenSearchIndexBackend` accepts `http_auth`, `use_ssl`, and `verify_certs` constructor parameters, with automatic fallback to `OPENSEARCH_USERNAME`/`OPENSEARCH_PASSWORD` environment variables

**Breaking change — `anchor-corpus.json` metadata forwarding:**
Custom string fields in `anchor-corpus.json` beyond `domain`, `version`, and `description` are no longer forwarded into `SourceDocument.metadata`. If you relied on a custom field (e.g. `"project"`) appearing in document metadata, add it under one of the three allowlisted keys or access it from the manifest directly.

### v0.2.1

- Ollama LLM provider (`--llm-provider ollama`)
- Tree-sitter AST adapters for Python, TypeScript, Go, Rust (`--ast`)
- PDF and DOCX corpus support (`anchor-eval[docs]`)
- Production ed25519 license key deployed; Svix replay protection via SQLite
- HTML report XSS protection

### v0.2.0

- `anchor run` / `anchor resume` grid runner with SQLite checkpoint store
- `anchor drift` semantic drift detection
- `anchor-corpus.json` manifest
- JVM pack (`jvm-oss-v0`) and Systems pack (`systems-oss-v0`)
- Docs packs: `docs-general-v0`, `docs-legal-v0`, `docs-support-v0`

## Roadmap

### v0.3 — Shard-aware generation (blocked)

v0.3 introduces corpus partitioning so large repos (50k+ files) can generate question sets
in parallel shards and merge them with a `CrossShardUniquenessFilter`. Also includes
`anchor merge-sets` to combine question sets from different corpus subdirectories.

**Why it's blocked:** the filter makes a precision tradeoff — it will discard some genuinely
unique questions because BM25 similarity across shards is noisier than within-shard checks.
The acceptable false-positive rate is unknown until validated on a real large-scale corpus.

**Gate:** a public 50k-file corpus benchmark must confirm `CrossShardUniquenessFilter`
precision before this ships.

## License

anchor-eval is licensed under the [Business Source License 1.1](https://github.com/pmishra73/anchor-eval/blob/main/LICENSE).
Scoring, CI, and question sets always work without a license.
A license is required only for generating new question sets.

**Pricing:** $49 / user / month · $149 / user / year — [see licensing details](https://github.com/pmishra73/anchor-eval/blob/main/docs/reference/licensing.md).
