Metadata-Version: 2.4
Name: contextscope
Version: 0.1.0
Summary: Attention-level diagnostics for LLM context windows.
License: MIT
Project-URL: Homepage, https://github.com/luoojason/contextscope
Project-URL: Repository, https://github.com/luoojason/contextscope
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: hf
Requires-Dist: torch>=2.1; extra == "hf"
Requires-Dist: transformers>=4.46; extra == "hf"
Provides-Extra: openai
Requires-Dist: openai>=1.40; extra == "openai"
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.34; extra == "anthropic"
Provides-Extra: all
Requires-Dist: contextscope[anthropic,hf,openai]; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-mock; extra == "dev"
Requires-Dist: respx>=0.22; extra == "dev"
Requires-Dist: ruff>=0.5; extra == "dev"
Requires-Dist: httpx>=0.27; extra == "dev"
Dynamic: license-file

# contextscope

**Attention-level diagnostics for LLM context windows.**

Long-context models degrade silently past ~100K tokens. Langfuse, Galileo, and Braintrust track output-level metrics — cost, latency, quality scores. None expose which parts of the context the model actually attended to. contextscope fills that gap.

```python
import contextscope
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-0.5B-Instruct", attn_implementation="eager"
)
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")

probe = contextscope.wrap(model, tokenizer=tokenizer)
result = probe.generate(messages=[{"role": "user", "content": "..."}])

print(result.diagnostics.attention_by_segment)
# {"system": 0.03, "early": 0.08, "middle": 0.42, "recent": 0.47}
```

## What it detects

Three context-degradation patterns, each with a score from 0 (healthy) to 1 (degraded):

| Pattern | What it means |
|---------|--------------|
| `lost_system_prompt` | System prompt receives less attention than its uniform share |
| `recency_bias` | Last 10% of context absorbs disproportionate attention |
| `ignored_early_context` | First 10% of context receives near-zero attention |

## Install

```bash
# Core — zero runtime dependencies
pip install contextscope

# HuggingFace adapter (real attention probing)
pip install contextscope[hf]

# OpenAI adapter (logprobs proxy)
pip install contextscope[openai]

# Anthropic adapter (perturbation probe)
pip install contextscope[anthropic]

# All providers
pip install contextscope[all]
```

## Three tiers, one interface

### Tier 1 — HuggingFace (real attention)

Full mechanistic probing via `output_attentions=True`. The model **must** be loaded with `attn_implementation="eager"` — SDPA and Flash Attention backends silently return `None` for attention weights. contextscope raises immediately if a non-eager model is passed rather than silently producing empty diagnostics.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import contextscope

model = AutoModelForCausalLM.from_pretrained("...", attn_implementation="eager")
tokenizer = AutoTokenizer.from_pretrained("...")

probe = contextscope.wrap(model, tokenizer=tokenizer)
result = probe.generate(messages=[...], max_new_tokens=200)
print(result.diagnostics)
```

Power-user knobs for attention aggregation:

```python
# Use only last 4 layers and first 8 heads (e.g. for retrieval-focused analysis)
result = probe.generate(messages=[...], layers=[-4, -3, -2, -1], heads=list(range(8)))
```

### Tier 2 — OpenAI (logprobs proxy)

Real attention is not available for closed models. contextscope computes Shannon entropy per output token from `top_logprobs` and returns entropy-based heuristics for recency bias and ignored early context.

**Limitations**: `lost_system_prompt` is always `None` (not detectable from output-side logprobs). Degradation scores for the other two patterns are entropy heuristics — always `severity="info"`. For definitive attention-level analysis, use the HF adapter.

```python
from openai import OpenAI
import contextscope

client = contextscope.wrap(OpenAI())
response = client.chat.completions.create(model="gpt-4o-mini", messages=[...])
print(response.contextscope)  # Diagnostics attached to the response
```

### Tier 3 — Anthropic (perturbation probe, opt-in)

Anthropic exposes no logprobs or attention weights. contextscope re-runs the call with three modified contexts and measures output divergence.

**Cost**: 3 extra API calls per diagnosis (~$0.10–0.30 at Sonnet rates, 10–30s extra latency). Opt-in via `perturbation_probe=True`.

```python
import anthropic, contextscope

client = contextscope.wrap(anthropic.Anthropic(), perturbation_probe=True)
response = client.messages.create(model="claude-sonnet-4-6", messages=[...])
print(response.contextscope)
```

Without `perturbation_probe=True`, the adapter returns minimal provenance diagnostics with all scores at 0.

## Diagnostics schema

```python
@dataclass(frozen=True)
class Diagnostics:
    lost_system_prompt: float | None   # None if no system prompt or not detectable
    recency_bias: float                # 0-1
    ignored_early_context: float       # 0-1

    attention_by_segment: dict[str, float] | None  # HF only
    output_entropy: list[float] | None              # HF + OpenAI

    warnings: list[Warning]            # structured, filterable by .code and .severity
    provider: str                      # "huggingface" | "openai" | "anthropic"
    method: str                        # "attention" | "logprobs" | "perturbation"
    context_token_count: int
    model_name: str

@dataclass(frozen=True)
class Warning:
    code: str      # "LOST_SYSTEM_PROMPT" | "RECENCY_BIAS" | "IGNORED_EARLY_CONTEXT"
    severity: str  # "info" | "warn" | "high"
    message: str
    evidence: dict[str, float]
```

## Thresholds

Warning thresholds are named constants in `contextscope.config.Thresholds`. Defaults are calibration-in-progress — override them per-call:

```python
from contextscope.config import Thresholds

thresholds = Thresholds(
    lost_system_prompt_warn=0.3,
    lost_system_prompt_high=0.7,
)
probe = contextscope.wrap(model, tokenizer=tokenizer, thresholds=thresholds)
```

## Development

```bash
git clone https://github.com/luoojason/contextscope
cd contextscope
pip install -e ".[all,dev]"
pytest -m "not slow"      # fast tests (mocked, no network)
pytest -m slow            # slow tests (downloads tiny HF model)
ruff check src/ tests/
```

## Status

v0.1.0 — calibration in progress. Threshold defaults will be updated as benchmark data accumulates (planned: MMLongBench-Doc suite).

Roadmap:
- Calibrated thresholds from benchmark suite
- CLI (`contextscope diagnose`)
- PyPI publish after TestPyPI dry-run
