Metadata-Version: 2.4
Name: contextops
Version: 0.3.4
Summary: Deterministic context linter for LLM applications — analyze, score, and optimize your LLM context payloads.
Author: Abhijeet Baug
License: Sustainable Use License
        
        Copyright (c) 2026 Abhijeet Baug
        
        Acceptance
        
        By using the software, you agree to all of the terms and conditions below.
        
        Copyright License
        
        The copyright holder (Abhijeet Baug, "Licensor") grants you a non-exclusive,
        royalty-free, worldwide, non-sublicensable, non-transferable license to use,
        copy, distribute, and create derivative works of the software, in each case
        subject to the limitations set out below.
        
        Limitations
        
        1. Internal / Non-Commercial Use Only. You may use or modify the software
           only for your own internal business operations, personal use, or
           non-commercial purposes.
        
        2. No Resale or Commercial Hosting. You may not provide the software, or
           any derivative work of the software, to third parties as a hosted or
           managed service, or as part of a commercial product or service offering,
           where the core value delivered to those third parties is substantially
           derived from the software's functionality.
        
        3. No Charge for Distribution. You may distribute or make the software
           (or a derivative work) available to others only if you do so free of
           charge, and only for non-commercial purposes.
        
        4. No Removal of Notices. You must not remove or obscure any licensing,
           copyright, or other notices contained in the software.
        
        Contributions
        
        Any contribution intentionally submitted by you for inclusion in the
        software shall be under the terms and conditions of this license, without
        any additional terms or conditions, unless explicitly agreed otherwise in
        writing with the Licensor.
        
        Patents
        
        This license does not grant you any right in any patent held by the
        Licensor. No patent license is granted by implication, estoppel, or
        otherwise.
        
        Notices
        
        You must ensure that anyone who gets a copy of any part of the software
        from you also gets a copy of these terms.
        
        No Other Rights
        
        These terms do not imply any licenses other than those expressly granted
        in this license.
        
        Termination
        
        If you use the software in violation of these terms, such use is not
        licensed, and your license will automatically terminate. If the Licensor
        provides you with a notice of your violation, and you cease all violation
        of this license no later than 30 days after you receive that notice,
        your license will be reinstated retroactively. However, if you violate
        these terms after such reinstatement, any additional violation of these
        terms will cause your license to terminate automatically and permanently.
        
        No Liability
        
        As far as the law allows, the software comes "as is," without any
        warranty or condition, and the Licensor will not be liable to you for any
        damages arising out of these terms or the use or nature of the software,
        under any kind of legal claim.
        
        Definitions
        
        "Software" means the present software and any associated documentation
        made available by the Licensor under these terms, as may be updated by
        the Licensor from time to time.
        
        "You" means the individual or entity agreeing to these terms.
        
        "Licensor" means Abhijeet Baug.
        
Project-URL: Homepage, https://github.com/Abhijeet777ui/contextops
Project-URL: Repository, https://github.com/Abhijeet777ui/contextops
Project-URL: Issues, https://github.com/Abhijeet777ui/contextops/issues
Project-URL: Documentation, https://github.com/Abhijeet777ui/contextops#readme
Project-URL: Changelog, https://github.com/Abhijeet777ui/contextops/blob/main/CHANGELOG.md
Keywords: llm,context,observability,rag,token,optimization,linter,ci,deterministic,prompt-engineering
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: tiktoken>=0.5.0
Requires-Dist: click>=8.0.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Dynamic: license-file

<div align="center">
  <img src="./pic/logo_gif.gif" alt="ContextOps logo" width="180" />

  # ContextOps

  **Deterministic static analysis for LLM context. Catch token waste, redundancy, and structural bloat *before* you pay for inference.**

  [![PyPI Downloads](https://static.pepy.tech/personalized-badge/contextops?period=total&units=INTERNATIONAL_SYSTEM&left_color=BLACK&right_color=GREEN&left_text=downloads)](https://pepy.tech/projects/contextops)
  [![PyPI](https://img.shields.io/pypi/v/contextops.svg)](https://pypi.org/project/contextops/)
  [![Python](https://img.shields.io/pypi/pyversions/contextops.svg)](https://pypi.org/project/contextops/)
  [![License](https://img.shields.io/badge/license-Sustainable%20Use-blue.svg)](./LICENSE)
  [![Stability](https://img.shields.io/badge/stability-contract-brightgreen)](./STABILITY.md)

</div>

## What is ContextOps?

ContextOps is a **deterministic, embedding-free linter for the context you send to an LLM**. Feed it the exact payload you're about to hand to the model (an OpenAI message list, a RAG bundle, or a plain string) and it returns, *before inference*:

- a **Context Health Score (CHS)** from **0–100**, computed from four structural penalties
- a **per-dimension breakdown** explaining *why* the score is what it is
- **findings** that point at the specific offending chunks
- **actionable recommendations** with estimated token savings

No LLM calls. No embeddings. No API keys. No network. The same input always produces the same output, which is what makes it safe to wire into CI.

> **Think of it as ESLint for LLM context.**

### The one-line mental model

```
Score = 100 − (Redundancy + Density + Structure + Concentration)
```

Each penalty is capped, so no single problem can tank the score on its own:

| Dimension | Max Penalty | What It Detects |
|-----------|-------------|-----------------|
| **Redundancy** | 30 pts | Lexically duplicated or near-duplicated context items |
| **Density** | 30 pts | Token waste from formatting, whitespace, and repetitive boilerplate |
| **Structure** | 20 pts | Imbalance between context types (retrieval flooding, system-prompt bloat, memory explosion, tool sprawl) |
| **Concentration** | 20 pts | Over-reliance on a single document or source |

### What you actually get

`contextops inspect` prints a real, readable report, not a wall of numbers:

```text
+==================================================+
|          CONTEXTOPS | Context Analysis            |
+==================================================+

  CI Gate Status:  WARN

  Context Score:  62 / 100  (NEEDS WORK)

  Score Breakdown:

    Redundancy     -18.5 / 30  ############
    Density        -4.2  / 30  ##
    Structure      -3.3  / 20  ###
    Concentration  -12.0 / 20  ########

    Total Penalty  -38.0 / 100

  Token Breakdown:
    Total tokens:        2,150
    Wasted tokens:       842
    Potential reduction: 39.2%

  Findings:
    [!]  REDUNDANT_CONTEXT (Likely): refund_policy.md chunks are ~87% similar
    [S]  Retrieval dominance: retrieval is 78% of context

  Recommendations:
    1. Redundant context: refund_policy.md appears twice
       Impact: +15.2 points | Save: 342 tokens
       Fix: Remove the duplicate item → save 342 tokens
```

*(Add `--roast` for brutally honest commentary, or `--json-output` for CI parsing.)*

### See it in action

<div align="center">
  <video src="https://contextops.vercel.app/demo.mp4" width="720" controls muted loop playsinline></video>
</div>

## Why ContextOps exists

Traditional software engineering has deterministic quality gates: compilers catch syntax errors, linters catch code smells, formatters enforce consistency. **LLM applications rarely have an equivalent layer for the context they stuff into a model.**

Instead, prompts quietly grow over time with:

- **Duplicated retrieval chunks**
- **Bloated system prompts**
- **Runaway conversation history**
- **Excessive tool output**
- **Hidden token waste**

These increase latency and cost, make model behavior less predictable, and often go unnoticed until production. ContextOps makes context quality **observable, measurable, and testable before inference**.

| Without ContextOps | With ContextOps |
|--------------------|-----------------|
| Silent context degradation | Observable context health |
| Wasted tokens = wasted $ | Token optimization guidance |
| Unpredictable model behavior | Deterministic quality gates |
| No CI integration for context | `contextops check` in GitHub Actions |
| Guesswork debugging | Root-cause diagnostics |
| No historical tracking | Local telemetry + trends |

### Who it's for

The one-line test: **do you write the code that assembles the `messages` array sent to an LLM API?** If yes, this is for you.

| Audience | How you'd use it |
|----------|------------------|
| **LLM app developers** (RAG, agents, chatbots) | `pip install contextops` → analyze your pipeline's payload before the API call |
| **LangChain / LlamaIndex / AutoGen users** | Attach the callback, just 2 lines, see below |
| **Prompt engineers** | A/B test template variations with `contextops diff` |
| **MLOps & Platform Teams** | `contextops check` as a CI/CD quality gate |
| **Researchers** | Deterministic scoring + ContextBench |

### Who it's *not* for (currently)

ContextOps analyzes the payload **you** assemble. If a closed tool assembles the context internally, there's nothing to intercept:

| User type | Why not |
|-----------|---------|
| ChatGPT / Claude.ai web users | No API access to the payload |
| Claude Code / Codex users | Closed binary, no context export |
| Cursor / Windsurf end-users | No native payload export |

Analogy: ESLint is useless to people *browsing* websites. It's for people *writing* JavaScript. ContextOps is for people *writing* LLM calls.

### When to use it

| When | What to run | Why |
|------|-------------|-----|
| **During development** | `contextops inspect context.json` | Catch bloat and duplication before you ship |
| **Before merging / in CI** | `contextops check context.json --min-score 75` | Exit-code gate that blocks context regressions like a linter |
| **Debugging production** | Export a bad trace from LangSmith/Helicone → `inspect` it | Root-cause weird model behavior structurally |
| **Comparing changes** | `contextops diff before.json after.json` | Prove a prompt/retrieval refactor actually improved the context |
| **Ongoing** | `contextops telemetry trends` | Track context health over time |

## How it works

ContextOps sits between your prompt/retrieval logic and the LLM API call. It statically analyzes the payload and scores its structural health.

<div align="center">
  <img src="./pic/flow.png" alt="ContextOps flow diagram" width="720" />
</div>

1. **Normalize:** accept OpenAI message lists, structured dicts (`system` / `messages` / `chunks` / `memory` / `tools`), or a plain string.
2. **Tokenize:** count tokens with `tiktoken` using the target model's encoding.
3. **Analyze:** run four independent analyzers: redundancy, density, structure, and concentration.
4. **Score:** combine the four capped penalties into a single 0–100 score.
5. **Recommend & gate:** emit actionable fixes, and (in `check` mode) pass or fail with an exit code.

```text
Retrieval / RAG ─┐
Conversation ────┼─► Context Payload ─► ContextOps ─► Score + Findings
Tools output ────┘                          │
                                            ├─► Pass ─► LLM Inference
                                            └─► Fail ─► CI block / alert
```

## What ContextOps does *not* do

It's a **structural** analyzer, by design. It intentionally does **not** evaluate:

- Prompt-engineering quality or intent
- Reasoning ability or hallucinations
- Factual correctness
- Retrieval relevance (semantic meaning: same idea, different words)

A high score means your context is *structurally clean*, not that the model will answer correctly.

## Quick start

```bash
pip install contextops
```

### 1. See it in action (zero setup)

```bash
contextops demo
```

Runs a pre-packaged, deliberately messy RAG context so you can see the full report and the roast.

### 2. Capture your context

Save the payload right before your API call:

```python
import json
import openai

messages = [
    {"role": "system", "content": "You are a helpful customer support bot."},
    {"role": "user", "content": "How long will my refund take?"},
]

# Add two lines before your API call
with open("context.json", "w") as f:
    json.dump(messages, f, indent=2)

response = openai.chat.completions.create(model="gpt-4o", messages=messages)
```

You can also export a bad trace from **LangSmith / Helicone / Datadog** and point ContextOps at the saved JSON.

### 3. Inspect it

```bash
contextops inspect context.json
```

*(Use `--roast` for brutally honest diagnostics, `--explain` to see the top score drivers.)*

### 4. Enforce it in CI

```bash
contextops check context.json --min-score 75
```

Exits `0` on pass, `1` on fail. Drop it straight into GitHub Actions (see [`.github/workflows/contextops.yml`](./.github/workflows/contextops.yml) for the pattern used by this repo).

## CLI reference

| Command | What it does |
|---------|--------------|
| `contextops demo` | Run the built-in demo context (roast on by default) |
| `contextops inspect <file>` | Analyze a context file and print a rich report |
| `contextops check <file> --min-score N` | CI gate — exit `0` (pass) / `1` (fail) / `2` (bad input) |
| `contextops diff <a> <b>` | Compare two snapshots to detect regressions |
| `contextops stability [file]` | Verify the determinism guarantee |
| `contextops telemetry status` · `log` · `trends` | Local-only score history and trends |
| `contextops badge [--score N]` | Generate a shields.io badge for your score |

Common flags on `inspect` / `check`:

- `--json-output` — machine-readable JSON
- `--model <name>` — model for token counting (default `gpt-4o`)
- `--profile <name>` — archetype: `general`, `rag`, `agent`, `chatbot`, `toolchain`
- `--explain` — show *why* each penalty fired
- `--roast` — score-band commentary
- `--config <path>` — JSON config with custom thresholds

For the full flag reference, see [USER_GUIDE.md](./USER_GUIDE.md).

## Python API

```python
from contextops.api.inspect import inspect_context

result = inspect_context(
    payload,          # dict, list, or plain string
    model="gpt-4o",   # model for token counting
    archetype="rag",  # archetype profile (optional)
)

print(f"Score: {result.score} / 100")
print(f"Wasted tokens: {result.token_breakdown.wasted_tokens}")
for rec in result.recommendations:
    print(f"  -> {rec.fix}")
```

Also available: `diff_contexts(a, b)` and `run_stability_report(payload)` from `contextops.api`, and `ContextOpsConfig` for custom thresholds.

## LangChain integration

```python
from contextops import ContextOps

# Log the score (default — non-blocking)
chain = chain.with_config({"callbacks": [ContextOps.auto()]})

# Block execution if context quality is too low
chain = chain.with_config(
    ContextOps.langchain_config(mode="block", min_score=75, profile="rag")
)
```

`ContextOps.auto()` returns a LangChain callback handler with three modes:

| Mode | Behavior |
|------|----------|
| `log` | Print the score report (default) |
| `warn` | Emit a warning below `min_score` |
| `block` | Raise `ContextOpsScoreError` below `min_score` — the LLM call never happens |

A full end-to-end RAG demo lives in [`examples/quickstart_rag.py`](./examples/quickstart_rag.py).

## Archetype profiles

Archetypes adjust structural thresholds for your use case. The global 0–100 score is never affected — only which warnings fire.

| Profile | When to use | Retrieval | System | Memory | Tool |
|---------|-------------|-----------|--------|--------|------|
| `general` | Default — mixed use cases | 70% | 50% | 50% | 60% |
| `rag` | Pure document retrieval | **95%** | 40% | 20% | 30% |
| `agent` | Autonomous agents with tool loops | 50% | 40% | 40% | **90%** |
| `chatbot` | Conversational apps with large history | 40% | 50% | **85%** | 30% |
| `toolchain` | Multi-tool pipelines | 50% | 40% | 30% | **95%** |

---

## ContextBench: the benchmark

[ContextBench](./ContextBench) is a **1,500-sample benchmark for evaluating the structural integrity of context windows**, the first benchmark of its kind.

### Why it exists

Every existing LLM benchmark measures what the model *does*: answer correctness, reasoning, hallucination rates, retrieval accuracy. None of them measure what the model is *given*. Yet modern LLM systems increasingly fail before inference even begins: 50 redundant documents retrieved when 5 would do, multi-agent pipelines passing bloated internal state, uncompressed tool output consuming thousands of wasted tokens, stale memory quietly corrupting a long-running session.

ContextBench evaluates that layer. The question is not "did the model answer correctly?" but "was the context window constructed efficiently and robustly before inference?"

### Evaluator-agnostic by design

This is the critical design decision: **ContextBench is not coupled to ContextOps.** Each sample carries ground-truth `expected_properties`, for example:

```json
"ground_truth": {
  "failure_modes": ["system_prompt_bloat"],
  "expected_properties": {
    "contains_redundancy": false,
    "contains_density_bloat": true,
    "contains_structure_imbalance": true,
    "contains_source_concentration": false
  }
}
```

Any evaluator can be scored against these labels objectively: flagging redundancy where `contains_redundancy` is `false` is a false positive, missing a declared failure mode is a false negative. That makes the benchmark fair ground for static analyzers, heuristic scorers, observability platforms, LLM-as-a-judge pipelines, and academic frameworks alike, with no built-in bias toward ContextOps' own scoring engine. ContextOps is *validated against* ContextBench, but the benchmark stands on its own: anyone can build a competing analyzer and evaluate it on the same dataset.

### What's in v1

Five categories, 300 samples each, built on domain-grounded enterprise data (legal contracts, stack traces, SQL dumps, clinical notes, Jira histories...) rather than synthetic filler:

| Category | What it tests |
|----------|---------------|
| **Optimal Architectures** | Healthy baselines, so evaluators must also avoid false positives, not just find broken contexts |
| **Structural Failures** | System-prompt bloat, retrieval flooding, memory explosion, tool-output overflow |
| **Redundancy Failures** | Near-duplicate clusters, boilerplate explosion, repeated memory loops |
| **Agent Architecture Failures** | Multi-agent context explosion, recursive planning loops, uncompressed state passthrough |
| **Temporal Context Drift** | Stale memory injection, retrieval drift, summary compression degradation |

The adversarial companion, [ContextSecBench](./ContextSecBench), adds 9,500 attack payloads: prompt-injection hiding, truncation smuggling, semantic denial-of-service, and context poisoning.

### Reproducible and versioned

Samples are synthesized by a generation engine rather than hand-curated, and every sample records mutation metadata describing exactly how it was built. Released versions are immutable (`ContextBench_v1`, then `v2`, and so on), so published scores stay comparable over time.

### Why it matters

A benchmark is what turns an idea into a discipline. Without one, "context optimization" is anecdote; with one, every optimizer, compressor, and analyzer becomes comparable on the same ground, regressions become detectable, and research becomes reproducible. As agent systems grow persistent and long-running, context construction quality, not model size, is increasingly what determines system performance. ContextBench is built to be the evaluation standard for that layer.

**[View the Leaderboard →](https://Abhijeet777ui.github.io/contextops/)**

## Requirements & guarantees

- **Python 3.10+**, with exactly two runtime dependencies: [`tiktoken`](https://pypi.org/project/tiktoken/) (token counting) and [`click`](https://pypi.org/project/click/) (CLI).
- **Deterministic:** same input, same score, every time. Enforced by a chaos test suite on every commit.
- **Fast:** `< 2s` for 5k tokens, `< 5s` for 20k, `< 10s` for 50k.
- **Local-only telemetry:** opt in with `CONTEXTOPS_TELEMETRY=1`; scores are written to `~/.contextops/telemetry.jsonl` and never leave your machine.

The formal versioning, schema, and determinism contract is in [STABILITY.md](./STABILITY.md).

## Documentation

| Document | Contents |
|----------|----------|
| [HIGH_LEVEL_DOC.md](./HIGH_LEVEL_DOC.md) | Full technical reference — scoring engine, all flags, API, ContextBench |
| [USER_GUIDE.md](./USER_GUIDE.md) | Getting-started guide with practical examples |
| [STABILITY.md](./STABILITY.md) | Stability contract, versioning policy, schema guarantees |
| [CONTRIBUTING.md](./CONTRIBUTING.md) | Setup, project structure, PR checklist |

## Contributing & license

Contributions welcome. Read [CONTRIBUTING.md](./CONTRIBUTING.md) first, and open an issue before non-trivial changes.

- **Website:** [contextops.vercel.app](https://contextops.vercel.app)
- **Repository:** [github.com/Abhijeet777ui/contextops](https://github.com/Abhijeet777ui/contextops)
- **License:** [Sustainable Use License](./LICENSE)
