Metadata-Version: 2.4
Name: clinescope
Version: 1.0.1
Summary: A Cline eval harness: scores diff quality + tool-trajectory from a real Cline messages.json trace.
Project-URL: Homepage, https://github.com/minh2416294/clinescope
Project-URL: Repository, https://github.com/minh2416294/clinescope
Project-URL: Issues, https://github.com/minh2416294/clinescope/issues
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: agent,cline,coding-agent,diff,eval,llm,regression
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.11
Provides-Extra: dev
Requires-Dist: mypy; extra == 'dev'
Requires-Dist: pytest; extra == 'dev'
Requires-Dist: pytest-cov; extra == 'dev'
Requires-Dist: ruff; extra == 'dev'
Description-Content-Type: text/markdown

# Clinescope

[![License: Apache-2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg?style=flat-square)](LICENSE)
![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg?style=flat-square)
![Coverage 94%](https://img.shields.io/badge/coverage-94%25-brightgreen.svg?style=flat-square)

Clinescope is an AI evaluation tool that lives in your Cline development workflow, reads your logs, and helps you write better prompts by checking tool choices, catching messy code rewrites, and ensuring updates don't break past work.

Clinescope reads a Cline log and scores four things:

- **`tool_selection`**: did the agent call the tools the task needed?
- **`diff_coherence`**: are its code patches valid and well-formed?
- **`diff_minimality`**: are its edits small and focused, not bloated rewrites?
- **`apply_recovery`**: when a patch failed, did the agent fix it?

> Clinescope is an independent, unofficial tool - not affiliated with, endorsed by, or sponsored by Cline or Cline Bot Inc. "Cline" is a trademark of Cline Bot Inc., used only to describe compatibility.

<p align="center"><img src="https://raw.githubusercontent.com/minh2416294/clinescope/main/docs/demo.png" alt="clinescope scoring a failing Cline run and, with --advice, coaching how to fix the agent's prompt for each failing scorer" width="640"></p>

## Why Clinescope (the wedge)

Most eval frameworks score chatbot Q&A. Clinescope scores **coding-agent execution traces** — and it
ships the one thing the incumbents leave to "write your own custom scorer": a **code-diff-quality
scorer**. DeepEval scores tool selection but not code patches or diffs; promptfoo, Langfuse, and
Braintrust hand the diff scorer to you; UK AISI's Inspect *runs* SWE agents but ships no diff-quality
scorer. Clinescope's `diff_coherence` / `diff_minimality` / `apply_recovery` scorers **are** that layer,
run against real captured Cline traces (see the [validation corpus](#validation-corpus)).

## Is the LLM judge any good? (judge validation)

An eval tool that uses an LLM to judge quality has to answer one question honestly: **does the judge
agree with a human?** Clinescope measures this the way a rigorous eval reader expects — chance-corrected
inter-rater agreement (**Cohen's κ**) between the LLM judge and a **human-labeled** gold set, with a
bootstrap confidence interval. The result on the current 50-item gold set (a free local `gpt-oss:20b`
judge vs. 50 blind human labels):

```
cohen_kappa:  0.0496    95% CI: [-0.1200, 0.2175]    N = 50
confusion (rows = human, cols = judge):
  human WASTEFUL      →  3 agree / 21 missed
  human NOT-WASTEFUL  → 24 agree /  2 missed
```

The confusion matrix tells the story: the free 20B judge is **strongly NOT-WASTEFUL-biased** — it calls
almost everything "fine," so on a balanced set it catches only 3 of 24 genuinely wasteful patches.
Because **κ ≈ 0 is far below the 0.5 floor, the judge is deliberately treated as advisory-only and kept
*out* of the CI gate** — `clinescope-gate` fires only on the deterministic scorers, never on a judge that
measured at chance level. That negative result is the point: Clinescope gates on the signals it trusts
and, provably, not on the one it doesn't. Recompute it yourself with no model call:

```bash
python -m clinescope.judge_run --report-only        # reads the committed cache; prints κ + CI
python -m clinescope.judge_multidraw --report-only  # how much κ moves across repeated draws
```

*Honest caveats:* N is still small so the CI is wide (it straddles zero), the judge is one free local
model on small edits, and a single-draw κ isn't reproducible to the digit — `gpt-oss:20b` flips labels
run-to-run even at temperature 0, which `judge_multidraw` measures directly (per-draw κ spread + Fleiss'
self-consistency). Growing the gold set from 26 to 50 harder, balanced, blind-labeled cases *lowered* the
measured κ — an honest floor, not a marketing figure. Robustness across multiple/frontier judge models
is on the roadmap.

## Get Started

1. **Install Clinescope**

    Requires Python 3.11+. Installing into a virtual environment is recommended.

    ```bash
    pip install clinescope
    ```

    Zero runtime dependencies (pure stdlib). The bundled sample traces, the real-trace
    validation corpus, and the human-labeled judge gold set all ship with the install, so
    `clinescope-corpus` and `python -m clinescope.judge_run --report-only` work out of the box —
    no `git clone` needed.

2. **Use Clinescope**

    **Get the score:**

    Point Clinescope at a Cline log file (a `messages.json` trace) to score the run — replace
    `path/to/messages.json` below with your own. (Cloned the repo instead of `pip install`? The
    same commands run as-is against the bundled [`examples/sample-trace.json`](examples/sample-trace.json).)
    ```bash
    clinescope path/to/messages.json --expected read_files apply_patch
    ```
    After `--expected`, list the tools you think the task needed. Clinescope checks whether the agent actually used them and scores the rest of the run automatically. Not sure which tool names to use? Run `clinescope --list-tools` to print the ones Clinescope knows.

    **Get full breakdown of every scorer:**
    ```bash
    clinescope path/to/messages.json --expected read_files apply_patch --verbose
    ```

    **Get advice to improve prompting:**
    ```bash
    clinescope path/to/messages.json --expected read_files apply_patch --advice
    ```

    **Compare several runs side by side:**

    Run the same task against different models (or Cline versions) and score them all in one table:
    ```bash
    python -m clinescope.compare run-a.json run-b.json run-c.json
    ```
    Each row is one run; the columns are the four scorers. To score `tool_selection` per run (each task expects different tools), pass a `--labels manifest.json` mapping each trace path to its `{"display": "...", "expected_tools": [...]}`.

## Validation Corpus

Clinescope ships a corpus of **real captured Cline runs** in [`examples/corpus/`](examples/corpus/), each hand-labeled in [`corpus.json`](examples/corpus/corpus.json) with its expected score profile, failure taxonomy, and the evidence its advice should name. A runner scores every trace, checks it against its label, prints a summary table, and **exits non-zero if any trace fails its label** — so the corpus is a real regression gate, not a demo. It ships with the install, so it runs anywhere:

```bash
clinescope-corpus            # or: python -m clinescope.corpus
```

This is the un-fakeable evidence that Clinescope catches real agent failures (and stays quiet on clean runs): the traces are real, the failures are real, and the runner proves Clinescope reproduces every labeled outcome. Six real traces cover three of the four failure modes; the fourth (`blind_rewrite`) is an honestly-stated gap — see [`examples/corpus/README.md`](examples/corpus/README.md) for the coverage table and why no local model produced it.

## Reporting Bugs

Small, discussed-first changes are welcome -- see [CONTRIBUTING.md](CONTRIBUTING.md) for dev setup, tests, and what a scorer change needs. You can file a [GitHub issue](https://github.com/minh2416294/clinescope/issues).

## License

[Apache-2.0](LICENSE). Copyright 2026 Tran Binh Minh.
