Metadata-Version: 2.4
Name: delphik
Version: 0.1.1
Summary: Pull the open benchmark-defect set into your terminal and run a fair, pinned eval.
Author: Jongwon Park
License-Expression: Apache-2.0
Project-URL: Homepage, https://posttrain.dev
Project-URL: Repository, https://github.com/delphik-ai/delphik-cli
Project-URL: Open Defect Store, https://github.com/delphik-ai/open-defect
Keywords: benchmark,eval,defects,harbor,llm,agent,swe-bench
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# delphik

Pull the open benchmark-defect set into your terminal, hold out the broken tasks, pin the version, and run your own fair eval.

`delphik` reads the open defect store ([github.com/delphik-ai/open-defect](https://github.com/delphik-ai/open-defect)) and tells you which tasks of a benchmark are currently broken or have a fix in flight — so you can exclude them and compare models on the same clean set.

## Install

```bash
pip install delphik        # (or: pipx install delphik)
```

No configuration needed — it reads the public open-defect repo directly.

## Use

```bash
# what's broken or in-flight in a benchmark right now?
$ delphik defects swebench-verified
swebench-verified  @ main
● 36 found   ● 22 fixing   → 49 tasks held out
  + 13 benchmark-level caveats (whole-benchmark; can't hold out per task)

# list the benchmarks in the store
$ delphik benchmarks
```

### Run a clean, pinned eval with [Harbor](https://github.com/harbor-framework/harbor)

`--exclude-args` prints Harbor `-x <task>` flags, ready for command substitution:

```bash
harbor run -d swebench-verified@2.0 \
  $(delphik defects swebench-verified --exclude-args) \
  -a claude-code -m <model>
```

Harbor excludes the held-out tasks (`-x` / `--exclude-task-name`) and pins the benchmark version (`-d name@version`), so any two runs are compared on the same clean, versioned set.

## Reproducibility

The held-out set is a snapshot of an open-defect commit. Pin it with `--ref`:

```bash
delphik defects swebench-verified --ref <commit>     # reproducible set
delphik defects swebench-verified --refresh          # re-pull latest
```

## What counts as "held out"

- **held-out tasks** — task-specific defects that are `found` (open) or `fixing` (fix in flight). These are excluded from a fair eval.
- **benchmark-level caveats** — defects in the harness/grading/dataset that affect the *whole* benchmark. They can't be held out per task, so they're reported separately, never folded into the count.
- `fixed` tasks are healthy again and are not held out.

## Options

| flag | meaning |
|---|---|
| `--unresolved` | held-out set (found + fixing) — the default view |
| `--exclude-args` | print `-x <task>` args for `$(...)` into Harbor |
| `--json` | emit the full DefectSet as JSON |
| `--ref <ref>` | read the store at a branch/tag/commit (default: `main`) |
| `--refresh` | re-download the store snapshot |
| `--repo <path>` | read a local `delphik-open-defects` checkout (dev) |

Data is open under CC BY 4.0. Apache-2.0 code.
