Metadata-Version: 2.4
Name: swe-lab
Version: 0.1.0
Summary: Tooling to enrich and audit SWE-Bench evaluation data (related-files annotation, and more)
Project-URL: Homepage, https://github.com/Luolc/swe-lab
Project-URL: Repository, https://github.com/Luolc/swe-lab
Project-URL: Issues, https://github.com/Luolc/swe-lab/issues
Author-email: Liangchen Luo <luolc.witty@gmail.com>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: agent,coding-agent,dataset,evaluation,llm,sandbox,swe-bench,swe-bench-pro
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Testing
Classifier: Typing :: Typed
Requires-Python: <3.14,>=3.13
Requires-Dist: huggingface-hub>=1.22.0
Requires-Dist: polars>=1.42.1
Requires-Dist: pydantic>=2.10
Requires-Dist: typer>=0.27.0
Description-Content-Type: text/markdown

# SWE Lab

Tooling to **build, run, enrich, audit and fix SWE (a.k.a. coding agent) evaluation data**.

## Tasks

- **Related-files annotation** (`src/swe_lab/pipelines/related_files/`) —
  for each task instance, produce a ground-truth list of the code snippets a
  model needs to read to solve it. **Shipped**: 100 instances annotated & QA'd.
  See [`pipelines/related_files/README.md`](src/swe_lab/pipelines/related_files/README.md).

- **Quality auditing** *(planned)* — flag "skewed" eval examples that no longer
  measure real capability (ambiguous specs vs. overly-specific tests, broken
  environments, contamination, brittle graders), in the spirit of OpenAI's
  [*Separating signal from noise in coding evaluations*](https://openai.com/index/separating-signal-from-noise-coding-evaluations/).
  Not started; it will land as a sibling under `pipelines/`.

The overall roadmap and design live in [`docs/README.md`](docs/README.md).

## Setup

### Prerequisites

- [uv](https://docs.astral.sh/uv/) for environment and dependency management
- [direnv](https://direnv.net/) for auto-activating the environment
- Python 3.13 (uv will install it automatically if missing)

### 1. Clone

```bash
git clone https://github.com/Luolc/swe-lab.git
cd swe-lab
```

Optional — the `--capture proxy` mode compiles the standalone
[`cc-reverse-proxy`](https://github.com/Luolc/cc-reverse-proxy) Go project. It is
**not** a submodule: by default it is looked up as a sibling checkout next to
this repo (`../cc-reverse-proxy/reverse_proxy.go`); clone it there, or point
`CC_REVERSE_PROXY_SRC` at its `reverse_proxy.go`. The default `stream` capture
needs none of this.

### 2. Set up the environment

```bash
uv sync          # create .venv and install all (incl. dev) dependencies
direnv allow     # auto-activate the venv on cd (uses .envrc)
```

If you don't use direnv, activate manually with `source .venv/bin/activate`.

Install the pre-commit hooks (ruff, pyink, isort, basedpyright, uv-lock):

```bash
uv run pre-commit install
```

### 3. Download the datasets

Dataset data files are gitignored and must be downloaded locally. See
[`datasets/README.md`](datasets/README.md) for the list of available datasets
and per-dataset download instructions.

## [Disclaimer](DISCLAIMER.md)

This is a personal project and is not affiliated with any company. The content does not reflect any specific company's projects, products or internal work.
