Metadata-Version: 2.4
Name: labcoat
Version: 0.1.2
Summary: Automated research lab notebook: hypotheses, code diffs, results, and conclusions for every experiment you run.
Project-URL: Homepage, https://github.com/como-research/labcoat
Author-email: Chase McDonald <chasecmcdonald@gmail.com>
License: MIT
License-File: LICENSE
Keywords: experiments,lab-notebook,labcoat,ml,reproducibility,research
Classifier: Development Status :: 3 - Alpha
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.10
Requires-Dist: litellm>=1.40
Requires-Dist: numpy>=1.24
Requires-Dist: pydantic>=2.0
Requires-Dist: rich>=13.0
Requires-Dist: scipy>=1.10
Requires-Dist: typer>=0.12
Provides-Extra: mcp
Requires-Dist: mcp>=1.0; extra == 'mcp'
Provides-Extra: wandb
Requires-Dist: wandb>=0.16; extra == 'wandb'
Description-Content-Type: text/markdown

# labcoat

An automated research lab notebook. Experiment trackers record *what the
metrics were*; labcoat records *why you ran it, what you changed, and what you
concluded* — automatically.

```bash
labcoat run -- ./launch_seeds.sh --algo sac --env Walker2d-v4
```

Every launch captures the git state (HEAD + dirty diff as a reproducible
patch), the exact command, and a hypothesis — prompted at launch, drafted by
an LLM from your diff. Results flow in from your existing tracker (wandb
first; labcoat never reimplements logging). Trials group automatically,
statistics come with confidence intervals and significance tests, and an LLM
pass drafts conclusions and surfaces anomalies in metrics you weren't
watching.

See [docs/walkthrough.md](docs/walkthrough.md) for a full worked example
(PPO vs SAC, sweeps, multi-seed stats, the review flow).

## Principles

- **labcoat never owns execution.** It wraps whatever you already run — a
  multi-seed launcher script, a slurm submit, a sweep agent. Tracker runs
  link to the experiment through propagated environment variables; completion
  is tracked through tracker run states, so detached launchers just work.
- **Never reinvent logging.** Metrics stay in wandb/tensorboard/etc., behind
  an adapter protocol.
- **The hypothesis is the soul.** Capture is low-friction (prompt with an
  LLM draft, `-m` flag, backfill later) but always present in the record.
- **Local-first.** SQLite in `.labcoat/` (gitignored); the generated
  `labbook/` markdown is committed so your research history travels with the
  repo.

## Installation

labcoat is on [PyPI](https://pypi.org/project/labcoat/). The recommended
install is as a **uv tool**, which puts the `labcoat` command on your PATH so
you can call it directly from any repo — no `uv run` prefix, no activating
environments:

```bash
uv tool install "labcoat[wandb]"

labcoat --version           # works from anywhere
labcoat run -- python train.py
```

Upgrade later with `uv tool upgrade labcoat`.

Alternatively, add it as a dependency of your research project
(`uv add "labcoat[wandb]"`) — then it runs as `uv run labcoat` or plain
`labcoat` inside the activated environment. The tool install is the better
default: the notebook wraps your project, it doesn't need to live inside it.

## Quick start

```bash
cd your-research-repo
labcoat init                       # writes labcoat.toml, creates .labcoat/
labcoat run -- ./launch_seeds.sh   # hypothesis prompt (LLM-drafted from your diff)
labcoat log                        # timeline; lazy-syncs tracker state first
labcoat review                     # accept/edit drafted conclusions, metric proposals
labcoat compare exp-0001 exp-0002  # bootstrap CIs, Welch/Mann-Whitney, Cohen's d, Holm
labcoat show exp-0002 --diff       # full record incl. the captured patch
labcoat search "entropy collapse"  # months-later recall
labcoat book                       # regenerate committed labbook/ markdown
labcoat mcp                        # serve the record to coding agents (MCP)
```

## Commands

`init` · `run` (`-m` hypothesis, `-y` accept drafts, `--group`, `--pin` shadow
commit) · `log` · `show` · `compare` · `review` · `sync` · `search` · `book` ·
`mcp` · `annotate` · `conclude` · `exclude` · `group` ·
`link` (attach a tracker run that launched outside `labcoat run`)

## Artifact experiments (generative models, VLM judging)

Not every experiment logs metrics — sometimes the result *is* a file: images
or videos from a generation API, rendered outputs, model responses. labcoat
handles these with **watched-directory capture** and a **VLM rubric**:

```toml
[adapters.artifacts]
watch = ["outputs/"]          # where your script writes/downloads results

[[criteria]]                  # the rubric — criteria play the role of metrics
name = "prompt_adherence"
description = "How faithfully does the image depict the requested prompt?"
scale = 10

[[criteria]]
name = "anatomical_quality"
description = "Hands, faces, limbs are plausible; no extra fingers."
scale = 10
primary = true
```

```console
$ labcoat run -m "cfg=4.5 fixes hand artifacts" -- python generate.py
  ↳ wrapper exited (0)
  ↻ artifacts: 8 new file(s) under outputs/ → exp-0003

$ labcoat sync
  ↻ exp-0003: judged 8 artifact(s) × 2 criteria
  ◈ artifacts: 2/8 images still show fused fingers

$ labcoat compare exp-0002 exp-0003
  anatomical_quality · n=8 vs n=8
    Δ = +2.1  [95% CI: +0.7, +3.5] · Welch p=0.012
```

How it works: files that appear under the watched dirs during a launch are
linked to the experiment (use `labcoat.artifact(path)` in your script for
outputs elsewhere). Each artifact is recorded by reference + sha256 — files
stay where your script put them, and scores/observations survive even if
outputs are cleaned. At sync, a vision-capable LLM scores every artifact
against the rubric; each artifact counts as one sample, so the full
statistics pipeline (CIs, significance, `compare`) works with n = number of
generations. Per-artifact observations aggregate into findings, and
judgments are cached by content hash — reopening an experiment never
re-judges old artifacts. Videos are judged from evenly spaced frames
(requires `ffmpeg`; captured-but-not-judged without it).

## LLM configuration

The LLM drafts hypotheses from your diffs, summarizes changes, groups runs
semantically, narrates discovery findings, and drafts conclusions. It is
configured in `labcoat.toml` (any [litellm](https://docs.litellm.ai/docs/providers)
model string) with credentials from the environment — labcoat never stores keys:

```toml
[llm]
model = "anthropic/claude-sonnet-4-6"
context_budget = 12000     # max tokens of experiment history per call
```

```bash
export ANTHROPIC_API_KEY=sk-ant-...   # or OPENAI_API_KEY, GEMINI_API_KEY, LITELLM_API_KEY
```

**Local models via ollama** — no key needed; just point at a local model:

```toml
[llm]
model = "ollama/llama3.1"
# api_base = "http://gpu-box:11434"   # only if ollama isn't on localhost:11434
```

`api_base` also works for any OpenAI-compatible server (vLLM, LM Studio):

```toml
[llm]
model = "hosted_vllm/meta-llama/Llama-3.1-70B-Instruct"
api_base = "http://localhost:8000/v1"
```

Without any of these, every LLM feature degrades gracefully: launches never
block, raw diffs are stored instead of summaries, grouping falls back to
byte-identity, and conclusions are written by hand in `labcoat review`. Set
`enabled = false` under `[llm]` to turn the LLM off explicitly, and
`labcoat exclude exp-NNNN` to keep any experiment out of LLM context.

## Development

The project is managed with [uv](https://docs.astral.sh/uv/). From a checkout:

```bash
uv sync                 # creates .venv from uv.lock (includes the dev group)
uv run pytest           # run the test suite
uv run labcoat --help   # run the CLI from source
```

Extras: `uv sync --extra wandb --extra mcp` to develop against the adapters.

Releases go to PyPI with:

```bash
uv build                # sdist + wheel into dist/
uv publish              # needs a PyPI token (UV_PUBLISH_TOKEN)
```

## Status

v0.1: the full loop works — launch capture (git patch snapshots, optional
shadow commits), LLM diff summaries + hypothesis drafts (litellm; degrades
gracefully without a key), wandb ingestion with adapter-state-driven lazy
sync, semantic auto-grouping, sweep arms auto-derived from run configs, the
statistics toolkit, the discovery pass over all logged series, drafted
conclusions with a review flow, committed `labbook/` rendering, search, and
an MCP server. Planned next: tensorboard/mlflow adapters and remote sync of
the record.
