Metadata-Version: 2.4
Name: wastebook
Version: 0.0.2
Summary: Automated research lab notebook: hypotheses, code diffs, results, and conclusions for every experiment you run.
Project-URL: Homepage, https://github.com/como-research/wastebook
Author-email: Chase McDonald <chasecmcdonald@gmail.com>
License: MIT
License-File: LICENSE
Keywords: experiments,lab-notebook,ml,reproducibility,research,wastebook
Classifier: Development Status :: 3 - Alpha
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.11
Requires-Dist: litellm>=1.40
Requires-Dist: numpy>=1.24
Requires-Dist: pydantic>=2.0
Requires-Dist: rich>=13.0
Requires-Dist: scipy>=1.10
Requires-Dist: typer>=0.12
Provides-Extra: gcs
Requires-Dist: gcsfs>=2023.1; extra == 'gcs'
Provides-Extra: hf
Requires-Dist: fsspec>=2023.1; extra == 'hf'
Requires-Dist: huggingface-hub>=0.20; extra == 'hf'
Provides-Extra: mcp
Requires-Dist: mcp>=1.0; extra == 'mcp'
Provides-Extra: s3
Requires-Dist: s3fs>=2023.1; extra == 's3'
Provides-Extra: wandb
Requires-Dist: wandb>=0.16; extra == 'wandb'
Description-Content-Type: text/markdown

<p align="center">
  <img src="docs/assets/wastebook.png" alt="Newton's Waste Book, February 1664" width="250">
</p>

# Waste Book

An automated research lab notebook. Experiment trackers record *what the
metrics were*; wastebook records *why you ran it, what you changed, and what you
concluded* — automatically.

```bash
wb run -- ./launch_seeds.sh --algo sac --env Walker2d-v4
```

Every launch captures the git state (HEAD + dirty diff as a reproducible
patch), the exact command, and a hypothesis — prompted at launch, drafted by
an LLM from your diff. Results flow in from your existing tracker (wandb
first; wastebook never reimplements logging). Trials group automatically,
statistics come with confidence intervals and significance tests, and an LLM
pass drafts conclusions and surfaces anomalies in metrics you weren't
watching.

See [docs/walkthrough.md](docs/walkthrough.md) for a full worked example
(PPO vs SAC, sweeps, multi-seed stats, the review flow).

## Principles

- **wastebook never owns execution.** It wraps whatever you already run — a
  multi-seed launcher script, a slurm submit, a sweep agent. Tracker runs
  link to the experiment through propagated environment variables; completion
  is tracked through tracker run states, so detached launchers just work.
- **Never reinvent logging.** Metrics stay in wandb/tensorboard/etc., behind
  an adapter protocol.
- **The hypothesis is the soul.** Capture is low-friction (prompt with an
  LLM draft, `-m` flag, backfill later) but always present in the record.
  And it's never the *only* record of intent: at every launch the LLM writes
  its own **observation** — an independent read of what the launch actually
  does (exact parameters changed, factors varied, metrics to watch, relation
  to prior experiments) — stored as an llm-authored note. Your one-liner says
  what you think; the observation documents what's going on, and both are
  searchable later (`wb search`, MCP).
- **Local-first.** SQLite in `.wastebook/` (gitignored); the generated
  `labbook/` markdown is committed so your research history travels with the
  repo.

## Installation

wastebook is on [PyPI](https://pypi.org/project/wastebook/). The recommended
install is as a **uv tool**, which puts the `wb` command on your PATH so
you can call it directly from any repo — no `uv run` prefix, no activating
environments:

```bash
uv tool install "wastebook[wandb]"

wb --version           # works from anywhere
wb run -- python train.py
```

Upgrade later with `uv tool upgrade wastebook`.

Alternatively, add it as a dependency of your research project
(`uv add "wastebook[wandb]"`) — then it runs as `uv run wb` or plain
`wb` inside the activated environment. The tool install is the better
default: the notebook wraps your project, it doesn't need to live inside it.

## Quick start

```bash
cd your-research-repo
wb init                       # writes wastebook.toml, creates .wastebook/
wb run -- ./launch_seeds.sh   # hypothesis prompt (LLM-drafted from your diff)
wb log                        # timeline; lazy-syncs tracker state first
wb review                     # accept/edit drafted conclusions, metric proposals
wb compare exp-0001 exp-0002  # bootstrap CIs, Welch/Mann-Whitney, Cohen's d, Holm
wb show exp-0002 --diff       # full record incl. the captured patch
wb search "entropy collapse"  # months-later recall
wb book                       # regenerate committed labbook/ markdown
wb mcp                        # serve the record to coding agents (MCP)
```

## Commands

`init` · `run` (`-m` hypothesis, `-y` accept drafts, `--group` attach-by-label,
`--pin` shadow commit, `-e`/`--experiment` reuse-or-create by id,
`-p`/`--project`) · `list` · `log` · `show` · `compare` · `groups` (list/name an
experiment's trial groups) · `ask` (do we have the data to answer this?) ·
`project` (`list`/`use`/`new`/`show`/`assign`) · `review` · `sync` · `status`
(one-screen system summary) · `llm` (verify the configured model with a live
round-trip) · `search` ·
`book` · `mcp` · `annotate` · `conclude` · `exclude` · `merge` (fold a
mis-grouped experiment into another) · `aggregate` (fold experiments together) ·
`impact` (does a code change reach the launched script?) · `link` (attach a
tracker run that launched outside `wb run`)

## Concepts

The record is a strict hierarchy. From the top:

```
repository
└── project          one line of experimentation (a research question)
    └── experiment   one hypothesis
        ├── launch   one `wb run` invocation (command + code snapshot)
        └── run      one tracker run (e.g. one wandb run) the launch produced
            └─ grouped into ──┐
        ┌─────────────────────┘
        └── trial group       one distinct configuration within the experiment
            └── trial         one run of that exact configuration (a seed/replicate)
```

**Project** — a line of experimentation inside the repo, framed by a research
*question* (e.g. `ppo-clip`: "does clip range matter?" vs `arch-search`).
Declared in wastebook.toml as `[projects.<name>]`, optionally with its own key
metrics. Everything an experiment does stays inside its project: grouping
candidates, `varies_from` lineage, and LLM context never cross projects. A
repo that never declares projects has exactly one, implicit `default` project
and behaves as if the layer didn't exist.

**Experiment** — one hypothesis under test. Created by `wb run` (with
auto-grouping deciding new-vs-attach), belongs to exactly one project, and
accumulates launches, runs, findings, notes, and a conclusion. Alongside the
hypothesis, the LLM records its own *observation* of each new experiment —
an independent description of what the launch actually changes.

Auto-grouping compares a launch against the **whole project, nearest code
first** — not just the latest experiment. Two identity tiers are checked
against every experiment: byte identity (same code + command), then a
**semantic impact digest** — a Merkle-style hash of the AST of every module
the entry point can reach — so reformatting, comment edits, and rewrites of
unreachable code all still count as *the same experiment*. Beyond identity,
candidates are ranked by **impact-weighted code distance**: changed lines
bucketed into AST blocks (functions/methods/module residue) and weighted by
whether that block can affect the launched script — provably-unreachable
churn weighs zero, comment-only and moved code weighs ~zero, a changed
command flag weighs like code. Run some experiments, spend a week on
something else, come back — the returning launch reattaches to the original
experiment (its trials land in the right trial groups), and a new
experiment's `varies_from` lineage points at its nearest ancestor, not the
unrelated detour.

**Launch** — one `wb run` invocation: the exact command plus a code
snapshot (git HEAD + dirty patch). An experiment run on three machines has
three launches.

**Run** — one tracker-side run (wandb etc.) discovered via the linking key.
One launch of a multi-seed script yields many runs. wastebook never creates
runs; it observes them.

**Trial group** — one distinct configuration within an experiment, derived
automatically from the config axes that vary across its runs (a sweep over
`lr` yields one group per lr value; seed-like keys never split groups). The
group is the unit statistics compare. Groups are nameable (`wb groups
exp-0012 --name 1 baseline`) and their identity is deterministic
(experiment × config axes), so two machines deriving them independently
converge to the same records on sync. An experiment whose runs all share one
configuration has a single implicit group, shown as `(all)`.

**Trial** — one run within a trial group: a replicate of that exact
configuration. Trials are what give a group its n.

> **Breaking change (v0.2).** This hierarchy replaces the v0.1 flat record:
> "arms" are now persisted trial-group entities (the `runs.arm` label column
> is gone) and every experiment belongs to a project. Pre-0.2 databases and
> sync records are not migrated — delete `.wastebook/wastebook.db` (and any
> record-sync remote) and re-init.

```toml
[projects.ppo-clip]
description = "PPO clipping ablations"
question = "Does clip range materially affect final return?"

[[projects.ppo-clip.metrics]]     # optional: replaces the global [[metrics]]
name = "eval/return"              # for this project (wholesale, not merged)
primary = true
```

```console
$ wb project use ppo-clip        # active project for this machine
$ wb run -p arch-search -- ...   # or per-launch
$ wb groups exp-0012             # the derived trial groups
  #  NAME      LABEL       RUNS  eval/return
  1  —         lr=0.0001      4  96 ± 1.4 (n=4)
  2  baseline  lr=0.001       4  81 ± 1.4 (n=4)
$ wb groups exp-0012 --name 1 "tuned"   # names survive re-syncs & travel
```

### Do we have the data to answer this? (`ask`)

`wb ask` (and the MCP tool `assess_hypothesis`) checks a **new** question
against everything already recorded — every experiment's trial groups and
which metrics each recorded — then answers `answerable`, `partial`, or `no`.
When the evidence exists it doesn't stop at pointing: it runs the real
statistics on the identified groups.

```console
$ wb ask "does a smaller learning rate help final return?"
◆ answerable — exp-0012 sweeps lr with eval/return recorded on both sides (n=4/group)

eval/return · final · n=4 vs n=4
  exp-0012::lr=0.001  (A): 81 ± 1.4 (n=4)
  exp-0012::lr=0.0001 (B): 96 ± 1.4 (n=4)
  Δ = +15   95% bootstrap CI [+13.2, +16.8]
  Welch p=0.0001 · d=10
  → B better on this metric; unlikely to be noise (p=0.0001)
```

Every group and metric the LLM cites is validated against the record before
any statistics run — hallucinated references are dropped, never computed.
Without an LLM the command degrades to the data inventory plus keyword
matches.

### Does this change actually affect my run? (`impact`)

wastebook builds a static import + call graph from the launch command and checks
whether your changed lines can even execute. It's used two ways:

- **Grouping is now provable.** When a change *cannot reach* the launched entry
  point, the new run is grouped as a trial of the prior experiment with
  certainty — no LLM guess. Reachable changes still get the LLM's
  does-it-matter judgment.
- **`wb impact`** inspects it directly — your uncommitted changes, or an
  existing experiment's:

```console
$ wb impact --entry train.py
  verdict UNRELATED — (2 unrelated)
  ○ plotting.py · render_curves — symbol is never referenced anywhere reachable
  ○ notebooks/scratch.py — module is never imported or spawned from the entry point
```

The analysis is **sound for the negative**: it reports `UNRELATED` only when it
can prove the change can't run. Dynamic imports, `getattr`, or a subprocess
whose target it can't resolve degrade to `UNCERTAIN` rather than risk a false
"unrelated" that would merge two genuinely different experiments. Subprocess
launchers (`wb run -- ./launch.sh`) are followed when the spawned script is
a literal path/`-m` target.

### Splitting one experiment across machines

Pass an explicit id to tie separate launches together — even from different
machines — without any shared state:

```bash
# on your laptop, and again on an EC2 box, and again on a slurm node:
wb run -e pfsp-sweep -- ./train.sh --seed 1
wb run -e pfsp-sweep -- ./train.sh --seed 2
```

The id becomes the tracker linking key, so every run stamped `pfsp-sweep`
aggregates into one experiment on whichever machine syncs — no git, no shared
DB. Reusing an existing id attaches another launch (the hypothesis is kept). To
reunite experiments that were created *separately*, merge them after the fact
with `wb aggregate <target> <source>…`.

## Syncing the notebook across workstations

The record — hypotheses, code snapshots (with the captured patch), launches,
runs, metrics, findings, notes, conclusions — lives in a local SQLite database,
so wastebook works fully offline. Point `[sync]` at a remote and that record
propagates to every workstation, no git commits from throwaway machines
required:

```toml
[sync]
remote = "s3://my-lab/wastebook"   # "wandb", or any fsspec url (s3://, gs://, hf://)
# auto = false                   # opt out of pull-before-read / push-after-write
```

- **`remote = "wandb"`** rides on the wandb project you already use — the record
  travels as a versioned `wastebook-record` artifact. No extra credentials.
- **An object store** (`s3://…`, `gs://…`, `hf://datasets/user/repo`, or a local
  path) needs the matching driver — `uv tool install "wastebook[s3]"` (or `gcs`,
  `hf`) — and reads credentials from the usual provider environment chains
  (`AWS_*`, `GOOGLE_APPLICATION_CREDENTIALS`, `HF_TOKEN`). wastebook never stores
  keys.

With `auto` on (the default), reads pull the latest first and record-producing
commands push when they finish; `wb sync` always reconciles both
directions. Merging is by stable UUID with last-writer-wins on each
experiment's own fields and a union of its launches/runs/notes, so concurrent
work on different machines converges without clobbering. A `-e` id gets the
same UUID on every machine, so those launches merge into a single experiment on
sync. The local database stays the source of truth: a missing or misconfigured
remote degrades to a warning, never a blocked command.

## Artifact experiments (generative models, VLM judging)

Not every experiment logs metrics — sometimes the result *is* a file: images
or videos from a generation API, rendered outputs, model responses. wastebook
handles these with **watched-directory capture** and a **VLM rubric**:

```toml
[adapters.artifacts]
watch = ["outputs/"]          # where your script writes/downloads results

[[criteria]]                  # the rubric — criteria play the role of metrics
name = "prompt_adherence"
description = "How faithfully does the image depict the requested prompt?"
scale = 10

[[criteria]]
name = "anatomical_quality"
description = "Hands, faces, limbs are plausible; no extra fingers."
scale = 10
primary = true
```

```console
$ wb run -m "cfg=4.5 fixes hand artifacts" -- python generate.py
  ↳ wrapper exited (0)
  ↻ artifacts: 8 new file(s) under outputs/ → exp-0003

$ wb sync
  ↻ exp-0003: judged 8 artifact(s) × 2 criteria
  ◈ artifacts: 2/8 images still show fused fingers

$ wb compare exp-0002 exp-0003
  anatomical_quality · n=8 vs n=8
    Δ = +2.1  [95% CI: +0.7, +3.5] · Welch p=0.012
```

How it works: files that appear under the watched dirs during a launch are
linked to the experiment (use `wastebook.artifact(path)` in your script for
outputs elsewhere). Each artifact is recorded by reference + sha256 — files
stay where your script put them, and scores/observations survive even if
outputs are cleaned. At sync, a vision-capable LLM scores every artifact
against the rubric; each artifact counts as one sample, so the full
statistics pipeline (CIs, significance, `compare`) works with n = number of
generations. Per-artifact observations aggregate into findings, and
judgments are cached by content hash — reopening an experiment never
re-judges old artifacts. Videos are judged from evenly spaced frames
(requires `ffmpeg`; captured-but-not-judged without it).

## LLM configuration

The LLM drafts hypotheses from your diffs, summarizes changes, groups runs
semantically, narrates discovery findings, and drafts conclusions. It is
configured in `wastebook.toml` (any [litellm](https://docs.litellm.ai/docs/providers)
model string) with credentials from the environment — wastebook never stores keys:

```toml
[llm]
model = "anthropic/claude-sonnet-4-6"
context_budget = 12000     # max tokens of experiment history per call
```

```bash
export ANTHROPIC_API_KEY=sk-ant-...   # or OPENAI_API_KEY, GEMINI_API_KEY, LITELLM_API_KEY
```

**Local models via ollama** — no key needed; just point at a local model:

```toml
[llm]
model = "ollama/llama3.1"
# api_base = "http://gpu-box:11434"   # only if ollama isn't on localhost:11434
# autostart_ollama = false            # opt out of launching the daemon for you
```

If the model is a local ollama one and the daemon isn't running, wastebook starts
it on first use (bounded wait, then degrades if it can't) — no `ollama serve`
by hand. It won't pull missing weights, though: run `ollama pull <model>` once.
Remote hosts (`api_base` elsewhere) are left untouched.

`api_base` also works for any OpenAI-compatible server (vLLM, LM Studio):

```toml
[llm]
model = "hosted_vllm/meta-llama/Llama-3.1-70B-Instruct"
api_base = "http://localhost:8000/v1"
```

Without any of these, every LLM feature degrades gracefully: launches never
block, raw diffs are stored instead of summaries, grouping falls back to
byte-identity, and conclusions are written by hand in `wb review`. Set
`enabled = false` under `[llm]` to turn the LLM off explicitly, and
`wb exclude exp-NNNN` to keep any experiment out of LLM context.

## Development

The project is managed with [uv](https://docs.astral.sh/uv/). From a checkout:

```bash
uv sync                 # creates .venv from uv.lock (includes the dev group)
uv run pytest           # run the test suite
uv run wb --help   # run the CLI from source
```

Extras: `uv sync --extra wandb --extra mcp` to develop against the adapters.

Releases go to PyPI with:

```bash
uv build                # sdist + wheel into dist/
uv publish              # needs a PyPI token (UV_PUBLISH_TOKEN)
```

## Status

v0.2: the full loop works — launch capture (git patch snapshots, optional
shadow commits), LLM diff summaries + hypothesis drafts (litellm; degrades
gracefully without a key), wandb ingestion with adapter-state-driven lazy
sync, semantic auto-grouping (with static code-reachability), projects
namespacing separate lines of experimentation, trial groups auto-derived from
run configs as first-class nameable entities, the statistics toolkit, the
discovery pass over all logged series, evidence assessment for new hypotheses
(`ask` / MCP `assess_hypothesis`), drafted conclusions with a review flow,
committed `labbook/` rendering, search, an MCP server, and remote record sync
(wandb or any object store) so the notebook travels across workstations.
Planned next: tensorboard/mlflow adapters.
