Metadata-Version: 2.4
Name: tlog-ml
Version: 0.2.0
Summary: Lightweight, local-first experiment logger for neural network training — wandb-shaped API, zero deps, terminal + HTML + web viewers
Author: Philippe Hansen-Estruch
License: MIT
Project-URL: Repository, https://github.com/philippe-eecs/tlog
Project-URL: Issues, https://github.com/philippe-eecs/tlog/issues
Keywords: experiment-tracking,logging,machine-learning,wandb,slurm,tui
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: System :: Logging
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pillow; extra == "dev"
Requires-Dist: numpy; extra == "dev"
Dynamic: license-file

# tlog

A lightweight, local-first experiment logger for neural network training.
wandb-shaped API, **zero dependencies** in your training environment, and three
clean ways to look at your runs from a SLURM cluster with nothing but a
terminal:

| viewer | command | when |
|---|---|---|
| **terminal dashboard** | `tlog watch` | live charts in a tmux pane — the default |
| **live web dashboard** | `tlog serve` | wandb-like browser UI through an SSH/VS Code port-forward |
| **self-contained HTML** | `tlog export -o report.html` | one file with charts + images; preview in VS Code, scp it, share it |
| **custom report** | `tlog report spec.md` | a markdown narrative with live chart/table/image blocks — write it yourself or have a coding agent compose it |

Everything is plain append-only JSONL in a run directory: grep-able,
rsync-able, crash-safe, no daemon, no cloud, no account.

```
 ● demo/baseline (da064b) · step 1500 · finished
  loss  eval  training  timing  memory  console

 loss/charb                              0.3158    loss/dino                              0.07182
  1.552 ┤⡧⣼                                        0.3882 ┤⡧⣼
        │⠇⢹⢿⣠⢀                                            │⡇⢹⣶⣀⣀
        │   ⠹⢹⠢⣧⣄⣀                                        │   ⠛⢹⠢⡦⣆⢀
        │      ⠁⠋⠋⠳⢶⢤⡀                                    │       ⠋⠉⠢⣴⣀⣀
        │           ⠘⠙⠦⠦⢴⢄⣀⡀                              │           ⠘⠙⢢⡧⢦⣀⢀
        │                 ⠉⠙⠛⠓⠶⠤⣤⣠⣠⣀⡀                     │                 ⠉⠙⠋⠳⠶⢤⣤⣴⣠⡄⡀
 0.2771 ┤                         ⠉⠙⠉⠛⠋⠓⠲⠚⠴⠖⠤⠤⠦⡦  0.06998 ┤                         ⠁⠙⠉⠋⠋⠑⠳⠒⠲⠶⠤⡶⣦⣦
        10                                 1490           10                                 1490

 loss/ssim                                0.131    loss/total                              0.5004
 0.6492 ┤⡧⣼                                         2.605 ┤⡧⣼
        │⠇⢹⢶⣀⢀                                            │⠇⢹⢶⣀
        │   ⠹⠹⠦⣶⣄⣀                                        │   ⠹⢹⠢⣦⣄⣀
        │       ⠛⠋⠣⣴⣠⡀                                    │       ⠋⠋⠲⣴⢤⣀
        │          ⠈⠘⠙⠲⠦⢤⣀⢀                               │           ⠘⠙⠦⠦⢤⣀⣀
        │                 ⠉⠙⠋⠓⢴⠤⢤⣤⣠⡀⡀                     │                 ⠈⠙⠋⠲⠴⠤⣤⣠⢠⡀⡀
 0.1174 ┤                         ⠁⠉⠉⠛⠊⠛⠲⠖⠴⠒⠤⡴⠦⣤   0.4672 ┤                         ⠉⠉⠉⠛⠊⠛⠲⠖⠴⠖⠤⠤⠦⣦
        10                                 1490           10                                 1490

 ←/→ pages · ↑/↓ scroll · 1-9 cols (auto) · s smooth (0) · l log (off) · q quit
```

*An actual `tlog watch` frame — braille-canvas charts in a plain tmux pane.*

## Install

```bash
# latest features (multi-run compare, terminal images, custom reports):
pip install "git+https://github.com/philippe-eecs/tlog"

# released to PyPI (currently 0.1.0 — older; the next tag publishes the rest):
pip install tlog-ml          # distribution is tlog-ml; you still `import tlog`

# for development:
git clone https://github.com/philippe-eecs/tlog && cd tlog
pip install -e ".[dev]"
```

The `git+` form is the one to use on a fresh cluster today — it pulls `main`
with everything below, no PyPI release needed. The core has **zero
dependencies** — nothing to conflict with your torch/jax pins. PIL is used
opportunistically if present (image encoding, report downscaling); otherwise
a pure-stdlib PNG encoder takes over.

## Quickstart

```python
import tlog

run = tlog.init(project="vitok", name="vae-L16", config=vars(args))

for step in range(steps):
    ...
    if step % log_freq == 0:
        tlog.log({"loss/total": loss, "training/lr": lr,
                  "timing/mfu_percent": mfu}, step=step)
    if step % eval_freq == 0:
        tlog.log({f"eval/{k}": v for k, v in eval_stats.items()}, step=step)
        tlog.log_images("eval/recon", [orig, recon], step=step)  # torch/np/PIL

tlog.finish()
```

Then, in another tmux pane:

```bash
tlog                          # == tlog watch: live dashboard of the latest run
tlog watch baseline high-lr   # overlay multiple runs (or a project dir) in one TUI
tlog ls                       # table of runs: step, last loss, slurm job, status
tlog tail                     # live captured console output of the latest run
tlog serve                    # web UI on :8585 (VS Code auto-forwards the port)
tlog export run-a run-b -o compare.html    # side-by-side report
```

Key namespaces (`loss/`, `eval/`, `timing/`, ...) become chart groups / TUI
pages automatically.

## What gets captured

`tlog.init()` records, without being asked:

- **SLURM**: job id, job name, partition, nodelist, array task id, and the
  actual `sbatch` script that launched the job (saved as `launch.sh`)
- **git**: commit, branch, dirty flag, and a `diff.patch` of uncommitted changes
- **environment**: argv, entrypoint, hostname, user, python/torch/CUDA
  versions, GPU models, world size
- **system metrics** (background thread, 10s interval): GPU util/mem/temp/power
  per device via nvidia-smi, CPU%, RAM — shown as their own chart groups
- **console**: stdout/stderr teed to `console.log` (tqdm-safe; viewers resolve
  `\r` overwrites)

## How it works

tlog is two decoupled halves that only meet at the filesystem: a **write
path** that lives inside your training process, and a **read path** (the
viewers) that runs anywhere that can see the same disk. There is no daemon,
no database, no socket between them — a run *is* a directory:

```
runs/<project>/<name>__<timestamp>__<id>/
├── meta.json          # identity + environment snapshot + restart history
├── config.json        # your hyperparameters (vars(args))
├── metrics.jsonl      # one JSON object per log() call, append-only
├── system.jsonl       # sampled GPU/CPU/RAM
├── console.log        # teed stdout/stderr
├── launch.sh          # captured sbatch script (under SLURM)
├── diff.patch         # uncommitted git changes
└── media/             # PNGs + index.jsonl mapping them to (key, step)
```

### The write path never blocks training

`log()` serializes one JSON line and appends it. Lines are written whole and
flushed, so a crash loses at most the line in flight and can never corrupt
history; `fsync` runs on a 30s timer to bound hard-failure data loss without
paying sync cost per step. Everything slow happens off the hot path: git
diff / nvidia-smi / `scontrol` captures run in a background thread after
init, system sampling and the liveness heartbeat are daemon threads, and
framework versions are read from `sys.modules` instead of importing anything.

### Preemption-safe by construction

SLURM requeues a preempted job with the same job id and bumps
`SLURM_RESTART_COUNT`. `init(resume="auto")` (the default) detects that,
finds the run directory it created before the preemption, and keeps
appending — recording a restart event in `meta.json`. Restarting from an
older checkpoint re-logs some steps; instead of rewriting files (dangerous),
**readers keep the last value logged per (metric, step)**, so charts come out
continuous and the storage stays strictly append-only. Explicit resume:
`tlog.init(id="a1b2c3", resume="must")`.

### The read path is one engine with three faces

`store.py` discovers runs, tails JSONL incrementally (remembering byte
offsets, parsing only complete new lines), applies keep-last dedup, and
downsamples with **min/max/mean buckets** — a one-step loss spike survives
being squeezed into a 200-px chart instead of being averaged away. Debiased
EMA smoothing (same formula as wandb) sits on top. The three viewers are just
renderers over this engine:

- **TUI**: each terminal cell is a 2×4 braille dot grid, so a tmux pane
  becomes a pixel canvas; charts are drawn with Bresenham lines and repainted
  on the alternate screen buffer. Pure ANSI — no curses, works over any SSH.
- **Web**: a stdlib `ThreadingHTTPServer` with JSON endpoints; the browser
  polls every 3s and refetches only runs whose files changed (mtime-keyed).
- **Export**: the *same* frontend with data, images (base64), and uPlot
  inlined into one HTML file. One codebase, a mode flag, two surfaces.

### Liveness without IPC

A daemon thread touches `heartbeat` every 15s. Viewers call a run *running*
if the heartbeat is fresh, *finished* if `finish()` marked it, and *dead* if
neither — which is how a SIGKILLed job shows up correctly with no process
ever being asked.

## Distributed training

`tlog.init()` is a no-op on non-zero ranks (it checks the `RANK` env var set
by torchrun/SLURM), so you can call it unguarded — or keep your existing
`if rank == 0:` guard; both are fine.

## Migrating from wandb

```diff
-import wandb
+import tlog

-wandb.init(project=args.project, name=args.name, config=vars(args))
+tlog.init(project=args.project, name=args.name, config=vars(args))

-wandb.log(avg, step=step)
+tlog.log(avg, step=step)

-wandb.finish()
+tlog.finish()
```

Runs land in `./runs` by default; set `TLOG_DIR=/scratch/$USER/runs` (or pass
`dir=`) to keep them on scratch.

## The viewers in detail

**`tlog watch [runs...]`** — braille line charts with min/max bands, one page
per metric group plus media and console pages; the grid auto-sizes to the
pane and scrolls when a group has more charts than fit.

- **Compare runs**: `tlog watch baseline high-lr` (or name a project dir to
  take all its runs) overlays every metric wandb-style, one color per run,
  with a legend line. The `r` key cycles which run the console page shows.
- **Media page**: logged images render *in the terminal* — by default as
  half-block thumbnails (`▀` + 24-bit color), which work in every terminal
  including through tmux over SSH. Runs are columns, steps are rows, exactly
  like the web media tab. On kitty/Ghostty (kitty graphics protocol) or
  iTerm2/WezTerm (inline images), true pixel images are used automatically —
  except inside tmux, which usually eats those escapes, so tmux gets
  half-block unless you force a protocol with `--images kitty|iterm2`.
  (`--images off` hides the page.)

Keys: `←/→` pages · `↑/↓` (or `j/k`) scroll charts / media steps / console
history · `m` cycle media key · `r` cycle focused run · `1`–`9` force column
count, `0` auto (or `--cols N`) · `s` smoothing (EMA 0 → 0.6 → 0.9 → 0.99) ·
`l` log scale · `q` quit.

**`tlog serve [root]`** — open `http://localhost:8585` through VS Code Remote
(auto port-forward) or `ssh -L 8585:localhost:8585 cluster`. Multi-run
overlay charts with synced cursors, smoothing slider, log scale, a media tab
laid out **runs-as-columns × steps-as-rows** for side-by-side recon/eval
comparison, a config tab that highlights differing hyperparameters, and live
console.

**`tlog export <runs...> -o report.html`** — the same UI frozen into a single
file (images downscaled to ≤512px by default; `--max-image-px 0` keeps
originals). No server, no internet — works in VS Code's HTML preview.

**`tlog report spec.md [runs...]`** — custom pages: write plain markdown and
drop in ```` ```tlog ```` blocks where you want live elements, then render to
one self-contained HTML file (`--open` pops a browser). Prose narrates;
blocks pull from the runs:

````markdown
## Eval quality

FID is the one metric where high-lr finishes ahead.

```tlog chart
key: eval/fid
smooth: 0.9        # optional EMA; raw stays as a faint line
logy: true
runs: baseline, high-lr   # optional — defaults to the runs on the CLI
```

```tlog table
columns: config.lr, eval/fid min, eval/ssim max, loss/total last
```

```tlog images
key: eval/recon
last: 2            # or steps: 500, 1500
```
````

Three block types: `chart` (multi-run SVG overlay), `table` (one row per
run; columns are metric keys with an optional `min`/`max`/`last` aggregator,
or `config.*` values), and `images` (runs-as-columns × steps-as-rows grid).
Because a run is just files and the spec is just markdown, reports are easy
for both humans and coding agents to compose — ask an agent to inspect your
runs and it can write the analysis *and* the page that shows the evidence
(see `examples/report.md`).

## Let a coding agent review your runs

Because a run is nothing but files on disk — `metrics.jsonl`, `config.json`,
PNGs under `media/` — a coding agent (Claude Code, etc.) sitting on the same
cluster can inspect a run with no API key, no server, and no browser: it
greps the metrics, opens the images, and reads the config directly. `tlog
report` is the other half of that loop — it gives the agent a way to *hand
back* what it found as something you can actually look at.

A typical remote-cluster workflow:

```bash
pip install "git+https://github.com/philippe-eecs/tlog"   # on the cluster
# ... training writes runs to ./runs as usual ...
```

Then, in a Claude Code session on that cluster:

> "Compare `baseline` and `high-lr`. Look at the loss curves and the eval
> recon images, then write me a `tlog report` with your read on whether the
> higher LR helped."

The agent reads the JSONL and the PNGs, writes a `spec.md` with prose plus
`chart`/`table`/`images` blocks, runs `tlog report spec.md baseline high-lr`,
and you get a single self-contained `spec.html` — its analysis up top, the
charts and side-by-side reconstructions as evidence below. scp it to your
laptop, skim it, and reply with feedback; the agent revises the spec and
re-renders. The diagrams, charts, and logs are all in one reviewable file,
and the agent never needed anything but the run directory.

## Demo without a GPU

```bash
python examples/fake_train.py --steps 2000 &
tlog watch
```

## Prior art

[trackio](https://github.com/gradio-app/trackio), [aim](https://github.com/aimhubio/aim),
TensorBoard, and MLflow all live in adjacent space. tlog's niche is the
combination: a zero-dependency stdlib-only core safe to drop into any
training env, files you can grep as the source of truth, SLURM-native
metadata + preemption semantics, a terminal dashboard designed for a tmux
pane on a GPU cluster, and single-file HTML reports — in ~2,700 lines of
Python you can read in an afternoon.

## Tests

```bash
python -m pytest tests/
```

## License

MIT
