Metadata-Version: 2.4
Name: wurld
Version: 1.1.0
Summary: Posed sensor video: RGB + per-frame camera pose + intrinsics + bit-exact depth in one playable WebM
License: MIT
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: chromapakz>=0.4.0
Requires-Dist: numpy>=1.24
Requires-Dist: pillow>=10
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Provides-Extra: mcap
Requires-Dist: mcap>=1.0; extra == "mcap"
Provides-Extra: record3d
Requires-Dist: pyliblzfse>=0.4; extra == "record3d"
Dynamic: license-file

# wurld

> **wurld** — the *World's Unbroken Record of Localization & Depth*
> (formerly "worldline"; renamed in 1.1.0 — old files read forever)
**Posed sensor video in one playable WebM** — RGB video + per-frame camera pose +
intrinsics + timestamps + bit-exact metric depth (and confidence / object IDs), in a
single file that an ordinary video player treats as plain RGB.

wurld is the missing interchange layer between the things that *produce* posed
RGBD video (phone capture apps, SLAM systems, robots, synthetic renderers) and the
things that *consume* it (NeRF/splat training, robot learning, world-model pipelines,
SLAM evaluation). Today every producer invents a directory layout — COLMAP dirs,
`transforms.json`, TUM text files, per-app zip formats — and every consumer maintains a
zoo of parsers. wurld replaces the zoo with one canonical-convention container:

- **Payload**: [chromapakz](https://github.com/kmatzen/chromapakz) tracks — VP9 RGB +
  lossless, bit-exact uint16 planes for depth / confidence / IDs. Decodes natively in
  the browser via WebCodecs.
- **Metadata**: one Matroska `WURLD` tag holding cameras (COLMAP-style models),
  per-frame poses and sensor timestamps, signal semantics (how uint16 maps to meters),
  and world info (metric scale, gravity). See [SPEC.md](SPEC.md).
- **Conventions are fixed, not declared**: RDF camera axes (OpenCV/COLMAP),
  camera-to-world, wxyz quaternions, meters, seconds. Converters normalize on write;
  consumers never branch on axis flags.

## Quickstart

```sh
python3 -m venv .venv && .venv/bin/pip install -e ".[dev]"

# write a synthetic demo sequence (analytic RGBD + orbit poses)
.venv/bin/wurld demo demo.wl.webm

# inspect
.venv/bin/wurld info demo.wl.webm

# convert real datasets (auto-detects TUM / transforms.json / COLMAP / Stray Scanner)
.venv/bin/wurld convert path/to/rgbd_dataset_freiburg1_desk desk.wl.webm
.venv/bin/wurld convert path/to/transforms.json scene.wl.webm
.venv/bin/wurld convert path/to/colmap_project scene.wl.webm --images path/to/images
.venv/bin/wurld convert path/to/stray_capture scan.wl.webm       # needs ffmpeg on PATH

# extract back out
.venv/bin/wurld extract demo.wl.webm out/ --format tum        # or transforms | colmap
```

Python API:

```python
import wurld as wl

seq = wl.read("demo.wl.webm")
seq.rgb                  # (T, H, W, 4) uint8, lazily decoded
seq.depth_meters(0)      # (H, W) float, NaN = invalid
seq.c2w(0)               # 4x4 camera-to-world (RDF, meters)
seq.K("0")               # 3x3 intrinsics (K("0", frame_index=i) honors overrides)
seq.frames[0].t          # sensor timestamp (authoritative, may be non-uniform)
seq.rigs                 # camera-to-rig calibration; seq.rig_c2w(i, "1") derives poses
seq.imu["imu0"].samples  # (N, 7) [t, gyro xyz, accel xyz]
```

Long sequences (>10k frames) automatically pack poses into a binary frame table
(45 bytes/frame; SPEC §7) — `wl.write(..., frames_format="binary")` forces it.

## Browser viewer

```sh
npm install                      # fetches the chromapakz decoder
python3 -m http.server 8000      # from the repo root
# open http://localhost:8000/viewer/index.html  and drop a .wl.webm
# (or .../viewer/index.html?src=../demo.wl.webm)
```

Zero-install, WebCodecs-decoded: seekable playback with per-frame point-cloud
reprojection, camera trajectory + frustum, RGB and metric-depth panes, and the same
file playing in a plain `<video>` element.

## Layout

- `SPEC.md` — the format (v0.1)
- `wurld/` — Python reference implementation (container, conventions, EBML tag
  layer, converters, CLI, synthetic test scene)
- `tests/` — round-trip suite (`pytest`): bit-exact depth, pose fidelity, converter
  round trips, COLMAP binary parsing, validation errors
- `viewer/` — single-file browser viewer
- `LANDSCAPE.md` — the market research that motivated this project

## Status / verified

- 29/29 tests pass: depth round-trips bit-exactly through VP9; poses, timestamps, and
  intrinsics survive every converter round trip; TUM native-unit depth is bit-exact
  end to end; binary frame tables, rigs, IMU streams, and both Stray resampling
  policies covered.
- `ffmpeg`/`ffprobe` read and fully decode wurld files with zero warnings (all
  three VP9 streams); the WURLD tags are invisible to standard demuxers.
- Viewer verified in Chrome (native WebCodecs decode, no WASM), including binary
  frame tables.

## 1.0

SPEC declared **1.0** (frozen semantics; 1.x additions are additive-only, and
§11 codifies file scope: one rig, one clock). chromapakz pinned to the released
**0.4.0** — no more git installs. LICENSE (MIT), CHANGELOG, CI (Python 3.11/3.13
on Linux+macOS plus a viewer parity job), and packaging validated (sdist+wheel
build clean, twine-checked, fresh-venv install + CLI smoke). Multi-RGB pixel
storage remains gated on the ChromaPakZ #47 design review; phone importers and
WurldCam remain fixture/harness-validated pending a real capture.

## v0.9: bounded-memory iteration, trim, auto-lazy

- **`Sequence.iter_frames(start, stop)`**: memory-bounded streaming decode —
  one Cluster (~1 s) decoded at a time, so hour-long files iterate without
  holding the take in RAM. Pre-cadence files fall back to a full decode with a
  logged warning.
- **`wurld trim in out --frames 30:60`**: cut a range into a new file —
  poses rebased, signals sliced bit-exactly, quantization specs preserved,
  in-range IMU kept, provenance noted in the world description.
- **Viewer auto-lazy**: the probe's `Content-Range` total switches large files
  (>32 MB) to on-demand cluster scrubbing automatically; `?lazy=1|0` forces
  either way.
- **Multi-RGB design proposal** filed as
  [ChromaPakZ #47](https://github.com/kmatzen/ChromaPakZ/issues/47) — the
  schema/ABI sketch for true stereo pixel storage, awaiting maintainer review
  before implementation.

## v0.8: partial decode, EuRoC stereo+IMU, release plumbing

- **`Sequence.fetch_frames(indices)`**: local partial decode — reading 3 frames
  of a 10k-frame file no longer decodes the other clusters (same cluster-splice
  machinery as remote access, bit-exact parity tested).
- **EuRoC MAV importer** (`wurld convert MH_01_easy out.wl.webm`): the first
  real exercise of rigs + IMU together — cam0 posed video via
  `T_WB @ T_BS`, both cameras' OPENCV calibration plus a `body` rig
  (camera-to-body extrinsics; `rig_c2w` derives cam1 poses), imu0 with
  imu-to-cam0 extrinsics and measured rate. Dependency-free mini-YAML parser
  for `sensor.yaml`. (Video carries cam0 only — one RGB track in SPEC v0.1.)
- **MCAP `/tf`**: FrameTransform world→camera per posed frame, so Foxglove's 3D
  panel places the moving camera correctly.
- **ChromaPakZ #45 merged** (signal keyframe cadence) and release PR
  [#46 (0.4.0)](https://github.com/kmatzen/ChromaPakZ/pull/46) staged — once
  published, wurld pins `chromapakz>=0.4.0` instead of git installs. iOS
  vendor libs rebuilt from merged main.

## v0.7: random video access over ranges

- **Cluster-independent decode** (enabled by
  [ChromaPakZ #45](https://github.com/kmatzen/ChromaPakZ/pull/45), pending merge):
  signal tracks now keyframe at the RGB cadence, so `[header + Cluster k]`
  decodes that second of footage bit-exactly in isolation (+1.0% file size).
- **`wurld.remote.fetch_frames(fetch, indices)`**: random access to any
  frames of a remote file — SeekHead → Cues → only the touched Clusters are
  fetched and spliced-decoded. Verified: bit-exact vs full decode, untouched
  clusters never downloaded, and a clear error for files written before the
  keyframe cadence.
- **Viewer lazy scrubbing** (`?lazy=1&src=…`): poses render from the header
  instantly; video clusters fetch on demand as you scrub. Verified in Chrome:
  jumping frame 1 → 81 of the demo took **4 ranged requests total** with the
  middle cluster never downloaded.
- **WurldCam confidence**: on-device recordings now carry ARKit confidence
  as a second lossless signal (0/1/2 labels), verified device-free through the
  macOS harness. (Note: `build-native.sh` needs a chromapakz checkout with
  #45 for cluster-independent app recordings.)

## v0.6: on-device wurld recording, ffmpeg-native metadata

- **WurldCam v2**: the iOS app now records **wurld directly on-device** —
  libvpx + the chromapakz C core cross-compiled for iOS
  (`ios/scripts/build-native.sh`), Swift bindings over the `dc_stream_*`
  streaming ABI, and a Swift port of the pose-weaving `StreamWriter`
  (`.wl.webm` / `.r3d` toggle in the UI). The recording pipeline is verified
  without a device: `ios/scripts/verify-pipeline.sh` compiles the app's own
  writer+encoder for macOS, records a synthetic take, and validates it with the
  Python reader (poses at f32 precision vs analytic ground truth, exact f64
  timestamps, NaN-invalid depth preserved, chunked layout + consolidated table,
  ffmpeg-clean). ARKit-on-hardware remains the one untested link.
- **ffmpeg needs no patch**: ffprobe already surfaces the complete WURLD
  JSON document as a format tag —
  `ffprobe -show_entries format_tags=WURLD -of json scene.wl.webm` yields
  cameras, conventions, and (for JSON-frame files) every pose, in any
  ffmpeg-based tool. Binary pose tables/IMU tags don't surface; everything else
  does. The roadmap's "ffmpeg demuxer patch" is retired as unnecessary.

## v0.5: Record3D, MCAP/Foxglove, ranged viewer, nerfstudio

- **Record3D importer** (`wurld convert capture.r3d out.wl.webm`): the .r3d zip
  layout confirmed against the app author's own snippets and four community
  parsers — column-major K, scalar-last ARKit quaternions, LZFSE float32-meters
  depth (NaN invalid, float16 export variant handled), 0/1/2 confidence. Needs
  `pip install wurld[record3d]`.
- **MCAP export** (`wurld extract scene.wl.webm out.mcap --format mcap`):
  Foxglove-ready jsonschema channels (`/camera/pose`, `/camera/image` jpeg,
  `/camera/depth` 16UC1 bit-exact codes, `/camera/calibration`, `/imu/<id>`) plus
  the full WURLD document as an MCAP metadata record. Needs `[mcap]` extra.
- **Progressive ranged viewer**: `viewer/index.html?src=...` now loads via Range
  requests — trajectory and all poses render from the ~18KB header before any
  video byte, then video streams through the network decoder. Falls back to full
  download when the server lacks ranges. `scripts/range_server.py` is a
  range-capable dev server (python's builtin lacks Range).
- **nerfstudio DataParser** (`wurld.integrations.nerfstudio_parser`): reads a
  .wl.webm directly (frames extracted to a `<file>.cache/` beside it), poses
  converted to nerfstudio's convention, metric depth via
  `depth_unit_scale_factor` — verified against a live nerfstudio 1.1.5 install.

## v0.4: range-request access + Polycam

- **SeekHead** (SPEC §9.1): batch files begin with a fixed-width SeekHead covering
  the header elements, the first Cluster, and Cues. Combined with the
  metadata-first layout, a static file on S3/CDN serves calibration and every pose
  in at most two ranged reads.
- **`wurld.remote`**: `fetch_header(http_fetcher(url))` pulls all metadata +
  poses without downloading video — verified <2% of file bytes on a 60-frame
  sequence; `cues_offset`/`header_extent` expose what a video-seeking client needs
  next.
- **Polycam raw importer**: keyframes/cameras JSON (`fx/fy/cx/cy`, `t_00..t_23`
  ARKit c2w), microsecond-timestamp stems (sorted numerically), 16-bit mm depth,
  0/127/255 confidence labels, `corrected_cameras`/`corrected_images` preferred
  with `corrected=False` opt-out, same `--at depth|rgb` resolution policy as
  Stray. *Like Stray: validated against synthetic fixtures built from polyform's
  published schema — a real capture to confirm conventions is welcome.*

## v0.3: fully streamable playback and recording

- **Streaming layout** (SPEC §9): all wurld metadata — calibration *and* every
  pose — now lands **before the first Cluster**; Cues are rebuilt for the new
  offsets. A progressive reader has full pose data before the first video byte;
  ffmpeg still decodes and *seeks* cleanly.
- **Live recording**: `viewer/wurld.js` provides `WurldRecorder` (wraps
  chromapakz's streaming encoder, weaves `WURLD_POSES` chunks before each
  Cluster, consolidates a pose table on finish) and `WurldLivePlayer`
  (extracts pose tags from the byte stream, forwards clean video bytes to the
  network decoder). A crash-truncated recording is still a valid, fully-posed file
  up to its last flushed chunk.
- **`viewer/live.html`**: record → stream → play in one page — the player consumes
  only the byte stream and reconstructs the point cloud live (verified in Chrome:
  46/46 frames + poses received during recording; finalized file passes buffered
  decode).
- **Python `wurld.stream.StreamReader`**: incremental parser for live/growing
  streams; batch `wl.read()` also accepts crash-truncated live files via chunk
  concatenation.
- **Python live recording — `wurld.StreamWriter`** (chromapakz ≥ 0.4.0):

  ```python
  w = wl.StreamWriter(out.write, cameras={"0": cam}, has_rgb=True,
                      signal_meta=[wl.SignalMeta("depth", "depth",
                          {"type": "inverse_depth", "near": 0.4, "far": 12.0})])
  w.add_frame(frame, rgb=rgba, signals={"depth": {"float": z}})   # per capture tick
  w.add_imu("imu0", samples)                                      # any cadence
  w.finish()
  ```

  Verified: bit-exact depth through the live path, IMU chunk concatenation,
  progressive parse parity, crash-truncation survival, and ffmpeg-clean output.
  Pose-only takes (RGB + poses, no depth) work too — enabled upstream by
  [ChromaPakZ #44](https://github.com/kmatzen/ChromaPakZ/pull/44).

## v0.2

- **Binary frame tables** (`WURLD_FRAMES` TagBinary, 45 B/frame) for 10^6+-frame
  sequences; automatic beyond 10k frames.
- **Multi-camera rigs**: camera-to-rig calibration block + `rig_c2w()` derivation.
- **Per-frame intrinsics overrides** (zoom / autofocus drift).
- **IMU streams**: packed 32 B/sample gyro+accel tags with extrinsics.
- **Stray Scanner importer** (first real capture-app source; ffmpeg-based, ARKit
  RUB→RDF pose conversion, honest RGB/depth resolution policy). *Validated against
  synthetic fixtures — a real capture to confirm conventions is welcome.*

## Roadmap (v1.0)

Real-device validation (one WurldCam capture + one third-party-app capture
confirms every convention); LeRobot depth-backend PR upstream (staged
privately); ChromaPakZ #46 merge + 0.4.0 publish, then pin here; multi-RGB
tracks per the #47 design once reviewed — at which point the EuRoC importer
stores both cameras' pixels and SPEC binds frame camera ids to RGB streams.
