Metadata-Version: 2.4
Name: framelock
Version: 0.4.0
Summary: Synchronized multi-camera recording for multi-view pose estimation: OAK-D + iPhone RTSP + mic on one sub-ms clock, with rig registration, corresponded exports, and 3D playback
Project-URL: Repository, https://github.com/ryanrudes/framelock
Project-URL: Documentation, https://github.com/ryanrudes/framelock/tree/main/docs
Author-email: Ryan Rudes <ryanrudes@gmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: arkit,aruco,calibration,camera,depthai,multi-view,oak-d,odometry,pose-estimation,registration,rtsp,synchronization
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: MacOS
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Multimedia :: Video :: Capture
Classifier: Topic :: Scientific/Engineering :: Image Processing
Requires-Python: >=3.12
Requires-Dist: av>=18.0.0
Requires-Dist: depthai>=3.7.1
Requires-Dist: irtsp>=0.5.0
Requires-Dist: numpy>=2.5.1
Requires-Dist: opencv-python>=5.0.0.93
Requires-Dist: rich>=13.3.3
Requires-Dist: sounddevice>=0.5.5
Provides-Extra: apriltag
Requires-Dist: pupil-apriltags>=1.0.4; extra == 'apriltag'
Provides-Extra: location
Requires-Dist: pyobjc-framework-corelocation>=10.0; (sys_platform == 'darwin') and extra == 'location'
Provides-Extra: viz
Requires-Dist: viser>=1.0; extra == 'viz'
Description-Content-Type: text/markdown

# framelock

[![CI](https://github.com/ryanrudes/framelock/actions/workflows/ci.yml/badge.svg)](https://github.com/ryanrudes/framelock/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/framelock?label=PyPI&color=blue)](https://pypi.org/project/framelock/)
[![Python](https://img.shields.io/pypi/pyversions/framelock?label=python)](https://pypi.org/project/framelock/)
[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)

**Synchronized multi-camera recording for multi-view pose estimation.**
Point an OAK-D, a handful of iPhones, and a microphone at the same
scene; framelock records everything into one session folder on one
sub-millisecond shared clock, solves where every camera *is* in a
single world frame, and hands you corresponded, GVHMR-ready exports.

- **One clock.** Hardware-locked shutters inside an OAK, a verified
  re-implementation of DepthAI's sync across devices, acoustic
  chirp calibration for phones — every sample from every sensor lands
  on one host-monotonic timeline, and every claim is cross-checked by
  an independent estimator before it's trusted.
- **One world frame.** Press `b` mid-take: the laptop shows a fiducial
  board with a live per-camera status strip, and the whole rig —
  static cameras, phones via ARKit odometry, all three OAK cameras
  fused through factory extrinsics — is solved automatically at
  session end (~1 cm / 1° per camera, gravity-leveled). Printed wall
  tags extend it anywhere the board can't reach.
- **Beyond pixels.** iRTSP phones stream ARKit pose, IMU, GPS,
  compass, barometer, and LiDAR *on the video's own clock* (no offset
  estimation — the app's clock design makes it exact); OAK-Ds
  contribute IMU, stereo depth, feature tracks, and thermals.
- **Verified, not hoped.** Gyro-vs-pixels sync checks, gravity
  cross-checks between accelerometers and ARKit, stereo-metric size
  votes, pose-noise grading — `--verify` tells you what the take is
  actually worth.

```bash
pip install framelock            # or: uv add framelock
pip install 'framelock[viz]'     # + 3D playback (viser)
```

One CLI, sensible subcommands — `framelock help` lists them:

| | |
|---|---|
| `framelock` / `framelock record` | record a session (bare flags work: `framelock --shutter-sync ...`) |
| `framelock register` | solve rig registration (board wave / tags / acoustic) |
| `framelock tags` | export printable wall fiducials at true physical scale |
| `framelock correspond` | corresponded exports + end-to-end verification |
| `framelock align` | acoustic label-bias / clock-drift measurement |
| `framelock preview` | synchronized side-by-side playback with transport |
| `framelock viz` | fly through the take in 3D (viser) |
| `framelock report` | session health report |

Beyond video: iRTSP phones stream odometry (fused IMU, ARKit 6-DOF pose,
GPS, compass, barometer, intrinsics, LiDAR depth) and framelock records it
on the **same timeline as the frames** — no offset estimation, the app's
clock design makes it exact (see `docs/OUTPUTS.md`). OAK-Ds contribute
their own IMU, stereo depth, feature tracks, and thermals via
`OakD(data=("imu", "depth", ...))`. Everything lands in the corresponded
export together, `framelock correspond --verify` checks the odometry
against the pixels, and `framelock viz` flies you through the take in 3D.

The whole rig is **registered into one world frame**
(`registration.json`). The primary ritual costs one keypress: press
`b` during a take and the laptop shows a fiducial board fullscreen
with a live per-camera status strip; show it to each camera (the
solver verifies the laptop's stillness from the data) and the rig is
solved automatically at session end — every camera ~1 cm / 1°, phones
via their ARKit odometry with timed drift anchors, all OAK cameras
fused through factory extrinsics, gravity-leveled z-up world (or pick
your own origin: `world_origin="board"` / any device / its Source
object). Board size is cross-voted by independent estimators (ruler >
stereo triangulation / ARKit motion > screen EDID), and every claim is
verified by an independent witness — gravity cross-checks between the
OAK accelerometer and each phone's ARKit measure the solved rotation's
error board-free. The witness rules are strict about *who may testify*:
only a camera that stayed put (its own IMU vouching for it) can convict
the laptop of moving, because a phone cannot tell board motion from its
own ARKit drift — measured on a live take, a phone's frame wandered
15.8 cm between two board showings while the parked OAK proved the board
still to 0.7 cm. That difference is reported as the drift it is, and
absorbed by timed anchors. Every take writes **`report.json`**: the
numbers behind every line the console printed — the witness table, each
phone's drift, how many frames each camera actually solved.
Printed **wall tags** ([docs/TAGS.md](docs/TAGS.md))
extend it: `framelock tags grid 0-11 --cols 4 --mm 40`, pin anywhere
fixed, and localized devices survey them automatically — every later
sighting pins phone drift or rescues a camera the board never reached.
The acoustic stop-and-chirp ritual (`g`) remains as a
no-line-of-sight fallback. Registered devices share a metric world
cell in the viz, with the board and tags drawn at their solved poses.

```bash
# The classic single-OAK setup (all flags from the original recorder):
uv run framelock --shutter-sync --exposure 8000 --iso 800 --warmup auto -n take1

# Add an iPhone (any RTSP-server camera app):
uv run framelock --shutter-sync --exposure 8000 --iso 800 \
    --rtsp iphone=rtsp://192.168.1.23:8554/live

# Digital zoom ramp 1x -> 2x between t=5s and t=10s of the recording:
uv run framelock --zoom-ramp 5:10:1.0:2.0
```

The CLI is a thin veneer — everything is a first-class Python API
(see **[examples/](examples/)** for runnable scripts covering each feature):

```python
from framelock import OakD, RtspCamera, Session, Warmup, ZoomRamp

oak = OakD(shutter_sync=True, shutter=1/125, iso=800,
           automations=[ZoomRamp(1.0, 2.0, start=5, end=10)])
phone = RtspCamera("rtsp://192.168.1.23:8554/live", name="iphone")

with Session("recordings", name="take1", warmup=Warmup.AUTO) as session:
    session.record(oak, phone)
```

Options are typed `StrEnum`s (`VideoFormat.MP4`, `Transport.TCP`,
`ExternalSync.MASTER`, `Control.ZOOM`, `ChirpKind.VELVET`, `Warmup.AUTO`) —
IDE-completable and validated, while plain strings keep working everywhere.
Dynamic control is first-class too: `Func("zoom", lambda t: ...)` drives a
control with any function of time, and `session.record(..., on_tick=...)`
runs arbitrary per-tick logic — `apply()` is rate-safe (per-control
resolution filtering built into every source), so both can emit values
naively (see `examples/04_live_control.py`).

Previews are optional (`ui=False`, or `preview=False` per source) and render
as one CCTV-style composite window — labeled tiles, REC indicator, "no
signal" placeholders. Start/stop is console-driven: `warmup=Warmup.MANUAL`
streams and settles until **you** press `r` in the terminal; `q` stops
(with the end chirp when `end_chirp=True`); `session.request_start()` /
`request_stop()` are the programmatic equivalents (thread-safe).

## Output layout

```
recordings/take1/
  session.json               manifest: sources, clock map, settings, accuracy notes
  sync_groups.csv            cross-device alignment (per group: each stream's seq + ts)
  registration.json          every device's pose in ONE world frame (+ board, tags)
  audio_alignment.json       measured per-source label bias / clock drift
  report.json                the evidence behind every console line (read this first)
  corresponded/              equal-frame-count videos + pose-per-frame exports
  oak/
    CAM_A.mp4  CAM_A.mcap  CAM_A.intrinsics.json  CAM_A.timestamps.csv
    CAM_B.*    CAM_C.*     calibration.json
    depth.mkv  features.csv  imu_*.csv  system.csv    with data=(...)
  iphone/
    video.mp4  video.timestamps.csv
    imu.csv  pose.csv  gnss.csv ...   when the iRTSP app streams them
```

Every video has a `*.timestamps.csv` (file frame index ↔ sequence number ↔
host-monotonic + epoch time) and `sync_groups.csv` joins them across devices,
so post-hoc alignment never depends on the live estimates. Full column
reference for every file: **[docs/OUTPUTS.md](docs/OUTPUTS.md)**.

```bash
uv run framelock viz recordings/take1   # fly through the take in 3D:
# camera frusta with their images, phone trajectories from ARKit odometry,
# play/pause/scrub/step/speed, click any camera for its full details
# (needs the viz extra: uv sync --extra viz)
```

## Synchronization: what you actually get

| Link | Mechanism | Accuracy |
|---|---|---|
| OAK, all three cameras (one device) | hardware FSIN (`shutter_sync=True`): the colour camera **drives** the pulse, both monos follow | **0.039 ms** colour↔mono, **0.013 ms** mono↔mono (measured, 1080p30) |
| OAK mono pair only (`shutter_sync="mono"`) | FSIN CAM_C→CAM_B; colour free-runs | monos ~0.01 ms; colour ±½ frame (3–16 ms) |
| Two OAK devices, one host | dai XLink timesync → host clock | timestamps <1 ms comparable; shutters free-run (±½ frame) unless FSIN-wired (PoE M8 / FFC hardware only — **not** the USB OAK-D) |
| iPhone / RTSP | software only (no hw path exists into a phone) | live: tens of ms (arrival-anchored); post-hoc with sidecars + clap calibration: ~1 frame |

The host-side `Synchronizer` re-implements DepthAI's `dai.node.Sync`
algorithm (verified against depthai-core v3.7.1 source) for any number of
streams, with one deliberate improvement: nearest-match grouping instead of
first-fit, so same-rate streams can't lock into a stable one-frame-off
pairing.

## Camera controls & automations

`OakD` accepts initial `focus`/`exposure` at construction (`zoom=True`
enables the zoom chain, which starts at 1x), live control via
`oak.apply(zoom=1.5)`, or a timeline:

```python
from framelock import FocusStep, ZoomRamp
OakD(automations=[
    ZoomRamp(1.0, 2.0, start=5, end=10),    # smooth digital zoom
    FocusStep([(0, 120), (8, 200)]),         # refocus at t=8s
])
```

Every built-in control has all three shapes — `ZoomRamp`/`ZoomStep`/`ZoomFunc`,
`FocusRamp`/`FocusStep`/`FocusFunc`, `ShutterRamp`/`ShutterStep`/`ShutterFunc`,
`IsoRamp`/`IsoStep`/`IsoFunc` — as conveniences over the generic
`Ramp(control, ...)` / `Step(control, ...)` / `Func(control, fn)` /
`Automation(control, keyframes, mode)`, which drive any control a source
supports.

Digital zoom is a runtime ImageManip center-crop on the color camera
(output size constant, H.264-safe). The intrinsics sidecar automatically
switches to **per-frame** `fx/fy/cx/cy` arrays that track the zoom exactly;
distortion coefficients are crop-invariant and stay valid throughout.

## Automatic offset calibration (audio)

The laptop's microphone doubles as an acoustic sync reference: record it as
a source (a wideband marker plays automatically at recording start — a
pleasant pseudo-random "tss" by default; --chirp-tone mls|velvet|sweep), then
cross-correlate it against the phone's audio track — any shared sound works
(the chirp, a clap, speech). Yields the phone's timeline offset at ~1 ms
precision plus its clock drift:

```bash
uv run framelock --rtsp iphone=rtsp://... --mic -n take1
uv run framelock align recordings/take1
#   label bias +43.10 ms (drift +3.2 ppm ...) -> RtspCamera(offset=-0.0431)
```

Sound is slow (~2.9 ms/m): if the sound source is at different distances
from the phone and the laptop, pass `--distance-phone` / `--distance-mic`
(meters) and optionally `--temperature` (°C, refines the speed of sound) and
the propagation difference is corrected. `--chirp-period N` repeats the
chirp for drift markers across long, quiet takes. Audition all the marker
sounds in **[docs/chirps/](docs/chirps/)** (one .wav per tone) before choosing
via `--chirp-tone` / `Microphone(chirp=...)`.

## Going deeper

- **[docs/SYNCHRONIZATION.md](docs/SYNCHRONIZATION.md)** — the full sync
  model: clock domains (including the macOS steady_clock trap), hardware
  vs software tiers with measured accuracies, the Synchronizer algorithm,
  and the phone-calibration workflow.
- **[docs/OUTPUTS.md](docs/OUTPUTS.md)** — every output file and column:
  sidecars, manifests, and how to join them for offline alignment.
- **[docs/TAGS.md](docs/TAGS.md)** — printed wall fiducials: export,
  print at true scale, affix, and let the rig survey them.
- **[docs/CAMERA_METADATA.md](docs/CAMERA_METADATA.md)** — the GVHMR
  intrinsics-sidecar spec the OAK sidecars target.
- **[examples/](examples/)** — runnable scripts: single OAK, OAK+iPhone
  with acoustic calibration, zoom automations, live `on_tick` control,
  custom sync chirps, offline analysis, mobile rigs (end chirp),
  air conditions / speed-of-sound models (Gavioso 2025 vs Cramer 1993),
  and multi-iPhone rigs (per-phone alignment + pixel-level verify).
- **[docs/MEASUREMENTS.md](docs/MEASUREMENTS.md)** /
  **[docs/CHIRP_BENCHMARK.md](docs/CHIRP_BENCHMARK.md)** — reference-rig
  measurement snapshots; regenerate for any session or machine with
  `framelock report session <dir>` / `framelock report chirps --live`.
- `framelock --help` / `framelock align --help` /
  `framelock preview --help` / `framelock report --help` — full CLI
  reference.

## Development

```bash
uv sync                      # deps (+ dev group: pytest, ruff, pre-commit)
uv run pytest                # full suite (~70 s, synthetic-physics GT tests)
uv run ruff check .          # lint (CI enforces both)
uv run pre-commit install    # once per clone: lint on every commit
uv run framelock preview     # side-by-side playback of the newest session
```
