Metadata-Version: 2.4
Name: framelock
Version: 0.3.0
Summary: Synchronized multi-camera recording for multi-view pose estimation: OAK-D + iPhone RTSP + mic on one sub-ms clock, with rig registration, corresponded exports, and 3D playback
Project-URL: Repository, https://github.com/ryanrudes/framelock
Project-URL: Documentation, https://github.com/ryanrudes/framelock/tree/main/docs
Author-email: Ryan Rudes <ryanrudes@gmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: arkit,aruco,calibration,camera,depthai,multi-view,oak-d,odometry,pose-estimation,registration,rtsp,synchronization
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: MacOS
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Multimedia :: Video :: Capture
Classifier: Topic :: Scientific/Engineering :: Image Processing
Requires-Python: >=3.12
Requires-Dist: av>=18.0.0
Requires-Dist: depthai>=3.7.1
Requires-Dist: irtsp>=0.2.0
Requires-Dist: numpy>=2.5.1
Requires-Dist: opencv-python>=5.0.0.93
Requires-Dist: rich>=13.3.3
Requires-Dist: sounddevice>=0.5.5
Provides-Extra: location
Requires-Dist: pyobjc-framework-corelocation>=10.0; (sys_platform == 'darwin') and extra == 'location'
Provides-Extra: viz
Requires-Dist: viser>=1.0; extra == 'viz'
Description-Content-Type: text/markdown

# framelock

[![CI](https://github.com/ryanrudes/framelock/actions/workflows/ci.yml/badge.svg)](https://github.com/ryanrudes/framelock/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/framelock)](https://pypi.org/project/framelock/)
[![Python](https://img.shields.io/badge/python-3.12%2B-blue)](https://pypi.org/project/framelock/)
[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)

**Synchronized multi-camera recording for multi-view pose estimation.**
Point an OAK-D, a handful of iPhones, and a microphone at the same
scene; framelock records everything into one session folder on one
sub-millisecond shared clock, solves where every camera *is* in a
single world frame, and hands you corresponded, GVHMR-ready exports.

- **One clock.** Hardware-locked shutters inside an OAK, a verified
  re-implementation of DepthAI's sync across devices, acoustic
  chirp calibration for phones — every sample from every sensor lands
  on one host-monotonic timeline, and every claim is cross-checked by
  an independent estimator before it's trusted.
- **One world frame.** Press `b` mid-take: the laptop shows a fiducial
  board with a live per-camera status strip, and the whole rig —
  static cameras, phones via ARKit odometry, all three OAK cameras
  fused through factory extrinsics — is solved automatically at
  session end (~1 cm / 1° per camera, gravity-leveled). Printed wall
  tags extend it anywhere the board can't reach.
- **Beyond pixels.** iRTSP phones stream ARKit pose, IMU, GPS,
  compass, barometer, and LiDAR *on the video's own clock* (no offset
  estimation — the app's clock design makes it exact); OAK-Ds
  contribute IMU, stereo depth, feature tracks, and thermals.
- **Verified, not hoped.** Gyro-vs-pixels sync checks, gravity
  cross-checks between accelerometers and ARKit, stereo-metric size
  votes, pose-noise grading — `--verify` tells you what the take is
  actually worth.

```bash
pip install framelock            # or: uv add framelock
pip install 'framelock[viz]'     # + 3D playback (viser)
```

One CLI, sensible subcommands — `framelock help` lists them:

| | |
|---|---|
| `framelock` / `framelock record` | record a session (bare flags work: `framelock --shutter-sync ...`) |
| `framelock register` | solve rig registration (board wave / tags / acoustic) |
| `framelock tags` | export printable wall fiducials at true physical scale |
| `framelock correspond` | corresponded exports + end-to-end verification |
| `framelock align` | acoustic label-bias / clock-drift measurement |
| `framelock preview` | synchronized side-by-side playback with transport |
| `framelock viz` | fly through the take in 3D (viser) |
| `framelock report` | session health report |

Beyond video: iRTSP phones stream odometry (fused IMU, ARKit 6-DOF pose,
GPS, compass, barometer, intrinsics, LiDAR depth) and framelock records it
on the **same timeline as the frames** — no offset estimation, the app's
clock design makes it exact (see `docs/OUTPUTS.md`). OAK-Ds contribute
their own IMU, stereo depth, feature tracks, and thermals via
`OakD(data=("imu", "depth", ...))`. Everything lands in the corresponded
export together, `framelock correspond --verify` checks the odometry
against the pixels, and `framelock viz` flies you through the take in 3D.

The whole rig is **registered into one world frame**
(`registration.json`). The primary ritual costs one keypress: press
`b` during a take and the laptop shows a fiducial board fullscreen
with a live per-camera status strip; show it to each camera (the
solver verifies the laptop's stillness from the data) and the rig is
solved automatically at session end — every camera ~1 cm / 1°, phones
via their ARKit odometry with timed drift anchors, all OAK cameras
fused through factory extrinsics, gravity-leveled z-up world (or pick
your own origin: `world_origin="board"` / any device / its Source
object). Board size is cross-voted by independent estimators (ruler >
stereo triangulation / ARKit motion > screen EDID), and every claim is
verified by an independent witness — gravity cross-checks between the
OAK accelerometer and each phone's ARKit measure the solved rotation's
error board-free. Printed **wall tags** ([docs/TAGS.md](docs/TAGS.md))
extend it: `framelock tags grid 0-11 --cols 4 --mm 40`, pin anywhere
fixed, and localized devices survey them automatically — every later
sighting pins phone drift or rescues a camera the board never reached.
The acoustic stop-and-chirp ritual (`g`) remains as a
no-line-of-sight fallback. Registered devices share a metric world
cell in the viz, with the board and tags drawn at their solved poses.

```bash
# The classic single-OAK setup (all flags from the original recorder):
uv run framelock --shutter-sync --exposure 8000 --iso 800 --warmup auto -n take1

# Add an iPhone (any RTSP-server camera app):
uv run framelock --shutter-sync --exposure 8000 --iso 800 \
    --rtsp iphone=rtsp://192.168.1.23:8554/live

# Digital zoom ramp 1x -> 2x between t=5s and t=10s of the recording:
uv run framelock --zoom-ramp 5:10:1.0:2.0
```

The CLI is a thin veneer — everything is a first-class Python API
(see **[examples/](examples/)** for runnable scripts covering each feature):

```python
from framelock import OakD, RtspCamera, Session, Warmup, ZoomRamp

oak = OakD(shutter_sync=True, shutter=1/125, iso=800,
           automations=[ZoomRamp(1.0, 2.0, start=5, end=10)])
phone = RtspCamera("rtsp://192.168.1.23:8554/live", name="iphone")

with Session("recordings", name="take1", warmup=Warmup.AUTO) as session:
    session.record(oak, phone)
```

Options are typed `StrEnum`s (`VideoFormat.MP4`, `Transport.TCP`,
`ExternalSync.MASTER`, `Control.ZOOM`, `ChirpKind.VELVET`, `Warmup.AUTO`) —
IDE-completable and validated, while plain strings keep working everywhere.
Dynamic control is first-class too: `Func("zoom", lambda t: ...)` drives a
control with any function of time, and `session.record(..., on_tick=...)`
runs arbitrary per-tick logic — `apply()` is rate-safe (per-control
resolution filtering built into every source), so both can emit values
naively (see `examples/04_live_control.py`).

Previews are optional (`ui=False`, or `preview=False` per source) and render
as one CCTV-style composite window — labeled tiles, REC indicator, "no
signal" placeholders. Start/stop is console-driven: `warmup=Warmup.MANUAL`
streams and settles until **you** press `r` in the terminal; `q` stops
(with the end chirp when `end_chirp=True`); `session.request_start()` /
`request_stop()` are the programmatic equivalents (thread-safe).

## Output layout

```
recordings/take1/
  session.json               manifest: sources, clock map, settings, accuracy notes
  sync_groups.csv            cross-device alignment (per group: each stream's seq + ts)
  registration.json          every device's pose in ONE world frame (+ board, tags)
  audio_alignment.json       measured per-source label bias / clock drift
  corresponded/              equal-frame-count videos + pose-per-frame exports
  oak/
    CAM_A.mp4  CAM_A.mcap  CAM_A.intrinsics.json  CAM_A.timestamps.csv
    CAM_B.*    CAM_C.*     calibration.json
    depth.mkv  features.csv  imu_*.csv  system.csv    with data=(...)
  iphone/
    video.mp4  video.timestamps.csv
    imu.csv  pose.csv  gnss.csv ...   when the iRTSP app streams them
```

Every video has a `*.timestamps.csv` (file frame index ↔ sequence number ↔
host-monotonic + epoch time) and `sync_groups.csv` joins them across devices,
so post-hoc alignment never depends on the live estimates. Full column
reference for every file: **[docs/OUTPUTS.md](docs/OUTPUTS.md)**.

```bash
uv run framelock viz recordings/take1   # fly through the take in 3D:
# camera frusta with their images, phone trajectories from ARKit odometry,
# play/pause/scrub/step/speed, click any camera for its full details
# (needs the viz extra: uv sync --extra viz)
```

## Synchronization: what you actually get

| Link | Mechanism | Accuracy |
|---|---|---|
| OAK mono pair (one device) | hardware FSIN (`shutter_sync=True`) | ~0.01 ms (shutter-locked) |
| OAK color vs monos | shared clock + locked exposure | sub-ms after a few s settling |
| Two OAK devices, one host | dai XLink timesync → host clock | timestamps <1 ms comparable; shutters free-run (±½ frame) unless FSIN-wired (PoE M8 / FFC hardware only — **not** the USB OAK-D) |
| iPhone / RTSP | software only (no hw path exists into a phone) | live: tens of ms (arrival-anchored); post-hoc with sidecars + clap calibration: ~1 frame |

The host-side `Synchronizer` re-implements DepthAI's `dai.node.Sync`
algorithm (verified against depthai-core v3.7.1 source) for any number of
streams, with one deliberate improvement: nearest-match grouping instead of
first-fit, so same-rate streams can't lock into a stable one-frame-off
pairing.

## Camera controls & automations

`OakD` accepts initial `focus`/`exposure` at construction (`zoom=True`
enables the zoom chain, which starts at 1x), live control via
`oak.apply(zoom=1.5)`, or a timeline:

```python
from framelock import FocusStep, ZoomRamp
OakD(automations=[
    ZoomRamp(1.0, 2.0, start=5, end=10),    # smooth digital zoom
    FocusStep([(0, 120), (8, 200)]),         # refocus at t=8s
])
```

Every built-in control has all three shapes — `ZoomRamp`/`ZoomStep`/`ZoomFunc`,
`FocusRamp`/`FocusStep`/`FocusFunc`, `ShutterRamp`/`ShutterStep`/`ShutterFunc`,
`IsoRamp`/`IsoStep`/`IsoFunc` — as conveniences over the generic
`Ramp(control, ...)` / `Step(control, ...)` / `Func(control, fn)` /
`Automation(control, keyframes, mode)`, which drive any control a source
supports.

Digital zoom is a runtime ImageManip center-crop on the color camera
(output size constant, H.264-safe). The intrinsics sidecar automatically
switches to **per-frame** `fx/fy/cx/cy` arrays that track the zoom exactly;
distortion coefficients are crop-invariant and stay valid throughout.

## Automatic offset calibration (audio)

The laptop's microphone doubles as an acoustic sync reference: record it as
a source (a wideband marker plays automatically at recording start — a
pleasant pseudo-random "tss" by default; --chirp-tone mls|velvet|sweep), then
cross-correlate it against the phone's audio track — any shared sound works
(the chirp, a clap, speech). Yields the phone's timeline offset at ~1 ms
precision plus its clock drift:

```bash
uv run framelock --rtsp iphone=rtsp://... --mic -n take1
uv run framelock align recordings/take1
#   label bias +43.10 ms (drift +3.2 ppm ...) -> RtspCamera(offset=-0.0431)
```

Sound is slow (~2.9 ms/m): if the sound source is at different distances
from the phone and the laptop, pass `--distance-phone` / `--distance-mic`
(meters) and optionally `--temperature` (°C, refines the speed of sound) and
the propagation difference is corrected. `--chirp-period N` repeats the
chirp for drift markers across long, quiet takes. Audition all the marker
sounds in **[docs/chirps/](docs/chirps/)** (one .wav per tone) before choosing
via `--chirp-tone` / `Microphone(chirp=...)`.

## Going deeper

- **[docs/SYNCHRONIZATION.md](docs/SYNCHRONIZATION.md)** — the full sync
  model: clock domains (including the macOS steady_clock trap), hardware
  vs software tiers with measured accuracies, the Synchronizer algorithm,
  and the phone-calibration workflow.
- **[docs/OUTPUTS.md](docs/OUTPUTS.md)** — every output file and column:
  sidecars, manifests, and how to join them for offline alignment.
- **[docs/TAGS.md](docs/TAGS.md)** — printed wall fiducials: export,
  print at true scale, affix, and let the rig survey them.
- **[docs/CAMERA_METADATA.md](docs/CAMERA_METADATA.md)** — the GVHMR
  intrinsics-sidecar spec the OAK sidecars target.
- **[examples/](examples/)** — runnable scripts: single OAK, OAK+iPhone
  with acoustic calibration, zoom automations, live `on_tick` control,
  custom sync chirps, offline analysis, mobile rigs (end chirp),
  air conditions / speed-of-sound models (Gavioso 2025 vs Cramer 1993),
  and multi-iPhone rigs (per-phone alignment + pixel-level verify).
- **[docs/MEASUREMENTS.md](docs/MEASUREMENTS.md)** /
  **[docs/CHIRP_BENCHMARK.md](docs/CHIRP_BENCHMARK.md)** — reference-rig
  measurement snapshots; regenerate for any session or machine with
  `framelock report session <dir>` / `framelock report chirps --live`.
- `framelock --help` / `framelock align --help` /
  `framelock preview --help` / `framelock report --help` — full CLI
  reference.

## Development

```bash
uv sync                      # deps (+ dev group: pytest, ruff, pre-commit)
uv run pytest                # full suite (~70 s, synthetic-physics GT tests)
uv run ruff check .          # lint (CI enforces both)
uv run pre-commit install    # once per clone: lint on every commit
uv run framelock preview     # side-by-side playback of the newest session
```
