Metadata-Version: 2.4
Name: fasthamer
Version: 0.8.0
Summary: Realtime HaMeR 3D hand mesh recovery on the Apple Neural Engine: image in, 3D hand mesh (SOMA-X or MANO) + MANO parameters + 3D/2D joints out
Author-email: Vimal Mollyn <vmollyn@andrew.cmu.edu>
License-Expression: MIT
Project-URL: Homepage, https://github.com/VimalMollyn/fasterhamer
Keywords: hand-tracking,mano,soma-x,hamer,coreml,neural-engine,3d-hand-pose
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: MacOS
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Image Recognition
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy
Requires-Dist: opencv-python
Requires-Dist: coremltools>=8.0
Requires-Dist: fasthands>=0.3.0
Provides-Extra: mediapipe
Requires-Dist: mediapipe>=0.10; extra == "mediapipe"
Dynamic: license-file

# fasthamer

Realtime [HaMeR](https://github.com/geopavlakos/hamer) 3D hand mesh recovery on
the Apple Neural Engine. Give it an image, get a 3D hand mesh, MANO
parameters, 3D joints, camera parameters, and 2D joints per hand — a
MediaPipe-Hands-style API, but with a full 3D hand mesh behind it.

- **Fast**: the whole network (ViT-H backbone + MANO head) runs as one CoreML
  program on the ANE and the mesh stage takes ~0.6 ms per hand — ~30 FPS
  end-to-end with two hands at 640px on a fanless M4 MacBook Air, torch-free
  at runtime.
- **Simple**: `pip install fasthamer` → `fasthamer.load()` → `result = hands(rgb)`.
  No accounts, no license forms: the default hand mesh is NVIDIA's
  Apache-licensed [SOMA-X](https://github.com/NVlabs/SOMA-X) hand, regressed
  from HaMeR's MANO parameters by [mano2soma-x](https://github.com/VimalMollyn/mano2soma-x)
  (~2 mm from the MANO mesh). If you have a MANO license you can get the MANO
  mesh instead (`hand_model="mano"`).
- **Complete outputs**: MANO `global_orient` / `hand_pose` / `betas`
  (rotation matrices *and* axis-angle), the hand mesh (2859-vertex SOMA-X or
  778-vertex MANO), 21 3D joints, camera intrinsics + per-hand translation,
  projected 2D joints.
- **Batteries included**: hand detection via
  [fasthands](https://github.com/VimalMollyn/Mediapipe-Hands-PyTorch-CoreML)
  (CoreML/ANE, ~1 ms) and a tiny C mesh rasterizer for overlays (~10x faster
  than pyrender, no GPU/OpenGL).

Requires macOS on Apple Silicon.

## Install

```bash
pip install fasthamer
fasthamer-webcam --mirror                 # live demo: 3D mesh overlay from your webcam
```

That's it. The first `fasthamer.load()` (or `fasthamer-setup`, if you prefer
to pre-download) fetches the CoreML model bundle (~470 MB) and the SOMA-X hand
assets (~40 MB) into `~/.cache/fasthamer`, then compiles the model for your
device (one-time, typically 10-30 s). Every load after that takes a couple of
seconds and works offline.

### Optional: the MANO mesh

`hand_model="mano"` gives you the 778-vertex [MANO](https://mano.is.tue.mpg.de)
mesh instead of the SOMA-X hand. MANO is free for non-commercial research but
license-gated, so fasthamer does not ship it: create an account at
https://mano.is.tue.mpg.de, accept the license there, then

```bash
fasthamer-setup --mano                           # prompts for a path, or Enter to download with your account
fasthamer-setup --mano ~/Downloads/mano_v1_2.zip # or use the file you already have
fasthamer-webcam --mirror --hand-model mano
```

fasthamer either imports a `MANO_RIGHT.pkl` / `mano_v1_2` folder / zip you
already have, or downloads `mano_v1_2.zip` with your MANO account email +
password (sent only to the MPI server, never stored), and converts it once to a
small `.npz` in `~/.cache/fasthamer/mano/`. The same prompt runs on the first
`load(hand_model="mano")` if you skip setup. Non-interactive environments:
point `FASTHAMER_MANO_PATH` at your file, or set `MANO_USERNAME` and
`MANO_PASSWORD`; `python -m fasthamer.mano_download` downloads the zip on its own.

## Quickstart

```python
import cv2
import fasthamer

hands = fasthamer.load()                       # mode="image", hand_model="soma" by default
rgb = cv2.cvtColor(cv2.imread("photo.jpg"), cv2.COLOR_BGR2RGB)
result = hands(rgb)

for hand in result:
    hand.is_right          # bool (left hands: see note below)
    hand.keypoints_2d      # (21, 2)  pixels in the input image
    hand.keypoints_3d      # (21, 3)  meters, hand-centered
    hand.keypoints_camera  # (21, 3)  meters, camera frame (= keypoints_3d + cam_t)
    hand.vertices          # (2859, 3) SOMA-X hand mesh, hand-centered (778 with hand_model="mano")
    hand.soma_joints       # (25, 3)  SOMA-X skeleton joints (None with hand_model="mano")
    hand.cam_t             # (3,)     hand -> camera translation
    hand.global_orient     # (3, 3)   MANO global orientation (rotation matrix)
    hand.hand_pose         # (15, 3, 3) MANO pose  (also .hand_pose_aa / .global_orient_aa)
    hand.betas             # (10,)    MANO shape

result.focal, result.cx, result.cy   # pinhole camera: u = f*X/Z + cx
overlay = hands.render(rgb, result)               # mesh overlay (C rasterizer)
overlay = fasthamer.draw_landmarks(overlay, result)  # 2D skeleton
```

## Video mode

`mode="video"` enables temporal hand tracking in the detector (faster and
smoother than per-frame detection) — feed frames sequentially:

```python
hands = fasthamer.load(mode="video")
while True:
    ok, frame = cap.read()
    result = hands(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB))
```

## Webcam demo

`fasthamer-webcam` (installed with the package) opens your webcam and overlays
the reconstructed meshes in realtime:

```bash
fasthamer-webcam --mirror              # selfie view
fasthamer-webcam --hand-model mano     # MANO mesh instead of SOMA-X (needs fasthamer-setup --mano)
fasthamer-webcam --soma                # MANO vs SOMA-X comparison view (blend/flicker/wire, key 'o')
fasthamer-webcam --skeleton            # 2D joints instead of the mesh
fasthamer-webcam --camera 0            # pick a camera (auto-probed by default)
fasthamer-webcam --stabilize           # lock Right/Left per hand across frames
fasthamer-webcam --camera clip.mp4     # run on a video file instead of a camera
fasthamer-webcam --record out.mp4      # also save the annotated output
fasthamer-webcam --no-display --max-frames 120   # headless benchmark, prints FPS
```

Keys: `q` / ESC quit, `m` toggle mesh, `s` toggle skeleton. It uses the same
low-latency loop as the threaded example below (freshest-frame capture thread
+ worker/display split), so the FPS shown is the real end-to-end rate — it is
capped by your camera's frame rate, and only drops below that when the model
is the bottleneck (hold your hands in view when benchmarking; with no hands
only the ~1 ms detector runs). See `fasthamer-webcam --help` for the full
option list (`--max-hands`, `--alpha`, `--force-handedness`,
`--fasthands-detector`, `--compute-units`, `--width/--height`, ...).

Examples:

- [`examples/image_demo.py`](examples/image_demo.py) — single image in, mesh +
  skeleton overlay out.
- [`examples/webcam_demo.py`](examples/webcam_demo.py) — minimal synchronous
  webcam loop (easiest to read).
- [`examples/webcam_threaded_demo.py`](examples/webcam_threaded_demo.py) —
  **low-latency** webcam demo: a background camera thread always serves the
  freshest frame, and a worker/display split keeps the window responsive while
  the mesh stays aligned to the frame it was computed on. Use this one if the
  simple demo feels laggy (`cv2.VideoCapture` otherwise hands you buffered,
  stale frames). Supports `--no-display --max-frames N` for benchmarking.

## Conventions

- Input images are **RGB uint8** (`H×W×3`).
- 3D outputs use a camera-style frame: x right, y down, z forward, meters.
  `hand.vertices` / `hand.keypoints_3d` are hand-centered; add `hand.cam_t`
  (or use the `*_camera` properties) for camera-frame coordinates. Projection
  uses the pinhole camera in the result (`result.project(pts)`).
- Joint order is MANO/OpenPose-style: wrist, then 4 joints each for thumb,
  index, middle, ring, pinky (`fasthamer.HAND_EDGES` has the skeleton). With
  the SOMA-X hand these are the SOMA skeleton's MCP/PIP/DIP/tip joints (and the
  MANO wrist position); they sit within a few mm of MANO's regressed joints,
  except the MCP row which is ~1 cm further down the palm. `hand.soma_joints`
  has all 25 SOMA joints.
- **Left hands**: HaMeR mirrors left-hand crops, so MANO parameters always
  describe a *right* MANO hand. `vertices`/`keypoints_3d` are already
  un-mirrored. To re-pose a left hand yourself, apply the params to a
  right-hand MANO model and negate x.
- `hands.faces` gives the triangle faces of the active hand model (SOMA-X
  5692, MANO 1538); flip winding for left hands.

## Options

```python
fasthamer.load(
    mode="image",             # or "video"
    max_hands=2,
    hand_model="soma",        # "soma" (SOMA-X hand, no MANO needed) or "mano"
    detector="fasthands",     # detection stack: "fasthands" or "mediapipe"
    fasthands_detector=None,  # model inside fasthands: "whim" or "mediapipe"
    model_dir=None,           # local model bundle dir (skips the download)
    soma_dir=None,            # local SOMA-X asset dir (skips the download)
    mano_path=None,           # hand_model="mano": your MANO_RIGHT.pkl / mano_v1_2 folder / zip
                              #   (default: the copy cached by fasthamer-setup --mano)
    rescale_factor=2.0,       # hand-box padding before cropping
    swap_handedness=False,    # if handedness looks inverted (mirrored inputs)
    stabilize_handedness=False,  # video: lock R/L per hand across frames
    handedness_iou=0.3,       # IoU to treat a detection as the same hand
    handedness_ttl=10,        # drop a track after N unseen frames
    handedness_switch_frames=5,  # switch only after N consecutive disagreements
    force_handedness=None,    # "right" / "left" to pin it outright
    compute_units="CPU_AND_NE",
)
```

### Choosing a hand detector

Two independent choices, and confusingly both offer something called
"mediapipe" — they are not the same thing:

- **`detector`** picks the detection *stack*: `"fasthands"` (default) runs
  MediaPipe Hands ported to CoreML on the Neural Engine, ~1 ms/frame;
  `"mediapipe"` runs Google's official MediaPipe **Tasks** API on
  TFLite/XNNPACK/GPU (needs `pip install "fasthamer[mediapipe]"`, and exposes
  `det_conf` / `track_conf`).
- **`fasthands_detector`** picks the detector model *inside* fasthands
  (requires `fasthands >= 0.4.0`): `"whim"` — the WHIM-fine-tuned
  full-hand-box detector, steadier than the palm detector; `"mediapipe"` —
  the original palm detector. Left as `None`, fasthands' own default (`whim`)
  applies.

```python
hands = fasthamer.load(mode="video", fasthands_detector="mediapipe")
```

The two detector models frame hands differently — on the bundled test image
the boxes are `[408,507,583,921]` (whim) vs `[414,518,584,906]` (palm) — and
that box drives the crop fed to HaMeR, so the choice affects the reconstructed
geometry, not just detection recall. Worth trying both on unusual footage
(egocentric/wrist cameras, heavy foreshortening).

### Handedness stability (video)

Detectors decide Right/Left from a single frame, so on hard footage —
foreshortened, self-occluding hands, e.g. a wrist/egocentric camera — the label
can flicker frame to frame. **This is not cosmetic:** HaMeR is a right-hand
model, so "left" hands are mirror-flipped before the network and un-mirrored
after. A flickered label produces mirror-wrong *geometry* on that frame (we
measure ~200 mm of vertex jump on an otherwise identical frame), which is why
relabeling the output afterwards is not enough.

`stabilize_handedness=True` fixes it upstream of the crop: each hand gets a
persistent IoU-matched track, and a spatially continuous hand keeps the label it
first appeared with. Tracks expire after `handedness_ttl` unseen frames so a
genuinely new hand can re-acquire.

The lock is *hysteretic*, not permanent: a tracked hand only switches its label
once the detector disagrees for `handedness_switch_frames` **consecutive**
frames (default 5). A single agreeing frame resets the streak, so alternating
flicker never accumulates into a switch — but a sustained correction still
wins, so a genuinely misidentified hand recovers instead of staying wrong for
the life of the track. Set `handedness_switch_frames=0` for a permanent lock.

```python
hands = fasthamer.load(mode="video", stabilize_handedness=True)
...
hands.reset()      # clear tracks when the video sequence restarts
```

For rigs where handedness is known and fixed (single-hand egocentric mounts),
`force_handedness="right"` pins every detection outright — a stronger
constraint that takes precedence over the tracker. Both are orthogonal to
`swap_handedness`, which is a global flip applied by the detector.

Environment variables: `FASTHAMER_MODEL_DIR` / `FASTHAMER_SOMA_DIR` point at
a local model bundle / SOMA-X asset directory; `FASTHAMER_ASSETS_URL` /
`FASTHAMER_SOMA_ASSETS_URL` override where they are downloaded from;
`FASTHAMER_MANO_PATH` points at your MANO model (`MANO_RIGHT.pkl`,
`mano_v1_2/`, or `mano_v1_2.zip`); `MANO_USERNAME` / `MANO_PASSWORD` let setup
download it non-interactively.

## How it works

MediaPipe-style palm detection (fasthands, ANE) finds hand boxes; each box is
cropped HaMeR-style and run through a single CoreML mlprogram containing the
ViT-H backbone (192×144 input, interpolated position embeddings) and the MANO
transformer head — 6-bit palettized weights — which outputs the MANO
parameters and camera.

The mesh stage then runs without torch:

- **SOMA-X hand (default).** The
  [mano2soma-x](https://github.com/VimalMollyn/mano2soma-x) ResMLP maps the
  MANO parameters to SOMA-X hand parameters (25 joint rotations, 20 identity
  coefficients, 24 bone scales, root translation); it runs as a CoreML model
  (fp32, ~0.4 ms). SOMA-X's hand model (identity PCA, closed-form skeleton
  fit, forward kinematics, skinning) is re-implemented in C
  (`fasthamer.soma_np` / `soma_stage.c`, ~0.15 ms per hand) and matches the
  reference py-soma-x implementation to 0.05 mm. SOMA-X skins with plain LBS,
  so since the soma-v2 assets (fasthamer 0.8) the regressor also predicts
  learned vertex correctives on the SOMA topology (a `posedirs`/`shapedirs`
  pair, added to the bind-pose rest shape before skinning) that remove the
  LBS pinch at bent joints: the regressed SOMA hand is within 0.15 mm (mean
  vertex distance, 1.3 mm worst vertex) of the MANO mesh HaMeR implies, versus
  ~2 mm / 8 mm before.
- **MANO hand (`hand_model="mano"`).** The MANO mesh (778 vertices) is
  computed in numpy from *your* `MANO_RIGHT.pkl` (`fasthamer.mano`, ~0.5 ms
  per hand, no chumpy/scipy); it matches HaMeR's smplx MANO layer to 1e-7 m.

End-to-end outputs match the reference HaMeR torch pipeline to a few mm
(fp16 + palettization noise). The overlay renderer is a small C rasterizer
with supersampled anti-aliasing, compiled once on first use.

## License

Code: MIT. The CoreML model bundle is converted from the MIT-licensed
[HaMeR](https://github.com/geopavlakos/hamer) checkpoint and contains no MANO
data. The SOMA-X hand model data is NVIDIA's
[SOMA-X](https://github.com/NVlabs/SOMA-X) (Apache-2.0). The mano2soma-x
regressor was trained on HaMeR's training annotations, which come from
research-only datasets and MANO fits, so treat it — like the HaMeR weights
themselves — as a research asset. The MANO model (`hand_model="mano"`) is
licensed by the Max Planck Institute for non-commercial scientific research
and is obtained by *you* from your own account at https://mano.is.tue.mpg.de
under its [license](https://mano.is.tue.mpg.de/license.html); fasthamer only
reads the copy you provide. Cite [HaMeR](https://arxiv.org/abs/2312.05251),
[SOMA](https://arxiv.org/abs/2603.16858) and
[MANO](https://mano.is.tue.mpg.de) in academic work.
