Metadata-Version: 2.5
Name: islkit
Version: 0.1.0
Summary: Indian Sign Language recognition from MediaPipe landmarks: features, a dual-branch TCN, and a live recognition service
Project-URL: Homepage, https://github.com/jbrathwa/islkit
Project-URL: Issues, https://github.com/jbrathwa/islkit/issues
Author: Jayraj
License-Expression: MIT
License-File: LICENSE
Keywords: accessibility,isl,landmarks,mediapipe,sign-language,tcn
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Image Recognition
Requires-Python: <3.13,>=3.12
Requires-Dist: mediapipe==0.10.18
Requires-Dist: numpy<2,>=1.26
Requires-Dist: torch>=2.5
Provides-Extra: train
Requires-Dist: pandas>=3.0.5; extra == 'train'
Requires-Dist: pyarrow>=25.0.1; extra == 'train'
Requires-Dist: xgboost>=3.4.1; extra == 'train'
Description-Content-Type: text/markdown

# islkit

Indian Sign Language (ISL) recognition from MediaPipe Holistic landmarks.

islkit turns camera frames into glosses. It covers every step: a landmark
feature encoder, a dual-branch temporal convolutional network (TCN), a loader
for the INCLUDE dataset, live inference with a confidence-gated decline, and a
small HTTP/SSE recognition service. It runs on CPU and is built to work on small
ARM boards as well as laptops.

```
camera → MediaPipe Holistic → RawFrame → encode_clip (T×352) → TCN → gloss | None
```

## Install

```sh
pip install islkit            # inference and the recognition service
pip install "islkit[train]"   # + pandas, pyarrow, xgboost for INCLUDE and the baseline
```

Requires **Python 3.12**. `mediapipe` is pinned to `0.10.18`, which also pins
`numpy<2`. That is the last MediaPipe release that keeps `mp.solutions` and
still runs on ARMv8.0-A (Cortex-A53) boards.

On macOS, XGBoost needs `brew install libomp`. Don't import `torch` and
`xgboost` in the same process: each bundles its own libomp, and the process
aborts with `OMP: Error #15`. `import islkit` keeps torch lazy for this reason.

## Design

**Features, not raw landmarks.** Holistic's flattened output has 1,662 values,
and 85% of them are face mesh. The encoder reduces each frame to **183 floats**
(352 with velocity):

| Block                          | Dims |
|--------------------------------|------|
| Hand-local shapes (2 × 21 × 3) | 126  |
| Wrist positions in body frame  | 6    |
| Upper-body pose (11 landmarks) | 33   |
| Non-manual scalars from face   | 4    |
| Validity mask                  | 14   |

The encoder uses three coordinate frames rather than one normalisation. Body
frame: origin at the shoulder midpoint, scaled by shoulder width. Hand-local
frame: origin at the wrist, scaled by the wrist-to-middle-MCP distance. Wrist
position is recovered separately in the body frame. This keeps handshape
separate from location.

**Principles the code enforces:**

- **Hand slots are geometric.** Slot 0 is the hand nearer the dominant-side
  shoulder. MediaPipe's handedness label flips under occlusion and is never read.
- **Mask, never zero-fill.** A missing hand is not a hand at the origin. A
  validity bit is carried per part and multiplied through the network.
- **The label map is frozen.** `LabelMap` is saved beside the weights. Class
  order rebuilt from a directory listing silently shifts indices.
- **Checkpoints are tied to the encoder.** A checkpoint records an encoder
  fingerprint, and `SignRecogniser` refuses to serve it through a different
  encoder.
- **Store raw landmarks.** Recordings hold raw MediaPipe output, so they can
  be re-encoded when the normalisation changes.
- **The model can say "I don't know".** Below the confidence threshold a
  prediction is `None`. For an accessibility device, silence is better than a
  confident wrong answer.

## Usage

### Encode a clip

```python
from islkit import encode_clip
from islkit.infer import HolisticExtractor

frames = []
with HolisticExtractor() as extractor:
    for frame_bgr in video_frames:          # BGR numpy arrays, e.g. from cv2
        _, raw, _ = extractor.process(frame_bgr)
        frames.append(raw)

clip = encode_clip(frames, T=48)            # (48, 352) float32
```

### Recognise a sign

```python
from islkit import SignRecogniser

recogniser = SignRecogniser("classifier.pt", threshold=0.6)  # labels_*.json beside it
prediction = recogniser.classify(frames)
print(prediction.gloss, prediction.confidence, prediction.top3)
```

`prediction.gloss` is `None` when the model declines.

### Train

```python
from islkit import build_model, load_include
from islkit.model import fit

data = load_include("path/to/isl-mediapipe-holistic-landmarks")   # data.X: (N, 48, 352)
model = build_model(n_classes=len(data.label_map))
fit(model, data.X, data.y)
```

`load_include` reads the Kaggle
[`indian-sign-language-mediapipe-holistic-landmarks`](https://www.kaggle.com/datasets/swaptr/indian-sign-language-mediapipe-holistic-landmarks)
dump of INCLUDE and caches the encoded arrays. That dump records no signer or
session, so a class-stratified k-fold over whole clips is the only honest split
available. It still leaks signers, so **report numbers from it as optimistic**.
For your own recordings, use leave-one-session-out: `build_dataset` returns a
`sessions` array for this.

`replace_head` and `freeze_backbone` support fine-tuning a pretrained backbone
on a smaller vocabulary. Always report per-class F1 (`per_class_f1`), not just
aggregate accuracy.

### Run the recognition service

```python
import threading
from islkit.infer import ClipStore, HolisticExtractor, SignRecogniser
from islkit.pipeline import CameraSource, RecognitionPipeline
from islkit.server import EventHub, make_server

recogniser = SignRecogniser("classifier.pt")
hub = EventHub()
pipeline = RecognitionPipeline(
    recogniser=recogniser,
    source_factory=lambda: CameraSource(0, "1280x720"),
    on_event=hub.publish,
    extractor_factory=HolisticExtractor,
    store=ClipStore(),
    on_pause=hub.clear_take,
)
threading.Thread(target=pipeline.run, daemon=True).start()
make_server(pipeline, hub).serve_forever()     # 127.0.0.1:9978
```

| Route           | Does                                   |
|-----------------|----------------------------------------|
| `GET /results`  | Server-sent events: tracking, takes, predictions |
| `POST /capture` | Start or pause capture                 |
| `GET /health`   | Pipeline and subscriber status         |

`islkit.view.make_view_server` serves an optional annotated MJPEG debug view
on a separate port.

Pretrained weights are not shipped with the package.

## Guides and experiments

- [`docs/`](docs/README.md): how the features, dataset, baseline, pretraining and fine-tuning
  work, each argued from committed results.
- [`experiments/`](experiments/README.md): the scripts that produce those results, runnable from
  a clone of this repository.

## Development

```sh
uv sync --extra train
uv run pytest
uv run ruff check .
```

The tests cover the properties that would silently break recognition: encoder
invariances, mask gating, label-map and checkpoint round-trips, and the
saved-clip layout.

## License

MIT. See [LICENSE](LICENSE).
