Metadata-Version: 2.4
Name: openmmla
Version: 0.2.0
Summary: OpenMMLA: A unified multimedia platform integrating audio and vision modules for analysis and computer aided instruction.
Author-email: Zaibei Li <zali@di.ku.dk>
License: MIT License
Project-URL: Homepage, https://github.com/ucph-ccs/openmmla
Classifier: Topic :: Multimedia :: Sound/Audio
Classifier: Topic :: Multimedia :: Video
Classifier: Topic :: Education :: Computer Aided Instruction (CAI)
Classifier: Operating System :: MacOS
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3.10
Classifier: License :: OSI Approved :: MIT License
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: asr-base
Requires-Dist: cryptography>=42.0.0; extra == "asr-base"
Requires-Dist: influxdb-client==1.44.0; extra == "asr-base"
Requires-Dist: jiwer==3.0.4; extra == "asr-base"
Requires-Dist: librosa==0.10.2.post1; extra == "asr-base"
Requires-Dist: llvmlite==0.43.0; extra == "asr-base"
Requires-Dist: matplotlib==3.9.2; extra == "asr-base"
Requires-Dist: networkx==3.3; extra == "asr-base"
Requires-Dist: paho-mqtt==2.1.0; extra == "asr-base"
Requires-Dist: pandas==2.2.2; extra == "asr-base"
Requires-Dist: PyAudio>=0.2.14; extra == "asr-base"
Requires-Dist: pydub==0.25.1; extra == "asr-base"
Requires-Dist: pyecharts==2.0.6; extra == "asr-base"
Requires-Dist: pymongo>=4.6.0; extra == "asr-base"
Requires-Dist: pyyaml==6.0.2; extra == "asr-base"
Requires-Dist: redis==5.0.7; extra == "asr-base"
Requires-Dist: tqdm==4.67.1; extra == "asr-base"
Provides-Extra: asr-server-nemo
Requires-Dist: azure-cognitiveservices-speech==1.36.0; extra == "asr-server-nemo"
Requires-Dist: cryptography>=42.0.0; extra == "asr-server-nemo"
Requires-Dist: Cython==3.0.11; extra == "asr-server-nemo"
Requires-Dist: dashscope==1.25.15; extra == "asr-server-nemo"
Requires-Dist: denoiser-compat==0.1.5.dev0; extra == "asr-server-nemo"
Requires-Dist: flask==3.0.3; extra == "asr-server-nemo"
Requires-Dist: gevent==24.2.1; extra == "asr-server-nemo"
Requires-Dist: googletrans==3.0.0; extra == "asr-server-nemo"
Requires-Dist: gunicorn==23.0.0; extra == "asr-server-nemo"
Requires-Dist: huggingface_hub==0.23.5; extra == "asr-server-nemo"
Requires-Dist: modelscope[framework]==1.16.1; extra == "asr-server-nemo"
Requires-Dist: nemo-toolkit[asr]<=1.23.0; extra == "asr-server-nemo"
Requires-Dist: onnxruntime==1.18.1; extra == "asr-server-nemo"
Requires-Dist: openai-whisper==20231117; extra == "asr-server-nemo"
Requires-Dist: pandas==2.2.2; extra == "asr-server-nemo"
Requires-Dist: pyyaml==6.0.2; extra == "asr-server-nemo"
Requires-Dist: rotary-embedding-torch==0.6.4; extra == "asr-server-nemo"
Requires-Dist: silero-vad==5.1.2; extra == "asr-server-nemo"
Requires-Dist: speechbrain==1.0.0; extra == "asr-server-nemo"
Requires-Dist: transformers==4.33.3; extra == "asr-server-nemo"
Requires-Dist: whisperx==3.3.1; extra == "asr-server-nemo"
Provides-Extra: asr-server-wespeaker
Requires-Dist: accelerate; extra == "asr-server-wespeaker"
Requires-Dist: azure-cognitiveservices-speech==1.36.0; extra == "asr-server-wespeaker"
Requires-Dist: cryptography>=42.0.0; extra == "asr-server-wespeaker"
Requires-Dist: dashscope==1.25.15; extra == "asr-server-wespeaker"
Requires-Dist: denoiser-compat==0.1.5.dev0; extra == "asr-server-wespeaker"
Requires-Dist: fire==0.4.0; extra == "asr-server-wespeaker"
Requires-Dist: flake8==3.8.2; extra == "asr-server-wespeaker"
Requires-Dist: flake8-bugbear; extra == "asr-server-wespeaker"
Requires-Dist: flake8-comprehensions; extra == "asr-server-wespeaker"
Requires-Dist: flake8-executable; extra == "asr-server-wespeaker"
Requires-Dist: flake8-pyi==20.5.0; extra == "asr-server-wespeaker"
Requires-Dist: flask==3.0.3; extra == "asr-server-wespeaker"
Requires-Dist: gevent==24.2.1; extra == "asr-server-wespeaker"
Requires-Dist: googletrans==3.0.0; extra == "asr-server-wespeaker"
Requires-Dist: gunicorn==23.0.0; extra == "asr-server-wespeaker"
Requires-Dist: h5py; extra == "asr-server-wespeaker"
Requires-Dist: hdbscan>=0.8.40; extra == "asr-server-wespeaker"
Requires-Dist: huggingface_hub==0.23.5; extra == "asr-server-wespeaker"
Requires-Dist: importlib-metadata; extra == "asr-server-wespeaker"
Requires-Dist: kaldiio==2.17.2; extra == "asr-server-wespeaker"
Requires-Dist: librosa; extra == "asr-server-wespeaker"
Requires-Dist: lmdb==1.3.0; extra == "asr-server-wespeaker"
Requires-Dist: matplotlib==3.5.1; extra == "asr-server-wespeaker"
Requires-Dist: mccabe; extra == "asr-server-wespeaker"
Requires-Dist: modelscope[framework]==1.16.1; extra == "asr-server-wespeaker"
Requires-Dist: numpy==1.22.4; extra == "asr-server-wespeaker"
Requires-Dist: onnxruntime; extra == "asr-server-wespeaker"
Requires-Dist: openai-whisper; extra == "asr-server-wespeaker"
Requires-Dist: pandas==2.2.2; extra == "asr-server-wespeaker"
Requires-Dist: peft; extra == "asr-server-wespeaker"
Requires-Dist: pre-commit==3.5.0; extra == "asr-server-wespeaker"
Requires-Dist: pycodestyle==2.6.0; extra == "asr-server-wespeaker"
Requires-Dist: pyflakes==2.2.0; extra == "asr-server-wespeaker"
Requires-Dist: pypeln==0.4.9; extra == "asr-server-wespeaker"
Requires-Dist: PyYAML==6.0; extra == "asr-server-wespeaker"
Requires-Dist: rotary-embedding-torch==0.6.4; extra == "asr-server-wespeaker"
Requires-Dist: s3prl==0.4.18; extra == "asr-server-wespeaker"
Requires-Dist: scikit-learn; extra == "asr-server-wespeaker"
Requires-Dist: scipy==1.10.0; extra == "asr-server-wespeaker"
Requires-Dist: setuptools<=81.0.0; extra == "asr-server-wespeaker"
Requires-Dist: silero-vad; extra == "asr-server-wespeaker"
Requires-Dist: soundfile; extra == "asr-server-wespeaker"
Requires-Dist: speechbrain==1.0.0; extra == "asr-server-wespeaker"
Requires-Dist: tableprint==0.9.1; extra == "asr-server-wespeaker"
Requires-Dist: torch==2.4.1; extra == "asr-server-wespeaker"
Requires-Dist: torchaudio==2.4.1; extra == "asr-server-wespeaker"
Requires-Dist: tqdm==4.66.3; extra == "asr-server-wespeaker"
Requires-Dist: transformers<5,>=4.43; extra == "asr-server-wespeaker"
Requires-Dist: umap-learn==0.5.6; extra == "asr-server-wespeaker"
Requires-Dist: whisperx==3.3.1; extra == "asr-server-wespeaker"
Provides-Extra: ips-base
Requires-Dist: cryptography>=42.0.0; extra == "ips-base"
Requires-Dist: influxdb-client==1.44.0; extra == "ips-base"
Requires-Dist: matplotlib==3.9.2; extra == "ips-base"
Requires-Dist: networkx==3.3; extra == "ips-base"
Requires-Dist: opencv-python==4.10.0.84; extra == "ips-base"
Requires-Dist: paho-mqtt==2.1.0; extra == "ips-base"
Requires-Dist: pandas==2.2.2; extra == "ips-base"
Requires-Dist: pupil_apriltags==1.0.4.post11; extra == "ips-base"
Requires-Dist: pyecharts==2.0.6; extra == "ips-base"
Requires-Dist: pymongo>=4.6.0; extra == "ips-base"
Requires-Dist: pyyaml==6.0.2; extra == "ips-base"
Requires-Dist: redis==5.0.7; extra == "ips-base"
Requires-Dist: scipy==1.14.0; extra == "ips-base"
Provides-Extra: vfa-base
Requires-Dist: cryptography>=42.0.0; extra == "vfa-base"
Requires-Dist: influxdb-client==1.44.0; extra == "vfa-base"
Requires-Dist: matplotlib==3.9.2; extra == "vfa-base"
Requires-Dist: opencv-python==4.10.0.84; extra == "vfa-base"
Requires-Dist: paho-mqtt==2.1.0; extra == "vfa-base"
Requires-Dist: pandas==2.2.2; extra == "vfa-base"
Requires-Dist: pymongo>=4.6.0; extra == "vfa-base"
Requires-Dist: pyyaml==6.0.2; extra == "vfa-base"
Requires-Dist: redis==5.0.7; extra == "vfa-base"
Requires-Dist: requests==2.32.3; extra == "vfa-base"
Requires-Dist: scipy==1.14.0; extra == "vfa-base"
Provides-Extra: vfa-server
Requires-Dist: cryptography>=42.0.0; extra == "vfa-server"
Requires-Dist: flask==3.0.3; extra == "vfa-server"
Requires-Dist: gevent==24.2.1; extra == "vfa-server"
Requires-Dist: gunicorn==22.0.0; extra == "vfa-server"
Requires-Dist: huggingface-hub==0.24.6; extra == "vfa-server"
Requires-Dist: matplotlib==3.9.2; extra == "vfa-server"
Requires-Dist: numpy==1.26.4; extra == "vfa-server"
Requires-Dist: openai==1.73.0; extra == "vfa-server"
Requires-Dist: opencv-python==4.10.0.84; extra == "vfa-server"
Requires-Dist: Pillow==10.4.0; extra == "vfa-server"
Requires-Dist: pupil_apriltags==1.0.4.post11; extra == "vfa-server"
Requires-Dist: pyyaml==6.0.2; extra == "vfa-server"
Requires-Dist: retina-face==0.0.17; extra == "vfa-server"
Requires-Dist: scikit-learn==1.6.1; extra == "vfa-server"
Requires-Dist: tf-keras==2.19.0; extra == "vfa-server"
Requires-Dist: timm==1.0.15; extra == "vfa-server"
Requires-Dist: torch==2.7.0; extra == "vfa-server"
Requires-Dist: torchvision==0.22.0; extra == "vfa-server"
Provides-Extra: vfa-vllm-runtime
Requires-Dist: qwen-vl-utils==0.0.14; extra == "vfa-vllm-runtime"
Requires-Dist: transformers>=4.57.0; extra == "vfa-vllm-runtime"
Requires-Dist: vllm>=0.11.0; extra == "vfa-vllm-runtime"
Provides-Extra: uber-base
Requires-Dist: cryptography>=42.0.0; extra == "uber-base"
Requires-Dist: influxdb_client==1.44.0; extra == "uber-base"
Requires-Dist: jiwer==3.0.4; extra == "uber-base"
Requires-Dist: matplotlib==3.9.2; extra == "uber-base"
Requires-Dist: networkx==3.3; extra == "uber-base"
Requires-Dist: pandas==2.2.2; extra == "uber-base"
Requires-Dist: pyecharts==2.0.6; extra == "uber-base"
Requires-Dist: pymongo>=4.6.0; extra == "uber-base"
Requires-Dist: pyyaml==6.0.2; extra == "uber-base"
Requires-Dist: redis==5.0.7; extra == "uber-base"
Requires-Dist: scipy==1.14.0; extra == "uber-base"
Provides-Extra: uber-server
Requires-Dist: celery[redis]==5.4.0; extra == "uber-server"
Requires-Dist: flask-cors==4.0.1; extra == "uber-server"
Requires-Dist: flask-socketio==5.3.6; extra == "uber-server"
Requires-Dist: flask==3.0.3; extra == "uber-server"
Requires-Dist: gevent==24.2.1; extra == "uber-server"
Requires-Dist: gunicorn==23.0.0; extra == "uber-server"
Requires-Dist: influxdb_client==1.44.0; extra == "uber-server"
Requires-Dist: matplotlib==3.9.2; extra == "uber-server"
Requires-Dist: networkx==3.3; extra == "uber-server"
Requires-Dist: numpy==2.1.0; extra == "uber-server"
Requires-Dist: pandas==2.2.2; extra == "uber-server"
Requires-Dist: pyecharts==2.0.6; extra == "uber-server"
Requires-Dist: pyyaml==6.0.2; extra == "uber-server"
Requires-Dist: scipy==1.14.0; extra == "uber-server"
Provides-Extra: tui
Requires-Dist: cryptography>=42.0.0; extra == "tui"
Requires-Dist: influxdb-client==1.44.0; extra == "tui"
Requires-Dist: pymongo>=4.6.0; extra == "tui"
Requires-Dist: redis==5.0.7; extra == "tui"
Requires-Dist: textual>=0.80.0; extra == "tui"
Requires-Dist: pyyaml==6.0.2; extra == "tui"
Dynamic: license-file

# OpenMMLA

OpenMMLA is an IoT-based multimodal data collection toolkit for learning analytics. It provides three pipelines, real-time speech recognition with speaker diarization (ASR), AprilTag-based indoor positioning (IPS) and vision-language video frame analysis (VFA), synchronized across base stations, stored in InfluxDB and MongoDB, and shown on a dashboard live and after the session.

- Documentation: https://openmmla.readthedocs.io
- Source code and issues: https://github.com/ucph-ccs/OpenMMLA
- Paper: https://doi.org/10.1145/3706468.3706525

## Installation

This package provides the `openmmla` library and the `mmla` command line. Each pipeline has its own extra, so a machine only installs what it runs:

```bash
pip install "openmmla[asr-base]"      # ASR base station
pip install "openmmla[ips-base]"      # IPS base station
pip install "openmmla[vfa-base]"      # VFA base station
pip install "openmmla[uber-base]"     # analytics and session exports
pip install "openmmla[uber-server]"   # dashboard backend and worker
pip install "openmmla[tui]"           # management console
```

The pipelines and the management console run against a checkout of the repository, which holds the pipeline configuration bundles (`pipelines/`), the shared settings (`config/`) and the Docker compose files of the AI services (`docker/`). The recommended setup is therefore an editable install from the clone:

```bash
git clone https://github.com/ucph-ccs/OpenMMLA.git
cd OpenMMLA
conda create -n tui python=3.10 -y && conda activate tui
pip install -e ".[tui]"
mmla tui
```

See the documentation for the system prerequisites (conda, git, tmux, PortAudio, FFmpeg), the system services (InfluxDB, MongoDB, Redis, Mosquitto, Nginx, dashboard) and the pipeline guides.

## Citation

```bibtex
@inproceedings{10.1145/3706468.3706525,
author = {Li, Zaibei and Yamaguchi, Shunpei and Spikol, Daniel},
title = {OpenMMLA: an IoT-based Multimodal Data Collection Toolkit for Learning Analytics},
year = {2025},
doi = {10.1145/3706468.3706525},
booktitle = {Proceedings of the 15th International Learning Analytics and Knowledge Conference},
pages = {872–879},
}
```
