Metadata-Version: 2.5
Name: quadembed
Version: 0.1.1
Summary: Multimodal embeddings for text, image, audio, and video in one shared vector space, trained on a single consumer GPU
Project-URL: Homepage, https://github.com/mithilai/QuadEmbed
Project-URL: Repository, https://github.com/mithilai/QuadEmbed
Project-URL: Model Weights, https://huggingface.co/Mithil-21/quadembed-nano
Project-URL: Write-up, https://medium.com/@mithilmaske/i-built-a-multimodal-embedding-model-from-scratch-on-an-rtx-4060-text-image-audio-and-video-ab1fef04f1cd
Author: Mithil Maske
License: CC-BY-NC-4.0
License-File: LICENSE
Keywords: audio,clip,contrastive-learning,embeddings,multimodal,retrieval,semantic-search,text-to-image,video
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Multimedia :: Sound/Audio :: Analysis
Classifier: Topic :: Multimedia :: Video
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: huggingface-hub>=0.24
Requires-Dist: numpy>=1.24
Requires-Dist: peft>=0.11
Requires-Dist: pillow>=9.0
Requires-Dist: torch>=2.1
Requires-Dist: torchvision>=0.16
Requires-Dist: transformers>=4.45
Provides-Extra: all
Requires-Dist: librosa>=0.10; extra == 'all'
Requires-Dist: opencv-python>=4.8; extra == 'all'
Requires-Dist: soundfile>=0.12; extra == 'all'
Provides-Extra: audio
Requires-Dist: librosa>=0.10; extra == 'audio'
Requires-Dist: soundfile>=0.12; extra == 'audio'
Provides-Extra: video
Requires-Dist: opencv-python>=4.8; extra == 'video'
Description-Content-Type: text/markdown

# QuadEmbed

Multimodal embeddings for **text, image, audio, and video** in one shared
768-dimensional vector space. Compare any modality against any other with a
dot product.

Trained end-to-end on a single RTX 4060 laptop GPU (8GB VRAM) by freezing
three pretrained encoders and training only two small projection heads,
reproducing the GELATO architecture behind Jina AI's jina-embeddings-v5-omni.

> ⚠️ **Non-commercial license.** QuadEmbed builds on
> `jina-embeddings-v5-text-nano`, which is CC-BY-NC-4.0. That restriction
> carries through to this package and its weights. Research and educational
> use is fine; commercial use is not, without a separate license from Jina AI.

## Install

```bash
pip install quadembed              # text + image + audio
pip install quadembed[video]       # adds video support (OpenCV)
pip install quadembed[all]         # everything, plus audio file loading helpers
```

## Quickstart

```python
from quadembed import QuadEmbed
from PIL import Image

model = QuadEmbed.from_pretrained()   # downloads weights from the Hub on first run

texts = ["a dog running on the beach", "a bowl of ramen noodles"]
text_embeds = model.embed_text(texts)
image_embeds = model.embed_image([Image.open("photo.jpg").convert("RGB")])

# embeddings are L2-normalized, so this is cosine similarity
similarity = text_embeds @ image_embeds.T
for text, score in zip(texts, similarity[:, 0].tolist()):
    print(f"{score:.3f}  {text}")
```

### Audio

Pass mono float32 arrays at 16 kHz (what the frozen Whisper encoder expects):

```python
import soundfile as sf

audio, sr = sf.read("clip.wav")
audio_embeds = model.embed_audio([audio.astype("float32")], sampling_rate=sr)
similarity = model.embed_text(["a dog barking"]) @ audio_embeds.T
```

### Video

```python
model = QuadEmbed.from_pretrained(modalities=("text", "video"))
video_embeds = model.embed_video_file("clip.mp4", num_frames=4)
similarity = model.embed_text(["a person skateboarding"]) @ video_embeds.T
```

### Loading only what you need

Each encoder costs memory and download time. Load a subset:

```python
model = QuadEmbed.from_pretrained(modalities=("text", "vision"))   # skip audio
model = QuadEmbed.from_pretrained(device="cpu")                    # force CPU
```

## How it works

Three frozen encoders, two trained projectors:

| Role | Model (frozen) | Trainable on top |
|---|---|---|
| Text (anchor) | `jinaai/jina-embeddings-v5-text-nano` (239M) | nothing, it defines the space |
| Vision | `google/siglip2-base-patch16-naflex` | vision projector (2.36M params) |
| Audio | `openai/whisper-large-v3` encoder | audio projector (0.98M params) |

Only ~3.3M parameters were ever trained, against nearly a billion frozen ones.
Video needs no encoder or projector of its own: frames are sampled, run
through the vision path, and mean-pooled over time.

Training used bidirectional in-batch InfoNCE (temperature 0.02) plus
Matryoshka representation learning over prefix dims {32, 64, 128, 256, 768},
so truncated embeddings remain usable if you need a smaller index.

## Measured performance

Cross-modal retrieval recall@k on held-out splits, text-query direction:

| Modality | R@1 | R@5 | R@10 | n |
|---|---|---|---|---|
| Image | 13.7% | 68.6% | 81.1% | 1024 |
| Audio | 67% | 97% | 100% | 33 |
| Video | 40% | 86% | 94% | 50 |

Random-chance R@1 on the 1024-candidate image eval is ~0.1%.

These are honest small-scale numbers. Vision was trained on ~172k
image-caption pairs, orders of magnitude less than production embedding
models, and image R@1 is the weakest metric as a result. R@5 and R@10 are
considerably stronger, so this is more useful for candidate retrieval and
reranking than for exact top-1 matching.

## Links

- **Source and training code:** [github.com/mithilai/QuadEmbed](https://github.com/mithilai/QuadEmbed)
- **Model weights:** [huggingface.co/Mithil-21/quadembed-nano](https://huggingface.co/Mithil-21/quadembed-nano)
- **Full write-up:** [I Built a Multimodal Embedding Model From Scratch on an RTX 4060](https://medium.com/@mithilmaske/i-built-a-multimodal-embedding-model-from-scratch-on-an-rtx-4060-text-image-audio-and-video-ab1fef04f1cd)

## Credit

QuadEmbed reproduces the architecture described in Jina AI's GELATO paper
([arXiv:2605.08384](https://arxiv.org/abs/2605.08384)). All credit for the
original architecture and training recipe belongs there. This is an
independent reproduction, not affiliated with or endorsed by Jina AI.
