Scientific quality
Canonical datasets establish numerical parity, non-inferior held-out quality, and coverage of the major representation geometries.
One panel, two contracts
This is a research-orientation artifact, not a redistribution of third-party datasets. Dataset adapters should pin source revisions, preserve upstream splits, and stream or cache original artifacts according to their licenses.
Systems performance
Paired jobs on identical hardware establish throughput, utilization, capacity, compilation, and time-to-quality claims.
Proposed compact panel
The goal is to measure the main signals, not reproduce every MTEB task. Each geometry receives a named evaluator and an explicit primary metric.
| Signal | Evaluator | Initial datasets | Primary metric |
|---|---|---|---|
| Graded symmetric similarity | SimilarityEvaluator | STSBenchmark.v2, SICK-R | Cosine Spearman |
| Pair decisions | PairClassificationEvaluator | Sprint Duplicate Questions | Maximum AP across declared similarities |
| Frozen linear separability | ClassificationProbeEvaluator | Banking77.v2 | Accuracy; macro F1 secondary |
| Unsupervised global geometry | ClusteringEvaluator | Twenty Newsgroups.v2 | V-measure |
| Text retrieval | InformationRetrievalEvaluator | SciFact, NFCorpus, ArguAna | nDCG@10; MRR/MAP/recall secondary |
| Multilingual retrieval | InformationRetrievalEvaluator | MIRACL | Macro-language and per-language nDCG@10 |
| Cross-lingual alignment | Ordinary IR/qrels adapter | FLORES-200 | F1 or retrieval accuracy |
| Image ↔ text | InformationRetrievalEvaluator | Flickr30k | Recall@1/5/10 and nDCG@10 |
| Audio ↔ text | InformationRetrievalEvaluator | AudioCaps | Hit-rate@5 and recall@1/5/10 |
| Video ↔ text | InformationRetrievalEvaluator | MSR-VTT | Recall@1/5/10 and nDCG@10 |
| JEPA transfer | JEPARepresentationEvaluator + frozen probe | ImageNet-1K | Linear-probe top-1; k-NN/collapse secondary |
SICK-R complements STS-B with controlled compositional examples. MIRACL and FLORES make multilingual behavior a first-class requirement. FLORES uses the ordinary query/corpus/qrels geometry—there is no special “translation evaluator.”
Frozen-embedding classification
The encoder is frozen. Representax embeds train, validation, and test examples once; fits only a lightweight linear classifier on train embeddings; selects probe hyperparameters on validation data; and reports held-out labels. This measures how linearly accessible task information is, not how well the encoder can be fine-tuned.
FreezeNo encoder updates.
ExtractCache declared embeddings once.
FitTrain only the pinned linear probe.
EvaluateReport held-out accuracy and macro F1.
Pin normalization, classifier family, regularization grid, class weighting, solver tolerance, maximum iterations, seed, and selection split. Identical embeddings must reproduce the maintained reference evaluator.
Dataset previews
Short excerpts show each data geometry and label meaning. The linked, revision-pinned upstream artifact remains authoritative; these excerpts must not become packaged fixtures.
STSBenchmark.v2
5,749 train · 1,485 validation · 1,362 test after duplicate removal.
- “A girl is styling...” / “A girl is brushing...” → 2.5
- “Men play soccer...” / “Boys are playing...” → 3.6
- “Woman measuring an ankle.” / “Woman measures an ankle.” → 5.0
SICK-R
9,927 test pairs.
- Kids in a yard / boys in a yard → 4.5
- Children inside / kids in a yard → 3.2
- Boys outdoors / kids near a man → 4.7
Sprint Duplicate Questions
101,000 test pairs; 1,000 positives.
- USB modem strength / weak modem signal → duplicate
- DuraMax GPS on/off / turn GPS on or off → duplicate
- USB signal / Nexus software update → not duplicate
Banking77.v2
9,993 train · 3,076 test.
- “How do I locate my card?” → card_arrival
- “My new card has not arrived.” → card_arrival
- “I ordered a card; help.” → card_arrival
Twenty Newsgroups.v2
Ten deterministic 1,000-document clustering samples.
- “Motorola MC143150...” → sci.electronics
- “Windows 3.1 for sale” → misc.forsale
- “Gospel Dating” → alt.atheism
SciFact
1,109 queries · 5,183 abstracts · 339 test qrels.
- 0-D biomaterials claim → stem-cell nanotechnology abstract
- 1,000 Genomes claim → rare-variant abstract
- Abnormal PrP claim → prion survey
NFCorpus
3,237 queries · 3,633 documents · 12,334 test qrels.
- Statins and breast cancer → cholesterol-raft paper
- Autophagy and longevity → caloric-restriction paper
- Dietary alkylphenols → nonylphenol paper
ArguAna
1,406 queries · 8,674 candidate arguments.
- Vegetarianism helps the environment → opposing local-food argument
- Killing animals is immoral → opposing human-rights argument
- Vegetarianism is healthier → opposing balanced-diet argument
MIRACL
- “Wo ist das Gebiet der Irokesen-Indianer in Kanada?”
- “Wie viele Arten von Jellyfish?”
- “Wie groß ist der größte Python?”
FLORES-200
997 development · 1,012 devtest aligned records.
- Four-month-old mice... ↔ German equivalent
- Dr. Ehud Ur, professor... ↔ Spanish equivalent
- Other experts are skeptical... ↔ German equivalent
Flickr30k
1,000 images · 5,000 captions · 5,000 qrels.
- Man with pierced ears...
- Beer-can crocheted hat
- Man with gauges and glasses...
AudioCaps
883 audio queries · 4,411 captions · 4,411 qrels.
- Rattling noise and sharp vibrations
- Rocket, explosion, and fire
- Humming, speech, and laughter
MSR-VTT
879 test videos with audio and captions.
- A person connects something to a system
- A woman creates a fondant baby and flower
- A boy plays Grand Theft Auto 5
ImageNet-1K
Freeze the backbone, extract the declared representation, fit a pinned linear probe on train, and report validation top-1.
- Primary: frozen linear-probe top-1
- Secondary: k-NN accuracy
- Diagnostics: variance, effective rank, covariance condition, SIGReg terms
Licenses marked non-commercial or unspecified require review before examples become durable public fixtures. The evaluator tests should use synthetic/checkpointed oracle values unless redistribution is clearly permitted.
How to interpret the evidence
STS-B remains relevant in 2026 for historical continuity and as a fast test of graded symmetric similarity. Sentence-BERT and SimCSE reported it directly; current model papers increasingly emphasize MTEB/MMTEB category and aggregate scores. It cannot stand in for retrieval, clustering, multilingual, or multimodal quality.
Sentence Transformers does not define universal systems numbers. It supplies maintained implementations, evaluator semantics, and useful recipes. Representax must publish the exact matched jobs and rerun both systems on identical hardware, software, examples, ordering, shapes, precision, optimizer, schedule, batch, and update count.
Published model-card scores can validate quality plumbing. Published throughput from different hardware cannot establish a systems claim.