markrel 🐟 Markov Chain Document Relevance

Why markrel?

🎯

Learns From Your Data

Unlike fixed thresholds, markrel learns P(relevance) from your specific domain. Each similarity bin discovers its own probability.

⚡

Lightning Fast

50,000+ predictions per second on CPU. Perfect for real-time applications and high-throughput pipelines.

🔍

Interpretable

See exactly why each document was selected. Inspect P(relevance) for every similarity bin.

🧩

Embedding Agnostic

Works with BERT, OpenAI, sentence-transformers, or TF-IDF. Use the embeddings that work best for you.

🔗

Calibrates External Scorers

JevCalibrator recalibrates an LLM judge or cross-encoder to your domain's true base rate — same bins, same interpretability. See how.

Benchmarks

Tested on WikiQA dataset (6,165 question-answer pairs)

Embedding Model	Dimensions	F1 Score	AUC	Speed
BGE-M3 ⭐	1024	0.343	0.815	51K/s
RoBERTa-large	1024	0.323	0.828	54K/s
MiniLM-L6	384	0.322	0.799	61K/s

Optimized Configurations

0.370

35 bins, euclidean

Recall

1.000

7 bins, euclidean

Precision

1.000

24 bins, cos+euc

Quick Start

Python

# Install
pip install markrel

# Import and train
from markrel import MarkovRelevanceModel

model = MarkovRelevanceModel(
    metrics=["euclidean"],  # Best single metric
    n_bins=35              # Optimized for F1
)

model.fit(queries, documents, labels)

# Predict relevance
probs = model.predict_proba(new_queries, new_documents)
# [0.82, 0.15, 0.91, ...]

With Modern Embeddings

from sentence_transformers import SentenceTransformer

# Load BGE-M3 (best per benchmarks)
encoder = SentenceTransformer('BAAI/bge-m3')

# Encode your texts
query_emb = encoder.encode(["what is ML?"])
doc_emb = encoder.encode(["machine learning is..."])

# Train with embeddings
model = MarkovRelevanceModel(
    metrics=["euclidean"],
    use_text_vectorizer=False
)
model.fit(query_emb, doc_emb, [1])

Calibrate External Scorers

An LLM judge like Jev is calibrated to its own question, not to your domain's base rate.
JevCalibrator learns P(actually relevant | external score) from your labels.

JevCalibrator

from markrel.integrations import JevCalibrator

# jev_probs: raw P(relevant) from the external scorer
# labels: ground truth the scorer did NOT produce
cal = JevCalibrator(n_bins=20, bin_strategy="quantile")
cal.fit(jev_probs, labels)

cal.predict_proba([0.83])
# [0.61]  <- Jev said 0.83; this domain hits ~61% there

cal.summary()   # per-bin: raw range vs. true rate

Fits on scores alone — no text, no embeddings, no retraining of the scorer.
Full guide →

School Your Documents with
Markov Chains