Metadata-Version: 2.4
Name: llmranker
Version: 0.2.0
Summary: LLM-based ranking and reasoning algorithms for search and recommendation
Project-URL: Homepage, https://github.com/shangeth/llmranker
Project-URL: Repository, https://github.com/shangeth/llmranker
Project-URL: Issues, https://github.com/shangeth/llmranker/issues
Author: Shangeth Rajaa
License-Expression: MIT
License-File: LICENSE
Keywords: litellm,llm,ranking,recommendation,reranking,search
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing :: Indexing
Requires-Python: >=3.9
Requires-Dist: litellm>=1.40.0
Requires-Dist: numpy>=1.23
Requires-Dist: pandas>=1.5
Requires-Dist: scikit-learn>=1.1
Requires-Dist: scipy>=1.9
Requires-Dist: tenacity>=8.2.0
Provides-Extra: dev
Requires-Dist: pytest-cov>=4.1; extra == 'dev'
Requires-Dist: pytest>=7.4; extra == 'dev'
Requires-Dist: ruff>=0.4; extra == 'dev'
Description-Content-Type: text/markdown

# llmranker

**LLM-based ranking and reasoning algorithms for search and
recommendation.** Currently includes pointwise, pairwise, listwise,
setwise, and tournament-style (TourRank) ranking, with more strategies
planned (see [`ROADMAP.md`](https://github.com/shangeth/llmranker/blob/main/ROADMAP.md)),
implemented on top of [LiteLLM](https://github.com/BerriAI/litellm) so the
same code runs against OpenAI, Gemini, Anthropic, Azure, Bedrock, local
Ollama models, or any of the 100+ providers LiteLLM supports.

[![PyPI](https://img.shields.io/pypi/v/llmranker.svg)](https://pypi.org/project/llmranker/)
[![License](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/shangeth/llmranker/blob/main/LICENSE)
[![CI](https://github.com/shangeth/llmranker/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/shangeth/llmranker/actions/workflows/ci.yml)
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/shangeth/llmranker/blob/main/examples/quickstart.ipynb)

```python
from llmranker import Candidate, LLMConfig, SetwiseRanker

ranker = SetwiseRanker(LLMConfig(model="gpt-4o-mini"), num_child=4, k=5)

candidates = [
    Candidate(id="1", text="A budget hostel in the city center."),
    Candidate(id="2", text="A five-star beachfront resort with a spa."),
    Candidate(id="3", text="A family-run guesthouse near the old town, kid-friendly."),
]

result = ranker.rank(query="affordable, family friendly, near historical sites", candidates=candidates)
print([c.id for c in result])  # ['3', '1', '2']
```

## What this is

Before you fine-tune a ranking model or build an embedding index, you can
often just *ask* an LLM which candidate is more relevant to a query. This
package is a toolkit of strategies for doing exactly that: scoring,
comparing, sorting, or tournament-ranking a list of candidates with any
LLM, no training required. Each strategy is grounded in published IR/NLP
research, cited per-strategy below and in full under [Citing the
underlying research](#citing-the-underlying-research):

| Strategy | How it works | LLM calls | Notes |
|---|---|---|---|
| **Pointwise** | Score each candidate independently (0-10) | `O(n)` | Cheapest, but ignores relative preference between candidates |
| **Pairwise** | Repeatedly ask "A or B?", sort via heapsort/bubblesort/allpairs | `O(n log n)` to `O(n²)` | Simple, robust comparisons; optional self-consistency bias-checking |
| **Setwise** | Ask "which of these `k` is best?", sort via `k`-ary heapsort/bubblesort/insertion | `O(n log n / log k)` | Fewer calls than pairwise for the same sort, longer prompts |
| **Listwise** | Ask the LLM to output a full ranking of a sliding window at once | `O(n / step)` | Fewest calls, but degrades as window size grows |
| **TourRank** | Group candidates like a sports tournament, LLM picks winners per group, repeat over several stages and tournament runs, sum points | More calls, ensembled over multiple runs | Most robust to candidate input order; see [TourRank paper](https://arxiv.org/abs/2406.11678) |

All five are zero-shot: no training data, no fine-tuning, no embeddings.
You give it a query and a list of candidates, it gives you a ranked list.

## Why LLM-based ranking

- **No training data.** New inventory, a new market, or a one-off internal
  tool rarely comes with click/purchase logs to train a ranker on.
- **Captures compositional, natural-language preference.** "Family
  friendly, near historic sites, not on the beach" is a conjunction of soft
  constraints that keyword search can't express and embedding search tends
  to blur together.
- **Cheap at the scale that matters for reranking.** You're not ranking
  your whole catalog with an LLM; you're reranking the top-k (dozens, not
  millions) that a cheap first-pass retrieval already narrowed down.

See [`examples/hotel_recommendation/`](https://github.com/shangeth/llmranker/tree/main/examples/hotel_recommendation) for
the full worked example this README's numbers come from, and
[`examples/`](https://github.com/shangeth/llmranker/tree/main/examples) for RAG document reranking, product search, and
multi-provider comparisons.

## Install

```bash
pip install llmranker
```

Set whichever provider's API key you're using as an environment variable:
LiteLLM reads the standard ones automatically (`OPENAI_API_KEY`,
`GEMINI_API_KEY`, `ANTHROPIC_API_KEY`, ...). See [LiteLLM's provider
docs](https://docs.litellm.ai/docs/providers) for the full list, including
self-hosted/local options that need no key at all.

## Quickstart

Prefer an interactive walkthrough? Open [`examples/quickstart.ipynb`](https://colab.research.google.com/github/shangeth/llmranker/blob/main/examples/quickstart.ipynb) in Colab.

```python
from llmranker import Candidate, LLMConfig, PairwiseRanker

ranker = PairwiseRanker(LLMConfig(model="gpt-4o-mini"), strategy="heapsort", k=10)

candidates = [Candidate(id=str(i), text=doc) for i, doc in enumerate(my_documents)]
result = ranker.rank(query="my search query", candidates=candidates)

for c in result:
    print(c.id, c.score)
```

## Swap providers in one line

Every ranker takes an `LLMConfig`, whose `model` field is a [LiteLLM model
string](https://docs.litellm.ai/docs/providers). Nothing else about your
code changes:

```python
LLMConfig(model="gpt-4o-mini")                     # OpenAI
LLMConfig(model="gemini/gemini-1.5-flash")         # Google Gemini
LLMConfig(model="claude-3-5-sonnet-20241022")      # Anthropic
LLMConfig(model="azure/my-deployment-name")        # Azure OpenAI
LLMConfig(model="bedrock/anthropic.claude-3-sonnet-20240229-v1:0")  # AWS Bedrock
LLMConfig(model="ollama/llama3")                   # local, via Ollama
```

See [`examples/multi_provider_swap.py`](https://github.com/shangeth/llmranker/blob/main/examples/multi_provider_swap.py).

## Choosing a strategy

Rough guidance, in order of what to reach for first:

- Start with **setwise** (`num_child=4-8`, `strategy="heapsort"`), the best
  cost/quality tradeoff for most use cases.
- If you want the simplest possible mental model (and don't mind more LLM
  calls), use **pairwise**.
- If latency matters more than call count and your candidate list is small
  (fits in one window), use **listwise**.
- Use **pointwise** when you need a standalone relevance score per
  candidate (e.g. for thresholding "is this even relevant at all") rather
  than just a ranking, or when `n` is large and you can't afford
  comparisons at all.
- Use **TourRank** when the order `candidates` arrives in is unreliable (or
  you don't have one) and you want a result that doesn't depend on it. It's
  more expensive than setwise, but explicitly designed to be robust to
  input order, unlike listwise's sliding window.
- If cost is the constraint and your candidate list is long, don't run an
  expensive strategy over everything: cascade a cheap ranker (pointwise) to
  narrow the field, then an expensive one (setwise) to carefully re-rank
  just the survivors. See [Cascading](#cascading-cheap-then-expensive).

## Concurrency

Every ranker takes a `max_concurrency` param (default `5`) that controls
how many LLM calls run at once via a thread pool. Calls are parallel by
default, and `max_concurrency=1` forces fully sequential behavior.

It only speeds up strategies whose calls don't depend on each other's
results:

| Strategy | Parallelized by `max_concurrency`? |
|---|---|
| `PointwiseRanker` | Yes: every candidate is scored independently |
| `PairwiseRanker(strategy="allpairs")` | Yes: every comparison is independent |
| `PairwiseRanker(strategy="heapsort"/"bubblesort")` | No: each comparison's outcome determines the next one |
| `SetwiseRanker` (any strategy, incl. `"insertion"`) | No: same reason, n-ary |
| `ListwiseRanker` | No: each window's input is the previous window's output |
| `TourRankRanker` | Yes, within a stage: every group's LLM call is independent of the others; stages and tournament runs themselves stay sequential |

For the non-parallelizable strategies, `max_concurrency` is accepted for
constructor-signature consistency but genuinely does nothing; that's
documented on each class rather than silently ignored.

```python
# fast: dispatches all scoring calls in parallel, up to 5 at once
PointwiseRanker(LLMConfig(model="gpt-4o-mini"))

# more parallel, if your provider/plan can take it
PointwiseRanker(LLMConfig(model="gpt-4o-mini"), max_concurrency=15)

# fully sequential, useful on a strict rate limit (e.g. a free tier)
PointwiseRanker(LLMConfig(model="gpt-4o-mini"), max_concurrency=1)
```

If you're hitting rate limits (`429`s) on a free or low tier, lower
`max_concurrency` rather than relying on retries alone. The built-in
retry/backoff in `llmranker.llm.call_llm` handles occasional transient
errors, but it won't save you from a provider that's rejecting bursts of
concurrent requests outright.

## Quality: reasoning, self-consistency, structured output

How hard a ranker works to get a reliable judgment is controlled by three
params on every ranker, kept separate from the `LLMConfig` that controls
which model it's talking to: `reasoning`, `num_samples`, and
`structured_output`.

```python
ranker = SetwiseRanker(LLMConfig(model="gpt-4o-mini"), reasoning=True)
```

### Reasoning

`reasoning=True` asks the model to think step by step before giving its
final answer, shown to help across a 2025 wave of reasoning-reranker
papers (Rank1, Rank-R1, and others). This is a *prompting* technique, not
a switch to a dedicated reasoning model; it works with any chat model. (If
you want to route to an actual reasoning-capable model instead, that's
just a model string, e.g. `LLMConfig(model="o1-mini")`, orthogonal to this
flag.) It doesn't change how many calls are made, only prompt and
completion content: expect longer, more expensive completions. A low
default `max_tokens` on some providers can truncate a reasoning chain
before it reaches the final answer; raise it via
`LLMConfig(extra_kwargs={"max_tokens": ...})` if you see that happen.

### Reducing position bias with `num_samples`

LLMs have a documented bias toward whichever candidate happens to be
listed first (or second, model-dependent) in a pairwise/setwise prompt,
independent of actual content. `num_samples` repeats each judgment that
many times and combines the results (mean for pointwise scores, majority
vote for pairwise/setwise choices, a Borda-style merge for listwise
rankings) instead of trusting a single call. On `PairwiseRanker` and
`SetwiseRanker`, each sample also randomly reassigns which candidate lands
on which label before asking, which cancels position bias as a side
effect: it's no longer tied to a fixed slot, just noise the majority vote
averages out.

```python
ranker = PairwiseRanker(LLMConfig(model="gpt-4o-mini"), num_samples=5)
```

This costs `num_samples` calls per judgment instead of 1, dispatched in
parallel via the same `max_concurrency` every ranker already uses, so it
adds spend rather than wall-clock time. `num_samples` only helps at
`LLMConfig(temperature=...)` above `0`: at the default `temperature=0.0`
every repeat returns the same answer, so `PointwiseRanker` logs a warning
if you raise `num_samples` without also raising `temperature`.
`TourRankRanker` has its own repeated-sampling mechanism
(`num_tournaments`) and ignores `num_samples`.

### Structured output

`structured_output=True` uses LiteLLM's normalized JSON-schema
`response_format` instead of regex-parsing free text, for providers that
support it:

```python
ranker = SetwiseRanker(LLMConfig(model="gpt-4o-mini"), structured_output=True)
```

If a model still returns malformed JSON despite the schema, parsing falls
back to the same regex parser used when `structured_output` is off, rather
than raising. `reasoning` and `structured_output` can't both be enabled at
once: reasoning needs free text ending in a final-answer marker, while
strict JSON-schema mode needs the entire completion to be the JSON
payload, leaving no room for reasoning text.

## Multi-criteria scoring

`PointwiseRanker` can score named sub-criteria separately instead of one
holistic judgment, then combine them — useful for compositional queries
("family friendly, near historical sites, affordable") where you want to
know *why* a candidate scored the way it did, or want explicit control
over which constraint matters most, rather than leaving that blend to
whatever the model implicitly does with a single score. Pass a `criteria`
dict or `"auto"`:

```python
# weighted sum: you name the criteria and their relative weight
# (weights don't need to sum to 1, they're normalized internally)
ranker = PointwiseRanker(
    LLMConfig(model="gpt-4o-mini"),
    criteria={"price_fit": 0.5, "location_fit": 0.3, "family_friendly": 0.2},
)

# priority-hierarchical: "high" mathematically dominates any possible
# combination of "medium"/"low", so a candidate can't compensate for
# failing a high-priority criterion by scoring well on lower ones
ranker = PointwiseRanker(
    LLMConfig(model="gpt-4o-mini"),
    criteria={"family_friendly": "high", "price_fit": "medium", "location_fit": "low"},
)

# auto: the model extracts the criteria from the query itself, combined
# with equal weight, so there are no criteria names to maintain per
# domain, at the cost of not choosing them yourself
ranker = PointwiseRanker(LLMConfig(model="gpt-4o-mini"), criteria="auto")
```

Off by default (`criteria=None`), identical behavior to plain holistic
scoring. Costs exactly the same as holistic scoring — one call per
candidate either way, since every named criterion is scored together in a
single response — except `"auto"` mode, which adds exactly one extra call
per `rank()` (not per candidate) to extract the criteria first; if
extraction produces nothing parseable, it falls back to holistic scoring
for that call rather than raising. `rank()`'s output candidates carry the
breakdown in `Candidate.metadata["criteria_scores"]` (merged with any
metadata already on the input candidate), so you can see per-criterion
scores, not just the combined one; `score()` keeps returning a plain
`float` and re-extracts on every call in `"auto"` mode, so prefer `rank()`
when scoring multiple candidates against the same query that way.

Composes normally with `reasoning` and `num_samples`; the existing
`reasoning`+`structured_output` restriction still applies, inherited
rather than a new rule.

## Cascading (cheap-then-expensive)

`CascadeRanker` composes two already-configured rankers instead of being a
new ranking algorithm itself: a cheap one narrows a long candidate list
down, then an expensive one carefully re-ranks just the survivors
(FrugalGPT-style cascading). Each stage keeps its own model and
reasoning/`num_samples`/`structured_output` settings, exactly as if it
were used standalone; `CascadeRanker` only owns how many survive the
first stage:

```python
from llmranker import CascadeRanker, LLMConfig, PointwiseRanker, SetwiseRanker

ranker = CascadeRanker(
    narrow=PointwiseRanker(LLMConfig(model="gpt-4o-mini")),
    refine=SetwiseRanker(LLMConfig(model="gpt-4o"), num_child=4),
    narrow_to=10,
)
result = ranker.rank(query="my search query", candidates=candidates)
```

`ranker.total_calls` / `total_prompt_tokens` / `total_completion_tokens`
sum both stages, and `ranker.config` reports the `refine` stage's config
(whichever model actually produced the final ranking) — it plugs into
`compare_rankers` (see [Evaluation & benchmarking](#evaluation--benchmarking))
just like any other ranker.

## Use case: hotel recommendation

The flagship example lives in
[`examples/hotel_recommendation/`](https://github.com/shangeth/llmranker/tree/main/examples/hotel_recommendation). It
reranks 7 hotels against natural-language guest preferences like *"family
friendly hotel with kids, close to historical places, not right on the
beach."* This is exactly the kind of compositional, subjective query that
trips up keyword and embedding search but an LLM reading full descriptions
handles naturally.

```bash
cd examples/hotel_recommendation
python run.py
```

It runs all five strategies against the same query and candidates and
prints a side-by-side comparison of ranking quality, LLM calls, tokens,
estimated cost, and latency using `llmranker.compare_rankers`.

## More use cases

- [`examples/rag_document_reranking.py`](https://github.com/shangeth/llmranker/blob/main/examples/rag_document_reranking.py):
  rerank RAG retrieval results before they go into a prompt, so context
  budget goes to passages that actually answer the question instead of
  merely-related near-duplicates.
- [`examples/product_search_reranking.py`](https://github.com/shangeth/llmranker/blob/main/examples/product_search_reranking.py):
  e-commerce search reranking against multi-constraint natural-language
  intent (price, fit, use case).
- Other good fits: job/candidate matching, content and media
  recommendation, support ticket triage, lead scoring: anywhere you have
  a short list of candidates and a query or profile to rank them against.

## Customizing for your domain

Every ranker accepts an `item_label` (used in the default prompts, e.g.
"hotel", "product", "document", ...) and an optional `system_prompt`
override if you want full control over the wording:

```python
ranker = SetwiseRanker(
    LLMConfig(model="gpt-4o-mini"),
    item_label="job candidate",
    system_prompt="You are a hiring assistant ranking candidates against a job description...",
)
```

## Evaluation & benchmarking

```python
from llmranker import RankingMetrics, compare_rankers

metrics = RankingMetrics()
metrics.get_metrics(true_ranking=["b", "a", "c"], predicted_ranking=["a", "b", "c"])
# {'ndcg': ..., 'mrr': ..., 'mae': ..., 'spearman': ..., 'kendall_tau': ...}

report = compare_rankers([ranker_a, ranker_b], query, candidates, true_ranking)
# pandas DataFrame: ranking quality + LLM calls/tokens/cost/latency, side by side
```

`true_ranking` is a ground-truth ordering of candidate ids (best to worst),
if you have one, e.g. from human labels or a held-out click log.

## API reference

| Module | Contents |
|---|---|
| `llmranker.types` | `Candidate(id, text, score, metadata)`, `Ranker` (structural protocol) |
| `llmranker.llm` | `LLMConfig`, `call_llm`, `truncate_to_tokens`, `estimate_cost` |
| `llmranker.rankers` | `PointwiseRanker`, `PairwiseRanker`, `SetwiseRanker`, `ListwiseRanker`, `TourRankRanker`, `CascadeRanker`; each takes `reasoning`, `num_samples`, `structured_output` |
| `llmranker.metrics` | `RankingMetrics` (NDCG, MRR, MAE, Spearman, Kendall's Tau) |
| `llmranker.benchmark` | `compare_rankers` |
| `llmranker.prompts` | Default prompt templates, plus `extract_final_answer`/`reasoning_suffix` for the `reasoning` flag |
| `llmranker.structured` | JSON-schema builders/parsers backing `structured_output` |
| `llmranker.criteria` | `resolve_weights` and text parsers backing `PointwiseRanker`'s `criteria` param |

Every ranker implements `rank(query, candidates) -> list[Candidate]` and
tracks `total_calls` / `total_prompt_tokens` / `total_completion_tokens`
after each call.

## Contributing

Issues and PRs welcome. Run tests with:

```bash
pip install -e ".[dev]"
pytest
```

Tests run entirely offline against a fake LiteLLM backend, so no API key
is needed to contribute.

See [`ROADMAP.md`](https://github.com/shangeth/llmranker/blob/main/ROADMAP.md) for what's researched but not built yet,
and why.

## Citing this package

If `llmranker` itself is useful to you, please cite the repository:

```bibtex
@misc{Rajaa_llmranker,
author = {Rajaa, Shangeth},
title = {{llmranker: LLM-based ranking and reasoning algorithms for search and recommendation}},
url = {https://github.com/shangeth/llmranker}
}
```

For a citation pinned to the exact version/commit you used, use GitHub's
"Cite this repository" button in the sidebar (APA or BibTeX) instead of the
snippet above: it reads [`CITATION.cff`](https://github.com/shangeth/llmranker/blob/main/CITATION.cff) live off whatever's
checked out, so it's always accurate without anyone needing to hand-update
a version number in this README.

## Citing the underlying research

If you use one of the ranking strategies implemented here, please also cite
the paper(s) behind it:

```bibtex
@article{zhuang2023setwise,
  title={A Setwise Approach for Effective and Highly Efficient Zero-shot Ranking with Large Language Models},
  author={Zhuang, Shengyao and Zhuang, Honglei and Koopman, Bevan and Zuccon, Guido},
  journal={arXiv preprint arXiv:2310.09497},
  year={2023}
}

@inproceedings{podolak2025setwiseinsertion,
  title={Beyond Reproducibility: Advancing Zero-shot LLM Reranking Efficiency with Setwise Insertion},
  author={Podolak, Jakub and Peri{\'c}, Leon and Jani{\'c}ijevi{\'c}, Mina and Petcu, Roxana},
  booktitle={Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year={2025}
}

@inproceedings{chen2025tourrank,
  title={TourRank: Utilizing Large Language Models for Documents Ranking with a Tournament-Inspired Strategy},
  author={Chen, Yiqun and Liu, Qi and Zhang, Yi and Sun, Weiwei and Ma, Xinyu and Yang, Wei and Shi, Daiting and Mao, Jiaxin and Yin, Dawei},
  booktitle={Proceedings of the ACM Web Conference 2025},
  year={2025}
}
```

## License

MIT, see [LICENSE](https://github.com/shangeth/llmranker/blob/main/LICENSE).
