Metadata-Version: 2.4
Name: meteora-rag
Version: 0.1.0
Summary: Rank-free, rationale-driven evidence selection for retrieval-augmented generation.
Author: Yash Saxena, Ankur Padia, Mandar S. Chaudhary, Kalpa Gunaratna, Srinivasan Parthasarathy, Manas Gaur
License-Expression: CC-BY-4.0
Project-URL: Homepage, https://github.com/YashSaxena21/METEORA
Project-URL: Documentation, https://github.com/YashSaxena21/METEORA#readme
Project-URL: Repository, https://github.com/YashSaxena21/METEORA
Project-URL: Issues, https://github.com/YashSaxena21/METEORA/issues
Project-URL: Paper, https://arxiv.org/abs/2505.16014
Keywords: rag,retrieval,evidence-selection,llm,ranking-free
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Operating System :: OS Independent
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: ATTRIBUTION.md
Requires-Dist: numpy>=1.24
Provides-Extra: sentence-transformers
Requires-Dist: sentence-transformers>=2.7; extra == "sentence-transformers"
Provides-Extra: hf
Requires-Dist: accelerate>=0.30; extra == "hf"
Requires-Dist: torch>=2.2; extra == "hf"
Requires-Dist: transformers>=4.40; extra == "hf"
Provides-Extra: training
Requires-Dist: accelerate>=0.30; extra == "training"
Requires-Dist: datasets>=2.19; extra == "training"
Requires-Dist: peft>=0.10; extra == "training"
Requires-Dist: torch>=2.2; extra == "training"
Requires-Dist: transformers>=4.40; extra == "training"
Requires-Dist: trl>=0.9; extra == "training"
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == "dev"
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: ruff>=0.5; extra == "dev"
Requires-Dist: twine>=5.0; extra == "dev"
Dynamic: license-file

# METEORA

METEORA selects evidence for retrieval-augmented generation (RAG): answering a
question using passages retrieved from documents. It takes a query and candidate
passages, then returns the passages to send to your answer-generation model.

Selection is guided by **rationales**: short search instructions such as “Look
for clauses requiring written consent before assignment.” Rationales can be
supplied by your application or generated by a language model. An optional
verifier checks the selected passages against those instructions before they
reach the answer-generation stage.

The package provides evidence selection, rationale generation, verification, and
training utilities. You connect your own document retriever and final answer
model. See the [paper](https://arxiv.org/abs/2505.16014) for the research method
and benchmark results.

## Installation

Requires Python 3.9 or later. The distribution and CLI are named `meteora-rag`;
the Python module is `meteora_rag`.

Install from PyPI:

```bash
python -m pip install meteora-rag
```

To install from source instead:

```bash
git clone https://github.com/YashSaxena21/METEORA.git
cd METEORA
python -m pip install .
```

To install a downloaded wheel, pass its path to pip.

The base installation needs only NumPy. Optional dependencies add model support:

| Extra | Purpose | Install |
| --- | --- | --- |
| `sentence-transformers` | Semantic embeddings for evidence selection | `python -m pip install "meteora-rag[sentence-transformers]"` |
| `hf` | Hugging Face language models for rationale generation and verification | `python -m pip install "meteora-rag[hf]"` |
| `training` | Direct Preference Optimization (DPO) training | `python -m pip install "meteora-rag[training]"` |

Extras can be combined, for example `python -m pip install "meteora-rag[sentence-transformers,hf]"`.
For installation from a source checkout, replace `meteora-rag` with `.`. Model weights are
loaded separately when a model-backed component is constructed.

## Quick start: find the return deadline

Imagine a customer asks: **“How long do I have to return an unopened item?”**
Your document search has returned five store-policy passages. Only one states
how long the customer has to make a return.

METEORA helps find that supporting passage in two steps:

1. A language model generates a **search rationale** describing the evidence to
   look for, such as a return deadline and any conditions on the item.
2. The selector compares that rationale with the retrieved passages and returns
   the matching evidence.

Install the model dependencies:

```bash
python -m pip install "meteora-rag[sentence-transformers,hf]"
```

The first run downloads a language model and an embedding model. This is a
complete example. From a source checkout, you can also run
`python examples/return_policy.py`.

```python
from meteora_rag import HFRationaleGenerator, MeteoraReranker, SentenceTransformerEncoder

# The question and the passages returned by a document search.
query = "How long do I have to return an unopened item?"
documents = [
    "Unopened items can be returned within 30 days of delivery.",
    "Orders normally ship within two business days.",
    "Customer support is available Monday to Friday.",
    "Gift cards cannot be exchanged for cash.",
    "Payments can be made by credit card or bank transfer.",
]

# Show the language model the desired format using a different question.
rationale_generator = HFRationaleGenerator(
    "Qwen/Qwen2.5-0.5B-Instruct",
    sample_shots=[
        {
            "query": "When will my order arrive?",
            "response": (
                "<rationale_1>[Delivery time] Search for dispatch and delivery timelines. "
                "Flag Instructions: Reject passages that contradict the stated delivery deadline.</rationale_1>"
            ),
        }
    ],
    domain="online store policies",
    num_rationales=1,
    include_flag_instructions=True,
    use_chat_template=True,
    max_new_tokens=160,
    do_sample=False,
)

# Use semantic embeddings to compare the generated rationale with the passages.
reranker = MeteoraReranker(
    SentenceTransformerEncoder("sentence-transformers/all-MiniLM-L6-v2"),
    rationale_generator=rationale_generator,
)

# This call generates the rationale and then selects the supporting evidence.
trace = reranker.trace(query, documents)

for rationale in trace.rationales:
    print("Generated rationale:", rationale.text)
for passage in trace.documents:
    print("Selected passage:", passage)
```

Example output from a model run (generated wording can vary):

```text
Generated rationale: Search for return policy details.
Selected passage: Unopened items can be returned within 30 days of delivery.
```

The other passages concern shipping, support hours, gift cards, and payment
methods. The selected passage gives the return deadline needed to answer the
customer's question: **30 days after delivery**.

The `sample_shots` entry is a formatting example about **delivery**. The model
creates the **return-policy** rationale for the current query when `trace()` is
called. The embedding model compares that generated instruction with the five
passages.

The returned passage is the evidence for answering the customer. Send
`trace.documents` to your answer-generation model. The generated flag instructions
can also be used by the optional verifier described below; this example runs
rationale generation and evidence selection.

For an API example that runs without downloading model weights, see
[`examples/minimal_selection.py`](https://github.com/YashSaxena21/METEORA/blob/main/examples/minimal_selection.py).

## Use METEORA in a retrieval pipeline

Pass the passages from your retriever to one of these methods:

| Method | Returns |
| --- | --- |
| `filter(query, documents, ...)` | A tuple containing the selected original documents |
| `rerank(query, documents, ...)` | A tuple of results with text, scores, ranks, and original documents |
| `trace(query, documents, ...)` | Rationales, selection details, verification decisions, and final results |

Documents can be strings, dictionaries with `text`, `content`, or `page_content`,
`Chunk` objects, or objects with a `page_content` or `text` attribute. Original
document objects are preserved in `filter()` results.

Continuing the quick start:

```python
print(trace.indices)
print(trace.to_dict()["selection"]["details"])
```

By default, results are ordered by score. Set `order="document"` to order them
by chunk index. Selection chooses its own cutoff; `top_k` is an optional upper
limit applied after selection and verification. It does not guarantee exactly
that many results.

Without supplied rationales or a configured rationale generator, the reranker
uses the query itself as one search rationale. This fallback does not call a
language model. Set `fallback_to_query_rationale=False` to require explicit
rationales or a generator.

## Generate rationales with a language model

For a separate example that only generates rationales about commercial contracts,
install the `hf` extra and run:

```bash
python examples/generate_rationales.py "Does the agreement allow termination for non-payment?"
```

The example uses `TinyLlama/TinyLlama-1.1B-Chat-v1.0` by default. Pass `--model`
with a different Hugging Face model ID or a local checkpoint directory. This is
an example model configuration; the package does not bundle a preference-tuned
METEORA checkpoint.

`HFRationaleGenerator` requires example query/response pairs, called
`sample_shots`, to demonstrate the desired output format. Use examples from your
domain. Each response can include a rejection rule inside the rationale block:

```xml
<rationale_1>[Termination] Search for termination clauses triggered by non-payment. Flag Instructions: Reject passages that reverse the stated termination conditions.</rationale_1>
```

Attach an `HFRationaleGenerator` as `rationale_generator=` when constructing a
`MeteoraReranker` to generate rationales for each query automatically. The
[API guide](https://github.com/YashSaxena21/METEORA/blob/main/docs/api.md#generation)
shows a complete configuration. Flag instructions are requested by default;
`include_flag_instructions=False` disables them.

## Verify selected evidence

Verification runs only when you supply `verifier=`. The model receives the query,
rationales, flag instructions, current chunk, and previously accepted evidence.
It must return a mapping or JSON text with both boolean decisions:

```json
{
  "relevant": true,
  "flagged": false,
  "reason": "The passage states the return window for unopened items.",
  "flag_types": []
}
```

This illustrates the response format. A real verifier model must decide these
values for each chunk. The package accepts a chunk only when `relevant` is true
and `flagged` is false. Missing or invalid decisions are rejected by default.
Model connection and execution errors propagate to the caller.

A separate, complete verification example uses the bundled contract passages
and a Hugging Face chat model:

```bash
python -m pip install "meteora-rag[hf]"
python examples/verify_evidence.py --model Qwen/Qwen2.5-0.5B-Instruct
```

The [API guide](https://github.com/YashSaxena21/METEORA/blob/main/docs/api.md#verification)
explains custom model adapters, diagnostics, and verification settings. A model's
judgment depends on its capabilities and the evidence supplied; verification is
not an independent fact-checking service.

The small model above demonstrates the request/response interface. It can produce
valid JSON while making incorrect relevance decisions. Use `--model` to choose a
checkpoint whose decisions you have evaluated for your task.

## Command-line interface

These commands run from the source checkout using the included sample files:

```bash
meteora-rag --help
meteora-rag --version

meteora-rag select \
  --chunks examples/data/chunks.json \
  --rationales examples/data/rationales.txt \
  --expansion-window 0 \
  --output outputs/selection.json
```

To split your own document into chunks of up to 256 words, with 25 overlapping
words between adjacent chunks:

```bash
meteora-rag chunk document.txt --chunk-size 256 --overlap 25 --output outputs/chunks.json
```

`select` uses the word-hashing encoder unless `--encoder-model` is supplied.
`python -m meteora_rag` exposes the same commands. See `meteora-rag select --help`
for all options.

## Train a rationale generator

DPO trains a model to prefer useful rationales over less useful ones. It requires
preferred and rejected rationale examples, or candidates evaluated against known
supporting passages. Raw questions alone are not sufficient training input.

The [training guide](https://github.com/YashSaxena21/METEORA/blob/main/docs/training.md)
describes the input schema, bundled sample data, preparation commands, training,
and loading a trained checkpoint. Data preparation works with the base package;
model training requires the `training` extra.

## How selection works

The Evidence Chunk Selection Engine (ECSE) combines three operations:

1. Pair each rationale with its most similar chunk. Multiple rationales may select
   the same chunk.
2. Average rationale embeddings, compare the average with all chunks, and choose
   a cutoff from changes in similarity scores.
3. Optionally include neighboring chunks to recover nearby context.

`MeteoraReranker` disables neighboring expansion by default because its inputs may
be independent retrieved passages. `MeteoraSelector` defaults to one neighbor on
each side. Expansion uses input positions; use it on an ordered document or
process documents separately. For very small candidate sets, adaptive pooling
may retain all candidates.

The [API guide](https://github.com/YashSaxena21/METEORA/blob/main/docs/api.md)
documents configuration, accepted data formats, outputs, and extension points.

## Examples and development

The [example guide](https://github.com/YashSaxena21/METEORA/blob/main/examples/README.md)
lists complete examples and their dependencies. A short
[Colab guide](https://github.com/YashSaxena21/METEORA/blob/main/docs/colab.md)
provides notebook installation instructions.

See [CONTRIBUTING.md](https://github.com/YashSaxena21/METEORA/blob/main/CONTRIBUTING.md)
for development and tests, and
[PUBLISHING.md](https://github.com/YashSaxena21/METEORA/blob/main/PUBLISHING.md)
for release instructions. The `Experiments/` directory contains archived research
programs with separate data and environment requirements; these are not included
in the installable package or release archives.

## License

METEORA is licensed under
[CC BY 4.0](https://creativecommons.org/licenses/by/4.0/).
See [LICENSE](https://github.com/YashSaxena21/METEORA/blob/main/LICENSE) and
[ATTRIBUTION.md](https://github.com/YashSaxena21/METEORA/blob/main/ATTRIBUTION.md)
for attribution details. External models, datasets, and libraries retain their
respective licenses.

## Citation

```bibtex
@inproceedings{saxena2026ranking,
  title={Ranking Free {RAG}: Replacing Re-ranking with Selection in {RAG} for Sensitive Domains},
  author={Yash Saxena and Ankur Padia and Mandar S. Chaudhary and Kalpa Gunaratna and Srinivasan Parthasarathy and Manas Gaur},
  booktitle={Forty-third International Conference on Machine Learning},
  year={2026},
  url={https://arxiv.org/abs/2505.16014}
}
```
