Metadata-Version: 2.4
Name: tdw-rag
Version: 0.1.0
Summary: Temporal decay weighting for RAG. Rerank retrieval by similarity x document age, with adstock-derived decay kernels.
Project-URL: Homepage, https://github.com/mohit-luthra/tdw-rag
Project-URL: Repository, https://github.com/mohit-luthra/tdw-rag
Project-URL: Issues, https://github.com/mohit-luthra/tdw-rag/issues
Author: Mohit Luthra
License: MIT
License-File: LICENSE
Keywords: adstock,information-retrieval,langchain,llamaindex,rag,reranking,retrieval,temporal,vector-search
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Indexing
Requires-Python: >=3.9
Requires-Dist: numpy>=1.21
Provides-Extra: demo
Requires-Dist: chromadb>=0.4; extra == 'demo'
Requires-Dist: matplotlib>=3.5; extra == 'demo'
Requires-Dist: sentence-transformers>=2.2; extra == 'demo'
Requires-Dist: streamlit>=1.30; extra == 'demo'
Provides-Extra: dev
Requires-Dist: build>=1.0; extra == 'dev'
Requires-Dist: pytest-cov>=4; extra == 'dev'
Requires-Dist: pytest>=7; extra == 'dev'
Requires-Dist: ruff>=0.4; extra == 'dev'
Requires-Dist: twine>=5.0; extra == 'dev'
Provides-Extra: langchain
Requires-Dist: langchain-core>=0.2; extra == 'langchain'
Requires-Dist: pydantic>=2; extra == 'langchain'
Provides-Extra: llamaindex
Requires-Dist: llama-index-core>=0.10; extra == 'llamaindex'
Requires-Dist: pydantic>=2; extra == 'llamaindex'
Provides-Extra: real
Requires-Dist: scikit-learn>=1.0; extra == 'real'
Description-Content-Type: text/markdown

# tdw-rag

**Temporal decay weighting for retrieval augmented generation.**

Vector search answers *what is most similar*. Most real questions ask *what is true now*. `tdw-rag` closes that gap by weighting similarity with a decay curve over document age, using kernels borrowed from adstock modelling in marketing mix models.

```bash
pip install tdw-rag
```

[![PyPI](https://img.shields.io/pypi/v/tdw-rag.svg)](https://pypi.org/project/tdw-rag/)
[![Python](https://img.shields.io/pypi/pyversions/tdw-rag.svg)](https://pypi.org/project/tdw-rag/)
[![Tests](https://github.com/mohit-luthra/tdw-rag/actions/workflows/ci.yml/badge.svg)](https://github.com/mohit-luthra/tdw-rag/actions)
[![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)

**[Live demo](https://mohit-luthra.github.io/tdw-rag/)** — drag the half-life and watch the ranking correct itself.

---

## The problem

Your knowledge base has four versions of the pricing page. They are almost word for word identical, because they describe the same thing. The only difference is the number, and the number changed three times.

```
query: "what does the pro plan cost right now"

cosine similarity ranking
  0.84  2021-03-11  Pricing guide: the Pro plan is 12 dollars per seat per month.   <-- retrieved
  0.82  2023-08-02  Pricing guide: the Pro plan is 18 dollars per seat per month.
  0.79  2026-05-14  Pricing guide: the Pro plan is 25 dollars per seat per month.   <-- correct
```

The oldest document wins, and it wins *because* it is oldest: it has been edited more, linked more, and phrased more canonically. Your model then states a five year old price as current fact, fluently and with no hedging. Nothing in the stack is broken. Embeddings have no notion of time, so retrieval cannot express the constraint that actually matters.

Metadata filters are the usual answer and they are too blunt. `date > 2025-01-01` throws away a document that is old *and* correct, and it needs a threshold you do not know in advance.

## The idea

Score each document by similarity **and** by a decay weight on its age:

```
final_score = similarity × decay(age_days)
```

This is adstock. In marketing mix modelling, an impression served eight weeks ago still drives sales today, at a fraction of its original strength, and the shape of that fraction is a fitted curve. A document is the same object. It was true when written, it is partially true now, and the decay rate depends on the domain. Pricing pages decay in weeks. API references decay in quarters. A proof of the Pythagorean theorem does not decay at all.

The whole library is that one line, plus the parts that make it survive contact with a real corpus.

## Quick start

```python
from tdw_rag import TemporalReranker

docs = [
    {"id": "d1", "score": 0.84, "date": "2021-03-11", "text": "Pro plan: 12 dollars per seat."},
    {"id": "d2", "score": 0.82, "date": "2023-08-02", "text": "Pro plan: 18 dollars per seat."},
    {"id": "d3", "score": 0.79, "date": "2026-05-14", "text": "Pro plan: 25 dollars per seat."},
]

reranker = TemporalReranker("power_law", half_life_days=180, beta=1.0)

for r in reranker.rerank("what does the pro plan cost right now", docs):
    print(f"{r.final_score:.3f}  {r.date}  {r.text}")
```

```
0.547  2026-05-14  Pro plan: 25 dollars per seat.
0.116  2023-08-02  Pro plan: 18 dollars per seat.
0.070  2021-03-11  Pro plan: 12 dollars per seat.
```

Every step stays inspectable:

```python
print(reranker.explain("what does the pro plan cost right now", docs))
```

```
query: "what does the pro plan cost right now"
kernel: power_law(half_life_days=180, beta=1, floor=0)  half-life=180.0d
gate: 1.00  effective strength: 1.00
--------------------------------------------------------------------
Pro plan: 25 dollars per seat.   [3 -> 1] sim=0.790 x decay=0.693 = 0.5474  (80d old)
Pro plan: 18 dollars per seat.   [2 -> 2] sim=0.820 x decay=0.141 = 0.1157  (1096d old)
Pro plan: 12 dollars per seat.   [1 -> 3] sim=0.840 x decay=0.084 = 0.0703  (1970d old)
```

## Results

Run it yourself, offline, in about two seconds:

```bash
tdw-rag benchmark
```

The built-in benchmark is 13 queries over 463 candidate documents. Ten topics each state a claim across four eras in near-identical wording, so similarity alone cannot resolve them. Relevance is graded, not binary: the current era scores 3, the previous era 1, everything older 0. Returning last year's answer is not catastrophically wrong, it is quietly wrong, and only graded relevance measures that. Three timeless control queries are included, where the correct answer is the *oldest* document in the pool.

| config | ndcg@10 | recall@10 | precision@5 | mrr | mean_age@10 | stale@5 |
|---|---|---|---|---|---|---|
| baseline (cosine only) | 0.6794 | 0.8590 | 0.4154 | 0.7115 | 311.0 | 0.5846 |
| exponential h=30 | 0.8550 | 0.7949 | 0.4462 | 0.9615 | 201.7 | 0.5538 |
| exponential h=180 | 0.9274 | 0.9744 | 0.6769 | 0.9308 | 175.2 | 0.3231 |
| weibull h=180 k=2 | 0.8792 | 0.8077 | 0.6000 | 0.9286 | 150.4 | 0.4000 |
| **power_law h=180 b=1** | **0.9522** | **1.0000** | **0.7231** | 0.9487 | 195.9 | **0.2769** |
| cliff h=180 | 0.8511 | 0.7179 | 0.5385 | 0.9279 | 147.0 | 0.4615 |
| weibull + rrf fusion | 0.8170 | 0.8333 | 0.4769 | 0.8974 | 147.6 | 0.5231 |

Best configuration: **nDCG@10 0.679 → 0.952, a 40% relative gain**, with `stale@5` (the share of the visible top five that is wrong) falling from 0.58 to 0.28.

`mean_age@10` is reported because it is a diagnostic, not a target. Driving it to zero means you built a recency sort and threw away your archive.

### Three things the benchmark taught me

**1. Heavy tails beat sharp cutoffs.** The power law kernel wins, and `cliff` — the kernel that best imitates a metadata date filter — is among the worst, costing 14 points of recall against baseline. Demote old documents; do not delete them.

**2. Gentle decay beats aggressive decay.** A grid search over 368 configurations (`tdw-rag fit`) put `gated_multiply` at strength 0.25 in all of the top ten. Plain multiplication drives a five year old document's weight to near zero, so a strong match becomes unreachable rather than merely demoted. This finding changed the library's defaults.

**3. Aggressive decay has a real cost, and it is recall.** With `strength=1.0` and a 7 day half-life, recall@10 falls *below* baseline. This is asserted as a test in `tests/test_tdw_rag.py::test_aggressive_decay_can_hurt_recall`, so the README cannot quietly drift away from the code.

A caveat worth stating plainly: this benchmark is synthetic. It isolates the temporal failure mode cleanly, which makes it a good measurement of the mechanism and a weak estimate of the gain on your corpus. The next section runs the same comparison on real, dated text, and reaches a materially different conclusion about which kernel to use.

## Results on a real corpus

The synthetic benchmark writes its own documents and simulates its own similarity scores. Both are fair criticisms, so there is a second benchmark that removes both.

```bash
pip install "tdw-rag[real]"     # scikit-learn, for the TF-IDF retriever
tdw-rag benchmark --real
```

The numbers below use scikit-learn's TF-IDF. Without it the package falls back to a pure-numpy implementation so the benchmark still runs on a bare install, but it is a weaker retriever and every row lands lower. The CLI says so when it happens.

The corpus is **242 GitHub release notes from 13 projects** (polars, duckdb, ruff, pydantic, transformers, scikit-learn, airflow and others), spanning December 2022 to August 2026, snapshotted 2026-08-01. Real prose written by real maintainers, carrying real publication timestamps. Similarity is TF-IDF cosine computed at query time, not a number drawn from a distribution.

Every release from every project goes into one pool. For each project the query is *"what changed in the latest release of X"*, its newest release scores 3, its previous release scores 1, everything else scores 0.

That design is what makes the result meaningful, because neither signal can win alone:

| method | ndcg@10 | recall@10 | mrr |
|---|---|---|---|
| recency only (pure date sort) | 0.1713 | 0.2308 | 0.1622 |
| similarity only (TF-IDF cosine) | 0.1745 | 0.3846 | 0.2538 |
| exponential h=30 | 0.3435 | 0.5000 | 0.3910 |
| power_law h=90 b=1 | 0.3934 | 0.6538 | 0.4506 |
| exponential h=90 | 0.4209 | 0.7308 | 0.4406 |
| weibull h=90 k=2 | 0.4398 | 0.7692 | 0.4288 |
| **cliff h=90 (fitted)** | **0.4675** | **0.8077** | 0.4146 |

**nDCG@10 0.175 → 0.468, a 168% relative gain**, and recall@10 more than doubles. Sorting by date scores 0.171 and sorting by similarity scores 0.175; combining them scores 2.7 times either one. The two weak signals are close to orthogonal, which is the entire argument for multiplying them rather than choosing between them.

Absolute numbers are low because the task is hard: two relevant documents out of 242, retrieved with TF-IDF. Treat the deltas as the result, not the levels.

### The finding I did not want

**The two benchmarks disagree about which kernel is best, and they disagree strongly.**

| | synthetic | real |
|---|---|---|
| best kernel | `power_law` (heavy tail) | `cliff` (sharp edge) |
| best combination | `gated_multiply` | `multiply` |
| best strength | 0.25 (gentle) | 0.50 |
| worst kernel | `cliff` | `power_law` was mid-table |

On synthetic data I concluded that heavy tails beat sharp cutoffs and that gentle decay beats aggressive decay. On real release notes, the kernel that best imitates a metadata date filter wins outright, and moderate decay beats gentle decay.

Both results are correct about their own corpus, and the reason is structural. Release notes have a genuine cliff: a version is current or it is superseded, with no gradient in between. The synthetic topics decay smoothly by construction, so a smooth kernel fits them. Had I shipped only the synthetic benchmark, I would have confidently recommended the wrong default for this corpus.

This is why the library ships `tdw_rag.fit` and refuses to recommend a universal kernel. The right decay curve is an empirical property of your corpus, not a property of the method. Fit it:

```bash
tdw-rag fit --real
# best: cliff(half_life_days=90, steepness=12) via multiply @ strength=0.5
#       ndcg@10=0.4675 (368 trials)
```

### Refreshing the corpus

The snapshot ships with the package so results are reproducible offline. To pull current data:

```bash
tdw-rag fetch-corpus                 # optionally: --token $GITHUB_TOKEN
tdw-rag benchmark --real
```

Numbers will drift as projects release. That is the corpus behaving correctly.

## The kernels

Every kernel maps age in days to a weight in `[0, 1]`, and every kernel is parameterised by the half-life you actually care about rather than an opaque rate constant.

| kernel | shape | use when |
|---|---|---|
| `exponential` | memoryless, constant proportional loss | the safe default, equivalent to geometric adstock |
| `weibull` | `shape<1` dumps value early with a long tail, `shape>1` holds flat then falls off a shoulder | documentation that stays authoritative then goes obsolete at a release |
| `power_law` | heavy tailed, never reaches zero | news and reference corpora, where old material should be demoted but stays reachable |
| `delayed_peak` | rises, peaks at `peak_days`, then decays | breaking events, where the first report is thinner than the analysis a week later |
| `cliff` | flat, then a logistic edge | a hard boundary such as a launch, migration, or policy change |
| `constant` | no decay | the baseline your comparison needs |

```bash
tdw-rag curve --kernel weibull --half-life 90 --shape 2
```

```
weibull(half_life_days=90, shape=2, floor=0)   half-life=90.2 days

    0d  1.000  ##################################################
   14d  0.983  #################################################
   30d  0.926  ##############################################
   60d  0.735  #####################################
   90d  0.500  #########################
  180d  0.063  ###
  365d  0.000
```

`delayed_peak` is the one with no equivalent in current RAG tooling. The assumption everywhere else is that newest is best. For anything contested or still developing, it is not.

## The temporal gate

Decaying every query is a mistake. *"Explain how cosine similarity works"* should not prefer a blog post from last Tuesday over a textbook from 2011.

The gate estimates how time-sensitive a query is and scales decay strength by it, so one retriever handles both kinds of question without a config change.

```python
from tdw_rag import HeuristicGate

gate = HeuristicGate()
gate("what is the latest pricing")          # 0.95
gate("explain how cosine similarity works") # 0.05
```

The default is lexical: zero dependencies, deterministic, and free per query, which matters when it runs inside every retrieval. Swap in a model when you need it:

```python
from tdw_rag import LLMGate, TemporalReranker

gate = LLMGate(client_fn=lambda prompt: my_llm.complete(prompt))
reranker = TemporalReranker("power_law", half_life_days=180, gate=gate)
```

`LLMGate` caches per query and falls back to the heuristic on any failure, so a rate limit degrades your ranking instead of breaking your retrieval.

## Combining similarity and decay

Multiplying is the obvious move and it is often wrong, because cosine similarity is not calibrated. Four strategies, set with `combine_strategy`:

| strategy | formula | behaviour |
|---|---|---|
| `gated_multiply` *(default)* | `sim × (floor + (1-floor) × decay)`, `floor = 1 - strength` | demotes old documents without deleting them |
| `multiply` | `sim × decay^strength` | aggressive, can zero out anything past a few half-lives |
| `linear` | `(1-s) × norm(sim) + s × decay` | recency can outvote similarity outright |
| `rrf` | reciprocal rank fusion of both orderings | ignores score scale, best when your embedding model returns a narrow similarity band |

## Fitting instead of guessing

Nobody knows their half-life. Two ways to stop making one up:

```python
from tdw_rag import suggest_half_life, fit

# Before you have labels: read the corpus age distribution.
suggest_half_life([d["date"] for d in corpus])   # e.g. 214.0

# Once you have labelled queries: search the space.
result = fit(labelled_queries, metric="ndcg@10")
print(result)
# best: weibull(half_life_days=30, shape=0.6) via gated_multiply @ strength=0.25
#       ndcg@10=0.9234 (368 trials)

reranker = result.build()
```

`fit_strength` sweeps decay strength on a fixed kernel, which is the plot to show anyone who asks how sensitive the result is to your tuning.

## LangChain

```bash
pip install "tdw-rag[langchain]"
```

```python
from tdw_rag.integrations.langchain import TemporalDecayRetriever

retriever = TemporalDecayRetriever.from_vectorstore(
    vectorstore,
    kernel="power_law",
    half_life_days=180,
    beta=1.0,
    date_field="published_at",
    k=5,
    fetch_k=40,
)

docs = retriever.invoke("what is the current pricing")
```

Drop it anywhere a retriever goes. Returned documents carry `tdw_similarity`, `tdw_decay_weight`, `tdw_final_score`, `tdw_age_days`, and `tdw_rank_delta` in their metadata, so you can debug a ranking rather than guess at it.

**`fetch_k` matters more than it looks.** Decay can only promote a document that was retrieved. If the fresh answer sits at similarity rank 25 and you fetch 10, no kernel saves you. Roughly ten times `k` is a sane starting point.

## LlamaIndex

```bash
pip install "tdw-rag[llamaindex]"
```

```python
from tdw_rag.integrations.llama_index import TemporalDecayPostprocessor

postproc = TemporalDecayPostprocessor.build(
    kernel="weibull", half_life_days=90, shape=2.0, date_field="published_at",
)

engine = index.as_query_engine(similarity_top_k=40, node_postprocessors=[postproc])
```

## Documents with no date

Real corpora are missing dates. `missing_date` controls the policy:

- `"neutral"` *(default)* — the median weight of the batch, so undated documents neither win nor lose
- `"keep"` — full weight, when undated usually means evergreen
- `"penalise"` — the minimum weight in the batch, when undated usually means unmaintained

Dates are parsed leniently: ISO strings, `DD-MM-YYYY`, `May 1, 2026`, `datetime` objects, and Unix timestamps in seconds or milliseconds. Anything unparseable is treated as missing rather than raising.

## Interactive demo

`docs/index.html` is a single self-contained file. Drag the half-life and the ranking reorders live, with each document plotted at its own age on the decay curve. The JavaScript kernels are checked against the Python implementations, so the demo cannot drift from the library.

```bash
python -m http.server 8000 --directory docs
```

For real embeddings over a real corpus:

```bash
pip install "tdw-rag[demo]"
streamlit run demo/streamlit_app.py
```

## Where this does not help

Worth being direct about, because a tool that claims to help everywhere helps nowhere:

- **Corpora with no meaningful time axis.** Product catalogues, code symbols, legal statutes that are amended rather than superseded.
- **Corpora where every document is fresh.** Nothing to demote.
- **Missing or unreliable dates.** If your `date` field is the crawl timestamp rather than the publication date, you are decaying noise. Fix the metadata first.
- **When the archive is the point.** Historical research wants the 2011 answer. Set `strength=0`, or trust the gate.
- **As a substitute for reindexing.** If the stale document should not be in the index at all, delete it. Reranking is for corpora where all versions legitimately coexist.

## Background

The approach is described in *Temporal Decay Weighting for Time-Aware Retrieval-Augmented Generation*, which applies adstock decay functions from marketing mix modelling to retrieval scoring. This repository is the reference implementation.

## Development

```bash
git clone https://github.com/mohit-luthra/tdw-rag
cd tdw-rag
pip install -e ".[dev]"
pytest -q          # 85 tests
ruff check .
```

## License

MIT
