# Retrieval Pipeline

How `recall()` finds relevant memories.

## Overview

Yadgar uses a multi-stage pipeline: route → expand → retrieve → fuse → gate → rerank → return.

```
query
  │
  ├─► Query routing         classify: temporal / code / relational / comparison / open-domain
  │
  ├─► Query expansion       pseudo-HyDE: embed a synthetic answer, not just the question
  │
  ├─► Parallel retrieval ─────────────────────────────────────┐
  │     Vector ANN search    cosine similarity on embeddings   │
  │     BM25 full-text       keyword matching (SurrealDB FTS)  │
  │     PageRank (PPR)       knowledge graph walk              │
  │     Spreading activation graph activation from seed nodes  │
  │                                                            │
  ├─► WRRF fusion           Weighted Reciprocal Rank Fusion ◄──┘
  │
  ├─► Confidence gate       reject low-confidence result sets
  │
  ├─► Cross-encoder rerank  FlashRank ONNX or GTE-ModernBERT
  │
  ├─► NLI entailment        DeBERTa entailment signal (open-domain only)
  │
  ├─► Multi-passage         cluster overlapping passages into evidence groups
  │
  └─► Adversarial filter    score-gap and diversity checks
```

## Query Routing

Before retrieval begins, the query is classified into a type that selects the optimal retrieval strategy:

| Type | Trigger keywords | Strategy |
|---|---|---|
| **temporal** | yesterday, last week, recently, before, after, ... | Temporal boost stage added; date-range filtering |
| **code** | function, class, bug, fix, refactor, API, ... | Entity graph seeds from code entities |
| **relational** | relationship, connection, related, between, cause, ... | Heavier PPR weighting |
| **comparison** | vs, versus, compare, difference, ... | Dual search: separate retrieval per option |
| **open-domain** | (default) | Full pipeline with NLI reranking |

Set `QUERY_ROUTING_ENABLED=false` to skip routing and always use the full pipeline.

## Query Expansion (pseudo-HyDE)

When `QUERY_EXPANSION_ENABLED=true` (default), the query is embedded twice:
1. The raw query text
2. A synthetic "hypothetical answer" generated by the embedding model

Both embeddings contribute to the candidate pool. This improves recall on queries where the relevant memory is phrased as a statement rather than a question.

## Candidate Retrieval

### Vector Search

Uses SurrealDB's ANN index over 384-dimensional sentence-transformer embeddings. Retrieves `max_results * CANDIDATE_POOL_MULTIPLIER` candidates (default 20×).

Model: `all-MiniLM-L6-v2` by default. Override with `EMBEDDING_MODEL`.

### BM25 Full-Text Search

SurrealDB's built-in full-text search with a custom stop-word list. Matches on the `content` and enriched text fields.

Weight in fusion: `WRRF_FTS_WEIGHT` (default 0.5).

### Personalized PageRank (PPR)

Walks the knowledge graph outward from entities mentioned in the query. Uses damping factor `PPR_DAMPING` (0.85) over `PPR_ITERATIONS` (50) iterations.

Weight in fusion: `WRRF_PPR_WEIGHT` (default 0.5).

### Spreading Activation

Propagates heat from seed entities through the knowledge graph. Activation decays by `GRAPH_SPREADING_DECAY` per hop (default 0.5), up to `GRAPH_SPREADING_MAX_DEPTH` hops.

Weight in fusion: `WRRF_SPREADING_WEIGHT` (default 0.3).

## WRRF Fusion

Weighted Reciprocal Rank Fusion combines all signal lists into a single ranked list:

```
score(d) = Σ_s  weight_s / (k + rank_s(d))
```

where `k = WRRF_K` (default 60) controls how much rank position matters vs weight.

Scores are normalised before fusion using `FUSION_NORM` (default `zscore`). Set to `minmax` or `raw` for alternatives.

Set `COMBMNZ_ENABLED=true` to multiply the fused score by the number of signals that contributed — penalises documents appearing in only one signal list.

## Retrieval Profiles

Three pre-configured profiles trade speed for quality:

| Profile | Signals | Reranking |
|---|---|---|
| `fast` | Vector + BM25 | None |
| `balanced` | Vector + BM25 + PPR + spreading | Cross-encoder (default) |
| `full` | All signals | Cross-encoder + NLI + multi-passage |

Set with `RETRIEVAL_PROFILE` or `yadgar config set retrieval_profile fast`.

## Confidence Gate

After fusion, the result set is evaluated for confidence. If confidence is low, a fallback strategy fires:

- `confidence_fallback_strategy: expand` — expand the candidate pool and re-fuse
- `confidence_fallback_strategy: relax` — lower score thresholds and retry

Gate triggers when:
- Fewer than `CONFIDENCE_MIN_RESULTS` results (default 3), or
- Score spread < `CONFIDENCE_SCORE_SPREAD_THRESHOLD` (default 0.15), or
- Top score < `CONFIDENCE_TOP_SCORE_THRESHOLD` (default 0.5)

Set `CONFIDENCE_GATING_ENABLED=false` to disable.

## Cross-Encoder Reranking

The top `RERANKER_TOP_K` (default 50) candidates are re-scored by a cross-encoder that takes the (query, memory) pair as input — capturing richer interaction than independent embeddings.

Two backends:

| Backend | Model | Notes |
|---|---|---|
| GTE-Reranker | `Alibaba-NLP/gte-reranker-modernbert-base` | Default, higher quality |
| FlashRank | `cross-encoder/ms-marco-MiniLM-L-6-v2` | Fallback, ONNX-accelerated, fast CPU |

Final score: `CE_WEIGHT * cross_encoder_score + (1 - CE_WEIGHT) * fusion_score`

Where `CE_WEIGHT = CROSS_ENCODER_WEIGHT` (default 0.6).

## NLI Entailment Scoring

When `NLI_RERANKING_ENABLED=true` and `NLI_ONLY_FOR_OPEN_DOMAIN=true` (both default), open-domain queries run a DeBERTa-based NLI model that scores whether each candidate *entails* the query.

Score blended in at weight `NLI_WEIGHT` (default 0.3).

## Multi-Passage Evidence Aggregation

When `MULTI_PASSAGE_RERANKING_ENABLED=true`, candidates with > `MULTI_PASSAGE_CLUSTER_OVERLAP_THRESHOLD` (0.3) token overlap are clustered. The cluster's aggregate score is boosted — retrieving a tight cluster of passages that all point to the same fact is stronger evidence than a single isolated passage.

## Adversarial Protection

When `ADVERSARIAL_DETECTION_ENABLED=true`:

- Score gaps > `ADVERSARIAL_SCORE_GAP_THRESHOLD` (0.05) between consecutive results are flagged — an injected memory with an unusually high score is a likely adversarial insertion
- `ADVERSARIAL_DIVERSITY_ENFORCEMENT=true` enforces result diversity to prevent a flood of near-identical injected memories dominating the result set
- Results with confidence < `ADVERSARIAL_MIN_CONFIDENCE` (0.3) are suppressed

## Temporal Retrieval

When a query is classified as temporal or contains date expressions:

1. Memories are scored by temporal proximity to the referenced period
2. `TEMPORAL_BOOST_WEIGHT` (0.4) controls the boost magnitude
3. Exact date matches get `TEMPORAL_EXACT_MATCH_BOOST` (2.0×) multiplier
4. Relevance decays over `TEMPORAL_DECAY_DAYS` (30) days

## Index-Time Enrichment

When `INDEX_ENRICHMENT_ENABLED=true`, memories are expanded at storage time:

| Source | What it adds |
|---|---|
| ConceptNet | Related concepts (IsA, UsedFor, HasProperty, ...) |
| COMET | Commonsense inferences (xAttr, xIntent, xWant) |
| doc2query | Synthetic queries a user might ask for this memory |
| Logic enrichment | Formal logic patterns extracted from content |

Enriched text is appended to the embedding input, improving retrieval of memories via related terms not literally present in the content.

## Directory Scoping (v5.62–v5.65)

`recall()` and `wiki_query()` filter results by `directory_context` before returning them to the caller. This is a separate, earlier gate than the branch filter — it prevents cross-project memory leakage entirely.

### Eligible-set rule

```
eligible = {caller_dir, 'global', '', None}
results = [r for r in results if r["directory_context"] in eligible]
```

`caller_dir` is the absolute project path the caller passed as the `directory` parameter. Results whose `directory_context` is none of the four members above are silently dropped before any scoring or reranking step. The check is implemented in `storage/directory.py::is_directory_eligible(dc, caller_dir)`, which is the single source of truth for this predicate — no other code should replicate the logic.

`'system'` is **not** in the eligible set. It was the primary mis-stamp sink (rows that landed in no real project) and was removed from eligibility in v5.65. Note: `dominant_directory()` still retains `'system'` in its `_SENTINELS` exclusion set, but that is the opposite role — it excludes `'system'` from the directory vote when deriving a stamp. These are unrelated.

### Hard-require `directory`

`recall()` and `wiki_query()` raise `ValueError` if `directory` is absent. There is no `os.getcwd()` fallback: the daemon runs in a container and its CWD is the container filesystem path, not the caller's project path. Callers must supply `directory` explicitly.

### Wiki-blend scoping (v5.65)

Before v5.65, wiki pages surfaced inside `recall()` bypassed `is_directory_eligible()` — cross-project wiki entries leaked into results. Fixed in v5.65: the same predicate now applies to the wiki-blend branch of `recall()` and to the supplement WHERE clause in `hooks/prompt-recall.py` (changed from `directory_context != $dir` to `directory_context IN ('', 'global')` to prevent cross-project supplement injection).

`project_brief()` similarly passes `directory=resolved` to `list_wiki_pages()` (v5.65).

### Quality floor and dedup (v5.62)

Two additional filters applied after direction scoping:

- **Quality floor** — results with a cross-encoder score near zero (controlled by `RECALL_QUALITY_FLOOR` env knob) are dropped. Catches low-signal co-occurrence rows that score ≈ 0 yet survive fusion.
- **Dedup** — repeated identical co-occurrence rows (same content, multiple co-occurrence edges) are collapsed to one result before return.

### `dominant_directory()` helper

Used at write time (v5.64) by `curation/strengthen.py`, `cls_store/promotion.py`, and `sleep_compute/dream.py`. When deriving a `directory_context` stamp for an auto-generated or promoted memory, these modules call `dominant_directory(candidates)` instead of hardcoding `'system'`. The helper excludes sentinel values (`None`, `''`, `'global'`, `'system'`) from the vote and returns the single real project path when unambiguous, or `'global'` for cross-project outputs (dreams).

### SQL builder

`_build_directory_clause(df)` in `storage/directory.py` converts a `DirectoryFilter` dataclass into the SurrealQL `WHERE` fragment. It is the single place where directory predicate SQL is produced.

## Branch-Aware Filter and Boost (v5.0)

`recall()` and `wiki_query()` filter results by git branch after fetch:

```
allowed = {default_branch, None}
if current_branch is not None:
    allowed.add(current_branch)
results = [r for r in results if r["branch"] in allowed]
```

Results where `branch == current_branch` then receive a 1.5× multiplier on `_retrieval_score` and the list is re-sorted descending. Non-git contexts (`current_branch is None`) skip the boost; the filter degenerates to `{default_branch, None}`.

The branch is captured at write time by `memorize`, `anchor`, `checkpoint`, and `wiki_add` via the `_detect_branch(directory)` helper (30s LRU cache). `bootstrap_project` and `update_active_work` intentionally leave the branch unset — project-level state is not branch-scoped.

`wiki_read(slug)` performs a three-step resolution: exact slug on current branch → on default branch → on `branch IS NONE`. First hit wins.

Known v5.x follow-up: filter is applied in Python after the SurrealQL fetch. Moving the filter into the `WHERE` clause eliminates the over-fetch buffer and is tracked for a future minor.

## Pipeline Stages (v5.0)

`recall()` is a thin orchestrator over named pipeline-stage helpers:

| Stage | Purpose |
|---|---|
| `_collect_fts_scores` | FTS BM25 + entity-FTS + COMET expansion |
| `_collect_vector_scores` | KNN vector search; returns `(vector_memory_ids, query_embedding)` |
| `_collect_ppr_scores` | Personalized PageRank from vector seeds |
| `_collect_spreading_scores` | Spreading activation from top vector seeds |
| `_collect_temporal_scores` | Temporal signal weighting |
| `_fuse_scores` | Confidence gating + WRRF / convex fusion |
| `_build_initial_results` | Assemble result list + cross-encoder diversity injection |
| `_apply_rerank_pipeline` | Heuristic → comparison → CE → NLI → multi-passage → profile/belief merge → MMR → trim → adversarial → rules engine → engram links → metacognition |

Behavior is pinned by `yadgar/tests/test_retrieval_core_characterization.py` (10 queries × 5 results, scored to `math.isclose(rel_tol=1e-9)`). Regenerate fixtures via `YADGAR_REGEN_FIXTURES=1` after intentional behavior changes.
