Metadata-Version: 2.4
Name: larpie
Version: 0.2.0
Summary: Attribution tooling for LLM responses.
Author-email: Mariem Abidi <abidimaryam51@gmail.com>
License: MIT
Project-URL: Homepage, https://github.com/Mariem-Abidi/larpie
Project-URL: Repository, https://github.com/Mariem-Abidi/larpie
Project-URL: Issues, https://github.com/Mariem-Abidi/larpie/issues
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: typer>=0.12
Requires-Dist: litellm>=1.40
Requires-Dist: sentence-transformers>=3.0
Requires-Dist: scikit-learn>=1.3
Requires-Dist: numpy>=1.24
Requires-Dist: torch>=2.0
Requires-Dist: transformers>=4.41
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Dynamic: license-file

# larpie

[![test](https://github.com/Mariem-Abidi/larpie/actions/workflows/test.yml/badge.svg)](https://github.com/Mariem-Abidi/larpie/actions/workflows/test.yml)

## The problem

LLM traces show what happened: the prompt that went in, the response that
came out. They don't show which specific context segments, retrieved
chunks, tool outputs, memory, earlier turns in the conversation, actually
caused that response. When a RAG or agent pipeline gives a wrong or
hallucinated answer, debugging it today usually means reading the whole
trace by hand and guessing.

## What larpie is

larpie is a developer tooling layer built around existing
context-attribution research. It implements two attribution algorithms,
at different points on that spectrum:

- `SimilarityAlgorithm` is a cheap proxy: it regenerates a response for
  each ablated context and compares embeddings, so it works against any
  hosted chat API (OpenAI, Anthropic, Ollama, via litellm) with no
  logprob access needed.
- `LikelihoodAlgorithm` implements the actual mechanism
  [ContextCite: Attributing Model Generation to Context](https://arxiv.org/abs/2409.00729)
  (Cohen-Wang, Shah, Georgiev, Madry; NeurIPS 2024) describes: teacher-forced
  scoring of the *original* response's log-likelihood under each ablated
  context, fit with the same LASSO-plus-bootstrap approach. This needs an
  engine with direct model access - currently a local Hugging Face model
  via `HFEngine` - since hosted chat APIs don't expose that.

Both use the same ablate-and-regress idea as
[AttriBoT: A Bag of Tricks for Efficiently Approximating Leave-One-Out Context Attribution](https://arxiv.org/abs/2411.15102)
(Liu, Kandpal, Raffel; ICLR 2025) too. larpie's own, simpler take on
AttriBoT's hierarchical idea is available as an opt-in mode (see
Hierarchical attribution, below): group segments explicitly, ablate whole
groups first, drill into only the ones that turn out to matter. AttriBoT's
other efficiency tricks (proxy models, cached-activation reuse) aren't
implemented - ablation here is otherwise the straightforward
exhaustive-or-sampled approach, not the faster approximation.

What larpie adds beyond either paper is the developer-tool layer around
them: real integration with LLM providers (via litellm, across OpenAI,
Anthropic, and Ollama, plus a local Hugging Face model for likelihood
attribution), segmentation that understands RAG metadata (which parts of
a conversation are retrieved chunks, tool outputs, or plain turns), a
capability-based engine architecture that fails loudly instead of
silently producing a wrong result when an engine can't support an
algorithm, a CLI workflow for inspecting logged calls, and SQLite-backed
caching so re-inspecting the same call is free.

## Quickstart

```bash
pip install larpie
```

This requires a local Ollama server with a model pulled (`ollama pull
llama3`), or any OpenAI/Anthropic model string if you'd rather pay for API
calls.

Wrap a real `litellm.completion` call so it gets logged, then inspect it:

```python
from litellm import completion
from larpie.middleware import with_logging

logged_completion = with_logging(completion)

response = logged_completion(
    model="ollama/llama3",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "The Eiffel Tower was completed in 1889. When was it finished?"},
    ],
)

# with_logging logs the call and stamps the id of the row it just wrote
# onto the response:
call_id = response.larpie_call_id
print(call_id)
```

```bash
larpie inspect <call_id>
```

`inspect` looks up the logged call, segments its messages, re-runs the
response with pieces of the context ablated, and prints a ranked table of
which part of the context actually drove the answer. `--model` defaults to
the model the call was logged with, so it isn't needed here; pass
`--yes`/`-y` to skip the paid-call confirmation prompt for non-Ollama
models (Ollama runs locally and never asks).

## See it in action

[`examples/rag_attribution_demo.py`](examples/rag_attribution_demo.py)
runs a small RAG conversation (one supporting chunk, two distractors, and
a fabricated fact the model can't already know from pretraining) against
a real local Ollama model and prints the ranked attribution table. Run it
yourself:

```bash
python examples/rag_attribution_demo.py
```

This is real output from an actual run (`ollama/qwen2.5:3b-instruct-q4_K_M`,
temperature 0), not illustrative numbers:

```
Closed-book response (no context, sanity check):
  'I need more specific context to answer this question accurately. The provided context does not contain information about the Aldemoor Lantern Festival or its founding year.'
  Good: no '1997' here - the model can't answer this without the supporting chunk.

Full-context response:
  'The Aldemoor Lantern Festival began in 1997.'

Ranked attribution results (query segment excluded):
  1. [seg_1] (rag_chunk) score=0.1789 ci=(0.1277834502965072, 0.21895693424139875) text="The Aldemoor Lantern Festival began in 1997, founded by local artist Mira Thorne to celebrate the town's bicentennial."
  2. [seg_2] (rag_chunk) score=0.0014 ci=(0.0, 0.025790114892243907) text='Mira Thorne is a sculptor known for her large-scale kinetic light installations across Aldemoor.'
  3. [seg_0] (system) score=-0.0025 ci=(-0.027141633458489772, 0.0) text="You are a helpful assistant. Answer the user's question using only the provided context snippets, as briefly as possible."
  4. [seg_3] (rag_chunk) score=-0.0059 ci=(-0.042601720261782446, -0.0) text='The neighboring Bellhaven Torch Parade started in 1985 and features similar lantern displays.'
```

The closed-book check confirms the model can't answer from pretraining
alone. Once the supporting chunk is added back, the model answers
correctly, and attribution correctly ranks that exact chunk first, with a
confidence interval sitting entirely above zero. Every other segment,
including a distractor that names a different year for a different
event, scores near zero with a confidence interval that includes zero.

## Architecture

larpie sits between your LLM application and the model provider as a thin
logging and analysis layer:

```
User's LLM application
  -> larpie middleware (logs the call to SQLite)
  -> Segmenter (splits context into turns, RAG chunks, tool outputs; dedups near-identical chunks)
  -> Attribution algorithm (declares which engine capability it needs)
  -> Engine capability layer (checks what the model/runtime can actually do)
  -> LLM provider (via litellm: OpenAI, Anthropic, Ollama)
```

**Engine** is how larpie talks to a specific model or runtime. An engine can
implement `generate` (produce text), `embed` (produce a vector), and
`score_continuation` (return the log-likelihood of a fixed continuation
given a context), but not every engine implements all three, and larpie
checks for the ones it actually needs at runtime instead of assuming.

**Algorithm** is how attribution gets computed from whatever capabilities an
engine offers. `SimilarityAlgorithm` needs `generate` and `embed`.
`LikelihoodAlgorithm`, implementing the ContextCite mechanism, needs
`score_continuation` instead: it compares the model's actual
log-likelihood of the original response with and without each piece of
context, rather than regenerating a response and comparing embeddings.

### Capability matrix

Not every engine can run every attribution algorithm.
`algorithms.base.check_capability` enforces this at runtime: a missing
capability raises a clear error instead of silently falling back to
something else.

| Engine | `generate` | `score_continuation` | `embed` | Compatible algorithm(s) |
|---|---|---|---|---|
| `LiteLLMEngine` (OpenAI / Anthropic / Ollama, via litellm) | yes | no | yes (local `sentence-transformers`) | `SimilarityAlgorithm` |
| `HFEngine` (local Hugging Face `transformers` model) | yes | yes | no | `LikelihoodAlgorithm` |
| vLLM / SGLang (planned) | yes | yes | n/a | `LikelihoodAlgorithm` (planned support for larger models than a local `transformers` instance can comfortably run) |

## Likelihood attribution

`LikelihoodAlgorithm` measures the actual log-likelihood of the *original*
response under each ablated context via teacher-forced scoring, not a
regenerate-and-compare-embeddings proxy like `SimilarityAlgorithm`. Run it
with:

```bash
larpie inspect <call_id> --algorithm likelihood --hf-model Qwen/Qwen2.5-0.5B-Instruct
```

`--hf-model` defaults to `Qwen/Qwen2.5-0.5B-Instruct` if omitted. It runs
fully local: no Ollama server, no API key, no network access beyond the
one-time model download from Hugging Face. A small model like that default
runs fine on CPU; anything meaningfully larger needs real GPU memory to be
usable in practice.

Real output from an actual run, on a small RAG conversation (a system
prompt, one supporting chunk, one distractor) using the same fabricated
"Aldemoor Lantern Festival" fact as the similarity example above:

```
This will make 8 engine.score_continuation() call(s): 7 ablation(s) + 1 baseline call.
[QUERY - not ranked] [seg_3] (user_turn) 'In what year did the Aldemoor Lantern Festival begin?'
segment    type               score  ci                         text
seg_1      rag_chunk         5.6799  (3.155, 7.061)             'The Aldemoor Lantern Festival began in 1997, founded by Mira Thorne.'
seg_2      rag_chunk         1.8188  (-0.016, 3.557)            'The neighboring Bellhaven Torch Parade started in 1985.'
seg_0      system           -2.2877  (-3.777, -0.134)           'Answer using only the provided context.'

Note: no noise-floor check for likelihood attribution; see README.
```

The supporting chunk (`seg_1`) scores clearly highest, with a confidence
interval sitting entirely above zero - the same "CI excludes zero means a
reliably detected effect" reading used everywhere else in larpie. Unlike
the similarity table, there's no `NOISE` column here; see Limitations for
why.

## Hierarchical attribution

Both algorithms accept `hierarchical=True` (default off, fully opt-in)
for contexts where several segments are known to come from the same
underlying source - multiple chunks retrieved from one document, for
example. Enable it from the CLI with:

```bash
larpie inspect <call_id> --hierarchical
```

or directly: `SimilarityAlgorithm(hierarchical=True)` /
`LikelihoodAlgorithm(hierarchical=True)`.

Instead of one ablation column per segment, hierarchical mode ablates
whole groups together in a first pass. Only a group whose group-level
confidence interval excludes zero - a reliably detected effect - gets a
second, smaller pass drilling into which of its own members actually
drives that. A group whose CI includes zero is left alone: every member
of that group inherits the group's own score and CI instead of getting
one computed individually, marked with `is_group_estimate=True` on the
result (`[group est.]` in the CLI table).

The tradeoff is explicit: fewer engine calls overall - a group that
doesn't matter costs one ablation instead of one per member - at the cost
of coarser resolution for exactly those groups. You get "this whole group
doesn't matter" rather than "which member of it doesn't matter,
specifically". A group that does matter isn't shortchanged: it still gets
drilled into individually, same resolution as the flat path.

This needs a real grouping signal to do anything: set a shared
`metadata["group_id"]` on messages that come from the same source (see
`larpie.segmenter.Segment.group_id`). Segments without a `group_id` are
each their own singleton group, so with none set anywhere,
`hierarchical=True` is a no-op - it falls back to exactly the flat,
per-segment behavior described above.

## Limitations

- Similarity attribution is a proxy. It compares the embedding of a
  regenerated response to the embedding of the original response; it does
  not measure the model's actual probability of the response (true
  likelihood attribution). Two different pieces of text can embed
  similarly even when the model's confidence in them differs a lot - a
  negated claim or a changed number often reads as barely different to an
  embedding model, even though it completely changes the answer.

  `SimilarityAlgorithm` accepts an optional `metric` callable
  (`Callable[[str, str], float]`, taking `(original_response,
  regenerated_response)` and returning a dissimilarity score in `[0, 1]`)
  so you can swap in a sharper comparison for your case instead of being
  stuck with embedding similarity's specific blind spots. larpie ships one
  concrete alternative, `numeric_aware_dissimilarity`, which catches
  changed numbers that embedding similarity misses:

  ```python
  from larpie.algorithms.similarity import SimilarityAlgorithm, numeric_aware_dissimilarity

  algo = SimilarityAlgorithm(metric=numeric_aware_dissimilarity)
  ```

  This doesn't solve the general problem - no single metric catches every
  kind of meaning change, and a swapped-in metric has its own blind spots
  in exchange for closing this one. It just makes the comparison step
  swappable, rather than fixed to the default's specific weaknesses.
- Likelihood attribution needs an engine that can compute
  `score_continuation` directly. That exists today for a local Hugging
  Face `transformers` model (`HFEngine` + `LikelihoodAlgorithm`, see
  above); vLLM and SGLang support doesn't exist yet and is still planned,
  mainly for models too large to run comfortably as a local `transformers`
  instance.
- There is no noise-floor check for likelihood attribution, unlike
  similarity attribution. That check works by calling `generate()`
  against the same unablated context several times and measuring how much
  the *sampled* output varies; `score_continuation` is one deterministic
  forward pass over fixed input with no sampling step, so repeating it
  would just return the same number every time - there is nothing to
  measure. The residual risk: on a backend whose forward pass isn't
  actually bit-reproducible (see the point below about local quantized
  models), that non-determinism could still produce spurious variation in
  scores that nothing here catches.
- If the model already knows a fact from its own pretraining, a retrieved
  chunk supporting that fact will correctly score near zero: removing it
  doesn't change the answer, because the model didn't need it to answer
  correctly. This is expected behavior, not a bug, but it means a low or
  zero score for a chunk doesn't mean the chunk was irrelevant to the
  topic, only that it wasn't necessary for this particular answer.
- larpie is built for complex contexts: 10+ segments, RAG chunks mixed
  with agent tool call histories and long-running memory, where reading
  the raw trace by hand doesn't scale. It is not meant as a general
  debugger for small or simple prompts; if your context is two or three
  chunks, just read them.
- Local quantized models are not guaranteed to be bit-reproducible even at
  temperature 0, since floating-point reduction order in batched or
  threaded inference isn't always deterministic on those backends.
  Caching gives you exact reproducibility for repeated inspection of the
  same logged call, since cached ablation results get reused instead of
  regenerated, but it does not make a fresh run against a live model
  deterministic on its own.
- The dedup threshold (0.9 cosine similarity) can merge near-duplicate
  templated text, e.g. numbered filler turns differing only by a label
  ("follow-up question number 1" vs. "number 2"). This isn't just
  theoretical: it happened during scale testing and had to be fixed by
  making the filler text genuinely distinct, not just relabeled.
- Likelihood attribution's real forward passes (`HFEngine.score_continuation`)
  must fit the full ablated context plus the original response within the
  local model's context window. This is handled per ablation: a vector
  that would exceed the limit raises a clear `ContextTooLongError`
  (checked against the model's actual `config.max_position_embeddings` -
  confirmed more reliable than `tokenizer.model_max_length`, which reports
  an unrelated, much larger number for the default Qwen model) instead of
  truncating silently or crashing inside the forward pass, and
  `LikelihoodAlgorithm` skips just that one ablation vector, printing how
  many were skipped and why. One case is still unhandled: since ablation
  only ever removes segments, the full, unablated baseline call is always
  the largest context in a run - if that alone already exceeds the limit,
  the whole run still aborts, since there's no smaller context left to
  fall back to.

## License

MIT. See [LICENSE](LICENSE).

## Development

```bash
pip install -e ".[dev]"
pytest
```
