Metadata-Version: 2.4
Name: rag-debug-cli
Version: 0.1.0
Summary: A debugger for RAG applications: diagnoses retrieval, context, and grounding failures.
Project-URL: Homepage, https://github.com/Perevoznyi-Creation/rag_debugger
Project-URL: Repository, https://github.com/Perevoznyi-Creation/rag_debugger
Project-URL: Issues, https://github.com/Perevoznyi-Creation/rag_debugger/issues
Author: Serhii Perevoznyi
License: MIT
Keywords: debugging,evaluation,hallucination,llm,rag,retrieval-augmented-generation
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Debuggers
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.10
Requires-Dist: groq>=0.11.0
Requires-Dist: numpy>=1.26.0
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: sentence-transformers>=3.0.0
Provides-Extra: all-judges
Requires-Dist: anthropic>=0.40.0; extra == 'all-judges'
Requires-Dist: openai>=1.50.0; extra == 'all-judges'
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.40.0; extra == 'anthropic'
Provides-Extra: dev
Requires-Dist: anthropic>=0.40.0; extra == 'dev'
Requires-Dist: langchain-core>=0.3.0; extra == 'dev'
Requires-Dist: llama-index-core>=0.11.0; extra == 'dev'
Requires-Dist: openai>=1.50.0; extra == 'dev'
Requires-Dist: pytest>=8.0.0; extra == 'dev'
Requires-Dist: ruff>=0.7.0; extra == 'dev'
Provides-Extra: langchain
Requires-Dist: langchain-core>=0.3.0; extra == 'langchain'
Provides-Extra: llama-index
Requires-Dist: llama-index-core>=0.11.0; extra == 'llama-index'
Provides-Extra: openai
Requires-Dist: openai>=1.50.0; extra == 'openai'
Description-Content-Type: text/markdown

# RAG Debugger

Your RAG chatbot is wrong.

But why?

RAG Debugger finds:

- bad retrieval
- irrelevant context
- hallucinated answers
- missing information

## Install

```
pip install rag-debug-cli
```

Or from source, for development:

```
git clone https://github.com/Perevoznyi-Creation/rag_debugger.git
cd rag_debugger
pip install -e ".[dev]"
```

## Usage

```python
from rag_debugger import debug_rag

report = debug_rag(
    question="What is the refund policy?",
    retrieved_documents=[
        "Customers can return products within 30 days.",
        "Warranty lasts for 2 years.",
    ],
    answer="Customers can return products within 30 days.",
)

print(report)
```

```
RAG Score: 85/100

Retrieval:
  ✓ (0.78) Customers can return products within 30 days.
  ✗ (0.31) Warranty lasts for 2 years.

Grounding:
  ✓ Answer supported by retrieved context

Problems:
  ⚠ Irrelevant document retrieved: Warranty lasts for 2 years.

Recommendation:
  Reduce retrieval k from 2 to 1
```

## CLI

```
rag-debug analyze examples/basic.json
rag-debug analyze examples/basic.json --html report.html
rag-debug evaluate examples/customer_questions.json
rag-debug evaluate examples/customer_questions.json --save before.json
rag-debug evaluate examples/customer_questions_v2.json --save after.json
rag-debug compare before.json after.json
```

Input JSON shape (single case):

```json
{
  "question": "How long are refunds?",
  "documents": ["Refunds take 5 days"],
  "answer": "Refunds take 30 days"
}
```

Dataset JSON shape (for `evaluate`, a list of cases, `label` optional):

```json
[
  {
    "label": "refund timing",
    "question": "How long are refunds?",
    "documents": ["Refunds take 5 days"],
    "answer": "Refunds take 30 days"
  }
]
```

`compare` diffs two `evaluate --save` result files by matching case labels — useful for checking whether a RAG change (chunk size, top_k, prompt) actually helped:

```
Before: 38%
After:  53%
Change: +15%

Improvements:
  warranty question (bad retrieval): 13 -> 72 (+59)
```

## Choosing an LLM judge

Grounding/hallucination checks are done by an LLM judge, pluggable per provider. Set one API key and, optionally, pick the provider:

```
export GROQ_API_KEY="..."        # free tier — default if RAG_DEBUGGER_JUDGE is unset
export OPENAI_API_KEY="..."
export ANTHROPIC_API_KEY="..."

export RAG_DEBUGGER_JUDGE="groq"                # uses that provider's default model
export RAG_DEBUGGER_JUDGE="groq:<model-name>"   # or pick a specific model
export RAG_DEBUGGER_JUDGE="openai:<model-name>"
export RAG_DEBUGGER_JUDGE="anthropic:<model-name>"
```

Each provider's default model is defined in `src/rag_debugger/judges/<provider>_judge.py`
(`DEFAULT_MODEL`) — check there or your provider's docs for current model names, since
they change over time and this README intentionally doesn't pin one.

Or override per call/command:

```python
debug_rag(question=..., retrieved_documents=..., answer=..., judge="openai:<model-name>")
```
```
rag-debug analyze examples/basic.json --judge anthropic:<model-name>
```

OpenAI and Anthropic support are optional extras (`pip install -e ".[openai]"`, `".[anthropic]"`, or `".[all-judges]"` for both) — only the SDK for the provider you actually use needs to be installed.

For local development, put your key(s) in a `.env.local` file (gitignored) — both `examples/basic.py` and the CLI load it automatically.

## LangChain integration

`DebugRetriever` wraps any existing `BaseRetriever` — it's a real `Runnable`, so it drops directly into an LCEL chain with no other changes:

```python
from rag_debugger.integrations.langchain import DebugRetriever

retriever = DebugRetriever(retriever=my_vectorstore.as_retriever())

chain = retriever | prompt | llm
answer = chain.invoke("What is the refund policy?")

report = retriever.debug(answer)
print(report)
```

`DebugRetriever` remembers the query and documents from the most recent `invoke()`; `.debug(answer)` runs `debug_rag()` against them. Pass `judge=` the same way as `debug_rag()` (a spec string or `Judge` instance) if you don't want the `RAG_DEBUGGER_JUDGE` default. See `examples/langchain_integration.py` for a runnable end-to-end example.

Requires the `langchain` extra: `pip install -e ".[langchain]"`.

## LlamaIndex integration

Same pattern, for LlamaIndex's `BaseRetriever`:

```python
from rag_debugger.integrations.llama_index import DebugRetriever

retriever = DebugRetriever(retriever=index.as_retriever())

nodes = retriever.retrieve("What is the refund policy?")
# ... build a response from `nodes` however your pipeline does ...
answer = "Customers can return products within 30 days."

report = retriever.debug(answer)
print(report)
```

`DebugRetriever` remembers the query and nodes from the most recent `retrieve()`; `.debug(answer)` runs `debug_rag()` against them (using each node's `get_content()`). It's a real `BaseRetriever`, so it's a drop-in replacement anywhere a LlamaIndex retriever is expected, including `RetrieverQueryEngine`. See `examples/llama_index_integration.py` for a runnable end-to-end example.

Requires the `llama-index` extra: `pip install -e ".[llama-index]"`.

## Logging

RAG Debugger uses the standard `logging` module (`logger = logging.getLogger(__name__)` per
module) and configures no handler of its own — as a library, it stays silent unless the
consuming application sets one up.

The CLI does configure a handler for you. Pass `-v`/`--verbose` (before the subcommand)
to see debug-level diagnostics — which judge/model was resolved, retrieval scoring
summaries, and per-case warnings during `evaluate` — on stderr:

```
rag-debug -v analyze examples/basic.json
```

Without `-v`, only warnings and errors are shown (e.g. a judge API call failing, or a
dataset case being skipped). Third-party libraries' own debug logs (httpx,
sentence-transformers, etc.) are not included even at `-v`, so this doesn't drown you
in noise.

## Error handling

Every judge provider wraps its SDK's API errors (auth failures, rate limits, network
issues) into a `RuntimeError` with a clear message, instead of letting an SDK-specific
exception escape uncaught — the CLI's `analyze`/`evaluate` commands catch this and print
a clean `Error: ...` line rather than a raw traceback. `evaluate_dataset()` /
`rag-debug evaluate` treats a per-case runtime failure (e.g. one transient API error in
a 50-question dataset) as that one case being skipped, not the whole run failing —
`DatasetEvaluation.failed` lists which cases errored and why, and both the printed
summary and any `--save`d results reflect only the cases that actually completed.

A dataset file with a case missing a required field (`question`/`documents`/`answer`)
is a different kind of problem — a dataset-authoring mistake, not a runtime hiccup — and
still fails the whole run immediately with a clear error, on the reasoning that silently
skipping a malformed case could hide a bug in how the dataset was generated.

## Known limitations

- Retrieval relevance uses raw cosine similarity from `all-MiniLM-L6-v2`. Real-world scores for genuinely relevant pairs often land around 0.3-0.5, not near 1.0 — the default threshold is tuned for this, but may need adjusting for your domain.
- The default Groq judge uses a free-tier model. Even at `temperature=0`, Groq's batched serving does not guarantee identical output across calls, so on borderline paraphrase cases (e.g. "return" vs "refund") the verdict can occasionally flip between runs. Switching to a larger/stronger model on any provider (`--judge openai:...` or `--judge anthropic:...`) generally improves consistency.

## Development

```bash
pip install -e ".[dev]"

python3 -m pytest tests/ -v      # run the test suite (no API keys/network needed)
ruff check .                     # lint
ruff format .                    # format
```

See [test_strategy.md](test_strategy.md) for what the automated suite covers (and
deliberately doesn't), and [SELF_TEST.md](SELF_TEST.md) for the manual checklist to run
before publishing a new version.
