Metadata-Version: 2.4
Name: rag-failcase-kit
Version: 0.1.0
Summary: Lightweight JSONL tracing for debugging failed RAG answers.
Project-URL: Homepage, https://github.com/slayer011019/rag-failcase-kit
Project-URL: Repository, https://github.com/slayer011019/rag-failcase-kit
Project-URL: Issues, https://github.com/slayer011019/rag-failcase-kit/issues
Author: rag-failcase-kit contributors
License: MIT
License-File: LICENSE
Keywords: debugging,jsonl,logging,rag,retrieval
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Debuggers
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.9
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: twine>=5.0; extra == 'dev'
Description-Content-Type: text/markdown

# rag-failcase-kit

`rag-failcase-kit` is a lightweight Python package for recording one RAG pipeline run as one JSONL trace.

When a RAG application returns an empty, inaccurate, or weakly relevant answer, it is often hard to tell where the failure happened. The issue may be retrieval quality, prompt construction, fallback logic, answer generation, or a runtime error. In small projects this often becomes a pile of `print` statements, which is difficult to search, compare, or replay later.

This package keeps the first version intentionally small: it records query, retrieved documents, scores, fallback usage, answer preview, latency, status, and errors into a JSONL file.

## Installation

From PyPI:

```bash
pip install rag-failcase-kit
```

From a local checkout:

```bash
pip install -e .
```

For development and release checks:

```bash
pip install -e ".[dev]"
```

## 30-Second Usage

```python
from rag_failcase_kit import RagLogger

logger = RagLogger(project="demo-rag", log_path="logs/rag_traces.jsonl")

with logger.trace("What is RAG?") as trace:
    trace.log_retrieval(
        [
            {
                "content": "Retrieval-augmented generation combines retrieval with text generation.",
                "source": "docs/rag.md",
                "score": 0.91,
            }
        ],
        top_k=3,
        min_score=0.7,
    )
    trace.log_answer("RAG retrieves relevant context and uses it to generate an answer.")
```

The log directory is created automatically. Each trace is appended as one JSON object per line.

## CLI Usage

The package installs a `rag-failcase` command:

```bash
rag-failcase summary logs/rag_traces.jsonl
rag-failcase failures logs/rag_traces.jsonl
rag-failcase inspect logs/rag_traces.jsonl --trace-id <trace_id>
```

All CLI commands print JSON output.

Example summary:

```json
{
  "total_traces": 12,
  "successes": 9,
  "failures": 3,
  "fallback_used": 4,
  "avg_latency_ms": 128.42
}
```

## Log Schema

Example JSONL record:

```json
{
  "trace_id": "9fcd7b91-6b12-46fb-8ff1-5f6f1ef49b15",
  "project": "demo-rag",
  "query": "What is RAG?",
  "retrieval": {
    "documents": [
      {
        "content_preview": "Retrieval-augmented generation combines retrieval with text generation.",
        "metadata": {
          "chunk_id": "rag-001"
        },
        "source": "docs/rag.md",
        "score": 0.91
      }
    ],
    "top_k": 3,
    "min_score": 0.7
  },
  "retrieved_count": 1,
  "fallback": {
    "used": false,
    "strategy": null,
    "reason": null
  },
  "answer": "RAG retrieves relevant context and uses it to generate an answer.",
  "answer_preview": "RAG retrieves relevant context and uses it to generate an answer.",
  "latency_ms": 3.241,
  "status": "success",
  "error": null,
  "created_at": "2026-07-09T00:00:00+00:00"
}
```

Document content is stored as a bounded preview to avoid unexpectedly large logs.

## Real RAG Integration

Wrap the part of your RAG pipeline that handles one user query:

```python
from rag_failcase_kit import RagLogger

logger = RagLogger(project="support-bot", log_path="logs/support_rag.jsonl")


def answer_question(query, retriever, llm):
    with logger.trace(query) as trace:
        documents = retriever.search(query)
        trace.log_retrieval(documents, top_k=5, min_score=0.75)

        if not documents:
            trace.log_fallback("no_documents", reason="Retriever returned no documents.")
            answer = "I do not have enough context to answer."
            trace.log_answer(answer)
            return answer

        answer = llm.generate(query=query, documents=documents)
        trace.log_answer(answer)
        return answer
```

`documents` can be dictionaries or document-like objects. For objects, the logger safely checks common fields such as `page_content`, `metadata`, `source`, and `score`.

## Optional Integrations

The package does not depend on FastAPI, LangChain, or LangGraph.

FastAPI example:

```bash
pip install fastapi uvicorn
uvicorn examples.fastapi_example:app --reload
```

LangGraph-style example:

```bash
python examples/langgraph_example.py
```

See:

- [examples/basic_usage.py](examples/basic_usage.py)
- [examples/fastapi_example.py](examples/fastapi_example.py)
- [examples/langgraph_example.py](examples/langgraph_example.py)

## Development

```bash
pip install -e ".[dev]"
pytest
python -m build
twine check dist/*
```

To test the console script in a clean environment:

```bash
python -m venv .venv-test
. .venv-test/bin/activate
pip install dist/rag_failcase_kit-0.1.0-py3-none-any.whl
rag-failcase --help
```

## Roadmap

- Add prompt logging with explicit redaction controls.
- Add richer failure categorization helpers.
- Add Markdown or HTML reports.
- Add log rotation and sampling controls.
- Add optional adapters for popular RAG frameworks without making them required dependencies.
