Metadata-Version: 2.4
Name: White-Walker
Version: 0.6.0
Summary: Multimodal Agentic RAG with hierarchical tree indexing — index PDFs & Markdown into navigable tree structures, persist in Supabase, and query with LLM-powered retrieval.
Author: NUHASHROXME
License: MIT
Project-URL: Homepage, https://github.com/YashNuhash/White-Walker
Project-URL: Repository, https://github.com/YashNuhash/White-Walker
Project-URL: Issues, https://github.com/YashNuhash/White-Walker/issues
Keywords: RAG,multimodal,agentic,LLM,retrieval,tree-indexing,supabase,nvidia,pdf,arxiv
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: supabase>=2.0
Requires-Dist: PyPDF2>=3.0
Requires-Dist: pdfplumber>=0.9
Requires-Dist: python-dotenv
Requires-Dist: httpx
Requires-Dist: requests
Requires-Dist: pymupdf
Requires-Dist: pyyaml
Dynamic: license-file

# 🧊 White Walker

[![PyPI version](https://badge.fury.io/py/white-walker.svg)](https://pypi.org/project/white-walker/)
[![Python](https://img.shields.io/pypi/pyversions/white-walker.svg)](https://pypi.org/project/white-walker/)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)

**Multimodal Agentic RAG** — Index PDFs & Markdown into hierarchical tree structures, persist in Supabase, and query with LLM-powered retrieval.

White Walker transforms documents into navigable tree indexes using LLM-based structural analysis, stores them in Supabase (or locally), and answers questions by intelligently traversing the tree to find relevant sections.

---

## ✨ Features

- 🌲 **Hierarchical Tree Indexing** — Automatically detects document structure (TOC, sections, subsections) and builds a navigable tree
- 🗄️ **Supabase Persistence** — Trees, raw pages, and metadata stored in PostgreSQL via Supabase
- 🤖 **Agentic Retrieval** — LLM navigates the tree to find relevant sections, then synthesizes answers with citations
- 📄 **PDF & Markdown Support** — Works with academic papers, technical docs, and any structured document
- ⚡ **Rate-Limited API** — Built-in token-bucket rate limiter for NVIDIA NIM API (38 RPM, configurable)
- 🔌 **Pluggable LLM Backend** — Default: NVIDIA NIM API, easily extensible

---

## 📦 Installation

```bash
pip install white-walker
```

---

## 🚀 Quick Start

### 1. Set Environment Variables

```bash
# .env file
SUPABASE_URL=https://your-project.supabase.co
SUPABASE_KEY=your-anon-key
NVIDIA_API_KEY=your-nvidia-nim-api-key
```

### 2. Set Up Supabase Tables

Run the SQL in `setup_supabase.sql` in your Supabase SQL editor to create the required tables (`rag_documents`, `rag_raw_pages`, `rag_tree_nodes`).

### 3. Index & Query

```python
from white_walker import WhiteWalkerClient

client = WhiteWalkerClient()

# Index a PDF
doc_id = client.index("paper.pdf")

# Query the indexed document
result = client.query(doc_id, "What is the main contribution of this paper?")

print(result["answer"])
print(result["citations"])
print(result["confidence"])  # "high", "medium", or "low"
```

---

## 🏗️ Architecture

```
PDF / Markdown
       │
       ▼
┌─────────────────┐
│   PDF Ingestor   │  Extracts text, detects TOC, builds hierarchical tree
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  Tree Indexer    │  LLM-based section detection, title verification,
│                  │  page-number alignment, node summarization
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  Supabase Store  │  Persists tree nodes, raw pages, document metadata
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│ Agent Retriever  │  LLM navigates tree → finds relevant nodes →
│                  │  fetches page content → synthesizes cited answer
└─────────────────┘
```

---

## ⚙️ Configuration

White Walker uses `white_walker/config.yaml` for defaults:

| Parameter | Default | Description |
|---|---|---|
| `model` | `moonshotai/kimi-k2.6` | NVIDIA NIM model for indexing & retrieval |
| `toc_check_page_num` | `20` | Max pages to scan for table of contents |
| `max_page_num_each_node` | `10` | Max pages per tree node |
| `max_token_num_each_node` | `20000` | Max tokens per tree node |

Override at initialization:

```python
client = WhiteWalkerClient(model="meta/llama-3.1-70b-instruct")
```

---

## 🔑 Environment Variables

| Variable | Required | Description |
|---|---|---|
| `NVIDIA_API_KEY` | ✅ | NVIDIA NIM API key for LLM calls |
| `SUPABASE_URL` | Optional | Supabase project URL (falls back to local storage) |
| `SUPABASE_KEY` | Optional | Supabase anon/service key |

---

## 📊 Evaluation

White Walker includes a RAGAS-based evaluation pipeline for benchmarking against multimodal agentic RAG datasets. See `evaluate_pipeline.py` for details.

---

## 📄 License

MIT License — see [LICENSE](LICENSE) for details.

---

## 🙏 Acknowledgments

Built on the foundation of [VectifyAI/PageIndex](https://github.com/VectifyAI/PageIndex), re-architected for local processing, Supabase persistence, and pluggable LLM backends.
