Metadata-Version: 2.4
Name: research-claim-validator
Version: 0.1.12
Summary: Python tools for validating research claims against scientific literature.
Author: Tolulope Ale
Keywords: research,claim-validation,scientific-validation,literature,evidence
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Provides-Extra: llm
Requires-Dist: anthropic>=0.18.0; extra == "llm"
Requires-Dist: openai>=1.12.0; extra == "llm"
Requires-Dist: pydantic>=2.4.0; extra == "llm"
Requires-Dist: pydantic-settings>=2.0.0; extra == "llm"
Requires-Dist: prometheus-client>=0.19.0; extra == "llm"
Requires-Dist: python-dotenv>=1.0.0; extra == "llm"
Requires-Dist: tenacity>=8.2.3; extra == "llm"
Provides-Extra: retrieval
Requires-Dist: arxiv>=2.1.0; extra == "retrieval"
Requires-Dist: Deprecated>=1.2.13; extra == "retrieval"
Requires-Dist: einops>=0.7.0; extra == "retrieval"
Requires-Dist: faiss-cpu>=1.7.4; extra == "retrieval"
Requires-Dist: httpx>=0.24.0; extra == "retrieval"
Requires-Dist: langgraph>=0.0.20; extra == "retrieval"
Requires-Dist: neo4j>=5.14.0; extra == "retrieval"
Requires-Dist: numpy>=1.24.0; extra == "retrieval"
Requires-Dist: pandas>=2.0.0; extra == "retrieval"
Requires-Dist: pypdf>=6.6.2; extra == "retrieval"
Requires-Dist: requests>=2.31.0; extra == "retrieval"
Requires-Dist: sentence-transformers>=2.2.0; extra == "retrieval"
Requires-Dist: spacy>=3.7.0; extra == "retrieval"
Requires-Dist: structlog>=24.1.0; extra == "retrieval"
Requires-Dist: torch>=2.0.0; extra == "retrieval"
Requires-Dist: tqdm>=4.65.0; extra == "retrieval"
Provides-Extra: corpus
Requires-Dist: aiofiles>=23.0.0; extra == "corpus"
Requires-Dist: anthropic>=0.18.0; extra == "corpus"
Requires-Dist: arxiv>=2.1.0; extra == "corpus"
Requires-Dist: Deprecated>=1.2.13; extra == "corpus"
Requires-Dist: einops>=0.7.0; extra == "corpus"
Requires-Dist: faiss-cpu>=1.7.4; extra == "corpus"
Requires-Dist: httpx>=0.24.0; extra == "corpus"
Requires-Dist: langgraph>=0.0.20; extra == "corpus"
Requires-Dist: neo4j>=5.14.0; extra == "corpus"
Requires-Dist: numpy>=1.24.0; extra == "corpus"
Requires-Dist: openai>=1.12.0; extra == "corpus"
Requires-Dist: pandas>=2.0.0; extra == "corpus"
Requires-Dist: prometheus-client>=0.19.0; extra == "corpus"
Requires-Dist: pydantic>=2.4.0; extra == "corpus"
Requires-Dist: pydantic-settings>=2.0.0; extra == "corpus"
Requires-Dist: pypdf>=6.6.2; extra == "corpus"
Requires-Dist: python-dotenv>=1.0.0; extra == "corpus"
Requires-Dist: requests>=2.31.0; extra == "corpus"
Requires-Dist: sentence-transformers>=2.2.0; extra == "corpus"
Requires-Dist: spacy>=3.7.0; extra == "corpus"
Requires-Dist: structlog>=24.1.0; extra == "corpus"
Requires-Dist: tenacity>=8.2.3; extra == "corpus"
Requires-Dist: torch>=2.0.0; extra == "corpus"
Requires-Dist: tqdm>=4.65.0; extra == "corpus"
Provides-Extra: api
Requires-Dist: aiofiles>=23.0.0; extra == "api"
Requires-Dist: fastapi>=0.104.0; extra == "api"
Requires-Dist: opentelemetry-api>=1.22.0; extra == "api"
Requires-Dist: opentelemetry-exporter-otlp>=1.22.0; extra == "api"
Requires-Dist: opentelemetry-instrumentation-fastapi>=0.43b0; extra == "api"
Requires-Dist: opentelemetry-sdk>=1.22.0; extra == "api"
Requires-Dist: prometheus-client>=0.19.0; extra == "api"
Requires-Dist: python-multipart>=0.0.6; extra == "api"
Requires-Dist: sse-starlette>=1.8.2; extra == "api"
Requires-Dist: uvicorn[standard]>=0.24.0; extra == "api"
Provides-Extra: ui
Requires-Dist: pandas>=2.0.0; extra == "ui"
Requires-Dist: sseclient-py>=1.8.0; extra == "ui"
Requires-Dist: streamlit>=1.30.0; extra == "ui"
Provides-Extra: full
Requires-Dist: aiofiles>=23.0.0; extra == "full"
Requires-Dist: anthropic>=0.18.0; extra == "full"
Requires-Dist: arxiv>=2.1.0; extra == "full"
Requires-Dist: Deprecated>=1.2.13; extra == "full"
Requires-Dist: einops>=0.7.0; extra == "full"
Requires-Dist: faiss-cpu>=1.7.4; extra == "full"
Requires-Dist: fastapi>=0.104.0; extra == "full"
Requires-Dist: httpx>=0.24.0; extra == "full"
Requires-Dist: langgraph>=0.0.20; extra == "full"
Requires-Dist: neo4j>=5.14.0; extra == "full"
Requires-Dist: numpy>=1.24.0; extra == "full"
Requires-Dist: openai>=1.12.0; extra == "full"
Requires-Dist: opentelemetry-api>=1.22.0; extra == "full"
Requires-Dist: opentelemetry-exporter-otlp>=1.22.0; extra == "full"
Requires-Dist: opentelemetry-instrumentation-fastapi>=0.43b0; extra == "full"
Requires-Dist: opentelemetry-sdk>=1.22.0; extra == "full"
Requires-Dist: pandas>=2.0.0; extra == "full"
Requires-Dist: prometheus-client>=0.19.0; extra == "full"
Requires-Dist: pydantic>=2.4.0; extra == "full"
Requires-Dist: pydantic-settings>=2.0.0; extra == "full"
Requires-Dist: pypdf>=6.6.2; extra == "full"
Requires-Dist: python-dotenv>=1.0.0; extra == "full"
Requires-Dist: python-multipart>=0.0.6; extra == "full"
Requires-Dist: requests>=2.31.0; extra == "full"
Requires-Dist: sentence-transformers>=2.2.0; extra == "full"
Requires-Dist: spacy>=3.7.0; extra == "full"
Requires-Dist: sse-starlette>=1.8.2; extra == "full"
Requires-Dist: sseclient-py>=1.8.0; extra == "full"
Requires-Dist: streamlit>=1.30.0; extra == "full"
Requires-Dist: structlog>=24.1.0; extra == "full"
Requires-Dist: tenacity>=8.2.3; extra == "full"
Requires-Dist: torch>=2.0.0; extra == "full"
Requires-Dist: tqdm>=4.65.0; extra == "full"
Requires-Dist: uvicorn[standard]>=0.24.0; extra == "full"
Provides-Extra: dev
Requires-Dist: black>=23.12.1; extra == "dev"
Requires-Dist: build>=1.0.0; extra == "dev"
Requires-Dist: flake8>=7.0.0; extra == "dev"
Requires-Dist: ipython>=8.20.0; extra == "dev"
Requires-Dist: mypy>=1.8.0; extra == "dev"
Requires-Dist: pytest>=7.4.3; extra == "dev"
Requires-Dist: pytest-asyncio>=0.23.3; extra == "dev"
Requires-Dist: pytest-cov>=4.1.0; extra == "dev"
Requires-Dist: pytest-mock>=3.12.0; extra == "dev"
Requires-Dist: responses>=0.24.1; extra == "dev"
Requires-Dist: twine>=5.0.0; extra == "dev"
Requires-Dist: wheel>=0.42.0; extra == "dev"
Provides-Extra: notebooks
Requires-Dist: datasets>=2.16.0; extra == "notebooks"
Requires-Dist: ipykernel>=6.25.0; extra == "notebooks"
Requires-Dist: ipywidgets>=8.1.0; extra == "notebooks"

---
title: Research Finding Validation System
description: A multi-agent Graph RAG system for validating scientific claims against research literature from ArXiv and OpenAlex.
author: Tolulope Ale
ms.date: 2026-04-24
ms.topic: overview
keywords:
  - graph-rag
  - multi-agent
  - scientific-validation
  - neo4j
  - faiss
  - langgraph
---

## Table of Contents

- [Overview](#overview)
- [Getting Started](#getting-started)
- [Architecture](#architecture)
- [Features](#features)
- [System Components](#system-components)
- [Workflows](#workflows)
- [Technology Stack](#technology-stack)
- [Project Structure](#project-structure)
- [Development Notebooks](#development-notebooks)
- [Testing and Evaluation](#testing-and-evaluation)

## Overview

This system automates the validation of scientific findings by orchestrating multiple specialized agents that search, process, and analyze research papers. The system employs a multi-workflow architecture that separates concerns into paper fetching, embedding/indexing, and retrieval/validation stages.

### Key Capabilities

- **Multi-source Paper Search**: Fetch papers from ArXiv and OpenAlex with semantic ranking
- **Intelligent Chunking**: Hierarchical text segmentation with section-aware processing
- **Knowledge Graph Construction**: Neo4j-based graph linking papers, entities, and relationships
- **Vector Similarity Search**: FAISS-powered semantic search across paper chunks
- **Semantic Similarity Linking**: Automatic discovery of related papers and content
- **LLM-Powered Validation**: Claude-based claim validation with structured reasoning
- **Modular Workflow Execution**: Independent execution of paper fetching, embedding, and validation
- **Real-time Streaming**: Server-Sent Events for live progress updates
- **Comprehensive Observability**: Distributed tracing, metrics, and structured logging

## Getting Started

### Python Package Usage

Install the lightweight package:

```powershell
pip install research-claim-validator
```

The base install supports the public API and no-evidence responses without
installing the heavier LLM, graph, or vector-search dependencies:

```python
from research_claim_validator import ClaimValidator

validator = ClaimValidator()
result = validator.validate("This research claim is supported by recent literature.")
print(result.to_dict())
```

For LLM-backed validation against supplied evidence, install the optional LLM
dependencies:

```powershell
pip install "research-claim-validator[llm]"
```

Set your LLM provider and API key before running validation:

```powershell
$env:LLM_PROVIDER = "anthropic"
$env:LLM_MODEL = "claude-sonnet-4-6"
$env:ANTHROPIC_API_KEY = "your-api-key"
```

On macOS or Linux:

```bash
export LLM_PROVIDER="anthropic"
export LLM_MODEL="claude-sonnet-4-6"
export ANTHROPIC_API_KEY="your-api-key"
```

Then pass evidence dictionaries to `validate(...)`. The validator bases its
answer only on the evidence you provide:

```python
from research_claim_validator import ClaimValidator

validator = ClaimValidator()
result = validator.validate(
    "Arctic sea ice extent has declined substantially in recent decades.",
    evidence=[
        {
            "title": "Example Arctic evidence",
            "text": (
                "Observations indicate that Arctic sea ice cover has rapidly "
                "retreated in recent decades as global temperatures have increased."
            ),
        }
    ],
)
print(result.to_dict())
```

In Jupyter notebooks, use the async API because notebooks already run an event
loop:

```python
result = await validator.validate_async(
    "Arctic sea ice extent has declined substantially in recent decades.",
    evidence=[
        {
            "title": "Example Arctic evidence",
            "text": (
                "Observations indicate that Arctic sea ice cover has rapidly "
                "retreated in recent decades as global temperatures have increased."
            ),
        }
    ],
)
result.to_dict()
```

For automatic literature ingestion and corpus-backed validation, install the
corpus dependencies:

```powershell
pip install "research-claim-validator[corpus]"
```

Corpus mode uses a local Neo4j knowledge graph plus a local FAISS vector index.
Neo4j must be running and configured before `auto_ingest=True`:

```powershell
$env:LLM_PROVIDER = "anthropic"
$env:LLM_MODEL = "claude-sonnet-4-6"
$env:ANTHROPIC_API_KEY = "your-api-key"
$env:NEO4J_URI = "neo4j://localhost:7687"
$env:NEO4J_USER = "neo4j"
$env:NEO4J_PASSWORD = "your-neo4j-password"
```

On macOS or Linux:

```bash
export LLM_PROVIDER="anthropic"
export LLM_MODEL="claude-sonnet-4-6"
export ANTHROPIC_API_KEY="your-api-key"
export NEO4J_URI="neo4j://localhost:7687"
export NEO4J_USER="neo4j"
export NEO4J_PASSWORD="your-neo4j-password"
```

Then create a persistent local corpus directory and validate against retrieved
paper evidence:

```python
from research_claim_validator import ClaimValidator

validator = ClaimValidator(corpus_dir="./claim-validator-corpus")
result = validator.validate(
    "Arctic sea ice extent has declined substantially in recent decades due to warming temperatures.",
    query="Arctic sea ice extent decline recent decades warming temperatures polar climate change",
    auto_ingest=True,
    max_papers=3,
)
print(result.to_dict())
```

The first corpus build can take several minutes because papers may be fetched,
PDFs processed, embeddings generated, a FAISS index written, and Neo4j metadata
populated. Later runs can reuse the same corpus directory and are usually much
faster.

See [SETUP.md](SETUP.md) for installation, environment configuration, and local development instructions.

For Docker-based deployment, see [DEPLOYMENT.md](DEPLOYMENT.md).

For Neo4j installation and configuration, see [NEO4J_SETUP.md](NEO4J_SETUP.md).

## Architecture

### Architectural Design Framework

The system follows a **multi-layer agent-based architecture** with clear separation of concerns:

```
┌─────────────────────────────────────────────────────────────┐
│                     Presentation Layer                      │
│  ┌─────────────────────┐         ┌────────────────────┐   │
│  │  Streamlit UI       │         │  FastAPI Endpoints │   │
│  │  - Real-time SSE    │         │  - RESTful API     │   │
│  │  - Workflow Control │         │  - OpenAPI Docs    │   │
│  └─────────────────────┘         └────────────────────┘   │
└─────────────────────────────────────────────────────────────┘
                              │
┌─────────────────────────────────────────────────────────────┐
│                    Orchestration Layer                      │
│  ┌──────────────────┐  ┌──────────────┐  ┌──────────────┐ │
│  │ Paper Fetching   │  │  Embedding   │  │  Retrieval   │ │
│  │   Workflow       │  │   Workflow   │  │   Workflow   │ │
│  └──────────────────┘  └──────────────┘  └──────────────┘ │
│                    (LangGraph-based)                        │
└─────────────────────────────────────────────────────────────┘
                              │
┌─────────────────────────────────────────────────────────────┐
│                       Agent Layer                           │
│  ┌────────────────────────────────────────────────────────┐│
│  │ Ingestion Agents                                       ││
│  │  • QueryGenerator  • ArxivScout  • PaperFetcher       ││
│  └────────────────────────────────────────────────────────┘│
│  ┌────────────────────────────────────────────────────────┐│
│  │ Processing Agents                                      ││
│  │  • ChunkAgent  • EntityAgent  • GraphWriter           ││
│  │  • VectorIndexer  • SimilarityLinker                  ││
│  └────────────────────────────────────────────────────────┘│
│  ┌────────────────────────────────────────────────────────┐│
│  │ Query Agents                                           ││
│  │  • Planner  • Retriever  • Neo4jEnricher              ││
│  │  • EvidenceBuilder  • Validator  • Critic             ││
│  └────────────────────────────────────────────────────────┘│
└─────────────────────────────────────────────────────────────┘
                              │
┌─────────────────────────────────────────────────────────────┐
│                      Service Layer                          │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────┐ │
│  │ Paper Source │  │     LLM      │  │  Embedding Gen   │ │
│  │  (ArXiv,     │  │  (Claude,    │  │  (Nomic, OpenAI) │ │
│  │   OpenAlex)  │  │   GPT-4)     │  │                  │ │
│  └──────────────┘  └──────────────┘  └──────────────────┘ │
└─────────────────────────────────────────────────────────────┘
                              │
┌─────────────────────────────────────────────────────────────┐
│                      Storage Layer                          │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────┐ │
│  │    Neo4j     │  │    FAISS     │  │  File Cache      │ │
│  │ (Knowledge   │  │  (Vector     │  │  (Embeddings,    │ │
│  │   Graph)     │  │   Store)     │  │   PDFs, ArXiv)   │ │
│  └──────────────┘  └──────────────┘  └──────────────────┘ │
└─────────────────────────────────────────────────────────────┘
                              │
┌─────────────────────────────────────────────────────────────┐
│                   Observability Layer                       │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────┐ │
│  │   Jaeger     │  │  Prometheus  │  │  Structured      │ │
│  │  (Tracing)   │  │  (Metrics)   │  │   Logging        │ │
│  └──────────────┘  └──────────────┘  └──────────────────┘ │
└─────────────────────────────────────────────────────────────┘
```

### Design Principles

1. **Separation of Concerns**: Three independent workflows (paper fetching, embedding, validation) can run separately or sequentially
2. **Agent Modularity**: Each agent is self-contained with clear inputs/outputs
3. **Stateful Orchestration**: LangGraph manages workflow state and agent transitions
4. **Async-First**: All I/O operations use async/await for optimal performance
5. **Caching Strategy**: Multi-level caching (embeddings, API responses, PDFs) reduces latency
6. **Observability**: Comprehensive tracing, metrics, and logging across all components
7. **Type Safety**: Pydantic models enforce data contracts throughout the system

### Data Flow

```
User Input → Query Generation → Paper Search → Paper Fetching
                                                      ↓
                                            PDF Processing → Chunking
                                                      ↓
                                            Entity Extraction → Graph Writing
                                                      ↓
                                            Embedding Generation → Vector Indexing
                                                      ↓
                                            Similarity Linking (Papers + Chunks)
                                                      ↓
Query Planning → Semantic Retrieval → Graph Enrichment → Evidence Building
                                                      ↓
                                            Validation → Confidence Evaluation → Critic
                                                      ↓
                                            Structured Result → User
```

## Features

### Paper Acquisition

- **Multi-Source Support**: ArXiv and OpenAlex integration
- **Intelligent Search**: LLM-generated queries or manual query input
- **Semantic Ranking**: Relevance scoring and filtering
- **Date Filtering**: Publication year range constraints
- **PDF Download**: Automatic full-text acquisition with fallback to abstracts
- **Rate Limiting**: Compliant with API rate limits
- **Caching**: Persistent storage of search results and PDFs

### Text Processing

- **Hierarchical Chunking**: Section-aware segmentation with 7-level priority system
- **Entity Extraction**: Named Entity Recognition using spaCy
- **Relation Extraction**: Dependency-based relationship identification
- **Citation Tracking**: Inter-paper citation graph construction

### Knowledge Representation

- **Graph Database**: Neo4j stores papers, chunks, entities, and relationships
- **Vector Store**: FAISS indexes semantic embeddings for similarity search
- **Similarity Linking**: 
  - Paper-level similarity (title + abstract)
  - Chunk-level similarity (K-NN semantic search)
  - Configurable thresholds and limits

### Retrieval and Validation

- **Two-Stage Retrieval**:
  1. Coarse semantic search (FAISS)
  2. Priority-based reranking with diversity control

- **Graph Enrichment**: Neo4j traversal for entity and citation context
- **Smart Context Building**: Token-aware evidence assembly (8000 token budget)
- **LLM Validation**: Claude-powered claim analysis with structured reasoning
- **Confidence Scoring**: Multi-factor confidence evaluation
- **Quality Assurance**: Critic agent reviews validation quality

### Observability

- **Distributed Tracing**: Jaeger integration for request flow visualization
- **Metrics Collection**: Prometheus-compatible metrics export
- **Structured Logging**: JSON logs with correlation IDs
- **Real-time Monitoring**: SSE streams for live workflow progress

## System Components

### Agents (17 Total)

#### Ingestion Agents (3)
- **QueryGenerator**: LLM-powered query formulation from findings
- **ArxivScout**: Paper search across ArXiv/OpenAlex with ranking
- **PaperFetcher**: PDF download and metadata extraction

#### Processing Agents (5)
- **ChunkAgent**: Hierarchical text segmentation
- **EntityAgent**: Named entity and relation extraction
- **GraphWriter**: Neo4j graph population
- **VectorIndexer**: FAISS index construction
- **ParallelWriter**: Concurrent graph and vector writes
- **SimilarityLinker**: Semantic similarity relationship creation

#### Query Agents (7)
- **DatabaseChecker**: Validates existing data availability
- **Planner**: Query planning and search strategy
- **Retriever**: Two-stage semantic retrieval
- **Neo4jEnricher**: Graph-based context enrichment
- **EvidenceBuilder**: Evidence compilation and ranking
- **Validator**: LLM-based claim validation
- **ConfidenceEvaluator**: Multi-factor confidence scoring
- **Critic**: Quality assurance and reasoning review

### Services

- **Paper Sources**: ArXiv and OpenAlex API clients
- **LLM Interface**: Anthropic Claude and OpenAI GPT-4 integration
- **Embedding Generator**: Nomic embedding models
- **Graph Manager**: Neo4j driver with connection pooling
- **Vector Store**: FAISS index with metadata management
- **Cache Manager**: Multi-level caching system

### Infrastructure

- **FastAPI Backend**: Async REST API with SSE streaming
- **Streamlit Frontend**: Interactive web UI with real-time updates
- **Neo4j Database**: Graph database for knowledge representation
- **Jaeger**: Distributed tracing system
- **Prometheus**: Metrics collection and monitoring

## Workflows

### 1. Paper Fetching Workflow

**Purpose**: Discover and download relevant research papers

**Flow**: `QueryGenerator (optional) → ArxivScout → PaperFetcher → END`

**Inputs**:
- Scientific finding (for LLM query generation) OR
- Manual query string
- Paper source (arxiv/openalex)
- Max papers count
- Date range filters

**Outputs**:
- Paper artifacts (metadata + PDFs)
- Session state for downstream workflows

### 2. Embedding Workflow

**Purpose**: Process papers into searchable knowledge graph and vector index

**Flow**: `ChunkAgent → EntityAgent → ParallelWriter(GraphWriter || VectorIndexer) → SimilarityLinker → END`

**Inputs**:
- Paper artifacts from fetching workflow OR
- Auto-discovery from pdf_cache/cache directories

**Outputs**:
- Neo4j knowledge graph
- FAISS vector index
- Similarity relationships

### 3. Retrieval Workflow

**Purpose**: Validate scientific claims using retrieved evidence

**Flow**: `Planner → Retriever → Neo4jEnricher → EvidenceBuilder → Validator → ConfidenceEvaluator → Critic → END`

**Inputs**:
- Scientific claim to validate
- Optional finding optimization
- Existing knowledge graph + vector index

**Outputs**:
- Validation result (supported/not supported/partial)
- Confidence score
- Evidence chunks with citations
- Reasoning and limitations

## Technology Stack

| Component | Technology | Purpose |
|-----------|-----------|---------|
| **Orchestration** | LangGraph 0.0.20 | Stateful workflow management |
| **Backend** | FastAPI 0.104+ | Async REST API with SSE |
| **Frontend** | Streamlit 1.28+ | Interactive web UI |
| **Knowledge Graph** | Neo4j 5.15 | Graph database |
| **Vector Store** | FAISS (IndexFlatIP) | Similarity search |
| **LLM** | Claude | Validation and reasoning |
| **Embeddings** | Nomic-AI nomic-embed-text-v1.5 | Semantic embeddings (768-dim) |
| **NLP** | spaCy en_core_web_md | Entity extraction |
| **Tracing** | Jaeger 1.52 | Distributed tracing |
| **Metrics** | Prometheus | System monitoring |
| **Caching** | File-based + Memory | Multi-level caching |
| **PDF Processing** | PyPDF2 | Text extraction |
| **Paper APIs** | ArXiv API, OpenAlex | Paper metadata |

## Project Structure

```
Groundtruth_validation_search/
├── research_claim_validator/
│   ├── agents/                    # 17 specialized agents
│   │   ├── arxiv_scout.py
│   │   ├── chunk_agent.py
│   │   ├── entity_agent.py
│   │   ├── graph_writer.py
│   │   ├── vector_indexer.py
│   │   ├── similarity_linker.py
│   │   ├── retriever.py
│   │   ├── validator.py
│   │   └── ...
│   ├── api/                       # FastAPI application
│   │   ├── main.py
│   │   ├── routers/
│   │   │   └── validation.py      # API endpoints
│   │   └── middleware/            # Rate limiting, tracing
│   ├── frontend/                  # Streamlit UI
│   │   ├── app_modular.py
│   │   ├── sse_client.py
│   │   └── components/            # UI components
│   ├── workflow/                  # Workflow orchestration
│   │   ├── orchestrator.py
│   │   ├── paper_fetching_workflow.py
│   │   ├── embedding_workflow.py
│   │   ├── retrieval_workflow.py
│   │   ├── event_emitter.py
│   │   └── state_manager.py
│   ├── services/                  # External integrations
│   │   ├── papers/                # ArXiv, OpenAlex
│   │   ├── llm/                   # Claude, GPT-4
│   │   ├── graph/                 # Neo4j manager
│   │   └── embeddings/            # Vector generation
│   ├── models/                    # Data models
│   │   ├── artifacts.py
│   │   ├── events.py
│   │   └── workflow_state.py
│   ├── config/
│   │   └── settings.py            # Configuration
│   ├── utils/                     # Utilities
│   │   ├── cache_manager.py
│   │   └── latency_tracker.py
│   └── observability/             # Tracing, metrics
│       ├── tracing.py
│       └── metrics.py
├── tests/                         # Test suite
│   ├── unit/
│   ├── integration/
│   ├── conftest.py
│   └── run_tests.py
├── cache/                         # Cached data
│   ├── arxiv_*.json              # ArXiv search results
│   └── emb_*.npy                 # Cached embeddings
├── pdf_cache/                     # Downloaded PDFs
├── vector_index/                  # FAISS index files
│   ├── faiss.index
│   └── metadata.json
├── logs/                          # Application logs
├── docker-compose.yml             # Service orchestration
├── Dockerfile                     # Backend container
├── Dockerfile.frontend            # Frontend container
├── requirements.txt               # Python dependencies
├── .env.example                   # Configuration template
├── README.md                      # This file
├── SETUP.md                       # Setup and installation guide
├── DEPLOYMENT.md                  # Deployment guide
├── NEO4J_SETUP.md                 # Neo4j setup guide
├── multiagent_rag.ipynb          # Development notebook
└── evaluation_ragas.ipynb        # Evaluation notebook
```

## Development Notebooks

### multiagent_rag.ipynb and knowledge_graph_optimized.ipynb

The primary development notebook demonstrating the complete multi-agent RAG system. This notebook served as the prototype for the production implementation and includes:

- End-to-end workflow examples
- Agent initialization and testing
- Knowledge graph construction
- Vector index building
- Validation examples with real papers
- Performance benchmarking

**Use cases**:
- Understanding system architecture
- Testing individual agents
- Prototyping new features
- Educational reference

### evaluation_ragas.ipynb

Comprehensive evaluation notebook using the RAGAS framework to assess system performance:

**Metrics Evaluated**:
- **Faithfulness**: Factual consistency of generated answers
- **Answer Relevancy**: Pertinence to the input question
- **Context Precision**: Relevance of retrieved context
- **Context Recall**: Proportion of relevant context retrieved

**Features**:
- Automated evaluation pipeline
- Ground truth dataset construction
- Comparison across configurations
- Performance visualization
- Metric aggregation and reporting

**Use cases**:
- System evaluation and benchmarking
- A/B testing configuration changes
- Quality assurance validation
- Research and publication

## Testing and Evaluation

### Unit Tests

```bash
# Run all unit tests
pytest tests/unit/

# With coverage
pytest tests/unit/ --cov=research_claim_validator --cov-report=html
```

### Integration Tests

```bash
# Run integration tests (requires Neo4j)
pytest tests/integration/

# Specific test
pytest tests/integration/test_workflow.py
```

### End-to-End Testing

```bash
# Run complete system test
python run_tests.py
```

### Evaluation Metrics

Run RAGAS evaluation (see `evaluation_ragas.ipynb`):

```python
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy

# Load test dataset
dataset = ...

# Run evaluation
results = evaluate(
    dataset=dataset,
    metrics=[faithfulness, answer_relevancy]
)

print(results)
```

**Last Updated**: April 2026

---

**Designed and Developed by Tolulope Ale**

*Multi-Agent GraphRAG System for Research Finding Validation*


