Metadata-Version: 2.4
Name: vidchain
Version: 1.0.2
Summary: VidChain: High-fidelity multimodal RAG framework featuring the IRIS Intelligence Agent
Author-email: Rahul Sharma <rahulsharma.hps@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/rahulsiiitm/videochain-python
Project-URL: Bug Tracker, https://github.com/rahulsiiitm/videochain-python/issues
Project-URL: Changelog, https://github.com/rahulsiiitm/videochain-python/blob/main/CHANGELOG.md
Keywords: video-rag,multimodal,ai,computer-vision,whisper,yolo,ollama,surveillance,nlp
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Multimedia :: Video
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: opencv-python>=4.8.0
Requires-Dist: ultralytics>=8.0.0
Requires-Dist: torch>=2.1.0
Requires-Dist: torchvision>=0.16.0
Requires-Dist: torchaudio>=2.1.0
Requires-Dist: pillow<12.0.0,>=9.0.0
Requires-Dist: openai-whisper>=20231117
Requires-Dist: moviepy>=2.0.0
Requires-Dist: imageio-ffmpeg>=0.4.9
Requires-Dist: librosa>=0.10.0
Requires-Dist: soundfile>=0.12.0
Requires-Dist: easyocr>=1.7.0
Requires-Dist: deepface>=0.0.90
Requires-Dist: tf-keras>=2.16.0
Requires-Dist: sentence-transformers>=2.7.0
Requires-Dist: chromadb>=0.5.0
Requires-Dist: faiss-cpu>=1.7.4
Requires-Dist: litellm>=1.30.0
Requires-Dist: google-generativeai>=0.5.0
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: ollama>=0.1.0
Requires-Dist: customtkinter>=5.2.2
Requires-Dist: requests>=2.31.0
Requires-Dist: numpy>=1.24.0
Requires-Dist: networkx>=3.0
Requires-Dist: darkdetect>=0.8.0
Requires-Dist: tqdm>=4.65.0
Requires-Dist: fastapi>=0.100.0
Requires-Dist: uvicorn>=0.23.0
Requires-Dist: nvidia-ml-py>=12.0.0
Provides-Extra: clip
Requires-Dist: transformers>=4.40.0; extra == "clip"
Provides-Extra: full
Requires-Dist: transformers>=4.40.0; extra == "full"
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
Requires-Dist: black>=23.0.0; extra == "dev"
Requires-Dist: ruff>=0.1.0; extra == "dev"
Requires-Dist: mypy>=1.0.0; extra == "dev"
Requires-Dist: build>=1.0.0; extra == "dev"
Requires-Dist: twine>=4.0.0; extra == "dev"
Dynamic: license-file

<div align="center">

# VidChain

**High-Fidelity Multimodal RAG Framework for Forensic Video Intelligence**

![PyPI](https://img.shields.io/badge/PyPI-VidChain-blue?style=flat-square) ![Engine](https://img.shields.io/badge/Engine-IRIS-orange?style=flat-square) ![RAG](https://img.shields.io/badge/RAG-Multimodal-9cf?style=flat-square) ![Vector DB](https://img.shields.io/badge/Vector%20DB-ChromaDB-lightgrey?style=flat-square) ![Orchestration](https://img.shields.io/badge/Orchestration-LiteLLM-brightgreen?style=flat-square) ![Local First](https://img.shields.io/badge/Local%20First-Yes-success?style=flat-square)

![Python](https://img.shields.io/badge/Python-3.11+-blue?style=flat-square) ![CUDA](https://img.shields.io/badge/CUDA-12.1+-green?style=flat-square) [![Downloads](https://static.pepy.tech/badge/vidchain)](https://pepy.tech/project/vidchain) [![PyPI version](https://badge.fury.io/py/vidchain.svg)](https://pypi.org/project/VidChain/) ![License](https://img.shields.io/badge/License-MIT-yellow?style=flat-square) [![GitHub stars](https://img.shields.io/github/stars/rahulsiiitm/videochain-python?style=flat-square)](https://github.com/rahulsiiitm/videochain-python/stargazers)

![VidChain Banner](https://raw.githubusercontent.com/rahulsiiitm/videochain-python/main/assets/image.png)

</div>

VidChain is a local-first multimodal RAG framework powered by the **IRIS Engine** (Intelligent Retrieval & Insight System). It decomposes video into visual, auditory, OCR, and temporal signal streams and fuses them into a queryable intelligence layer, intended for forensic analysis, security auditing, and automated video summarization with on-device privacy by default.

---

## Table of Contents

1. [Overview](#overview)
2. [Features](#features)
3. [Installation](#installation)
4. [Configuration](#configuration)
5. [Quick Start](#quick-start)
6. [CLI Reference](#cli-reference)
7. [SDK: Modular Sensor Matrix](#sdk-modular-sensor-matrix)
8. [REST API](#rest-api)
9. [Architecture](#architecture)
10. [Troubleshooting](#troubleshooting)
11. [Contributing](#contributing)
12. [License](#license)

---

## Overview

VidChain turns raw video into a queryable intelligence layer. Each ingested video is processed through a modular sensor pipeline (visual, audio, OCR, motion, behavioral), decomposed into an isolated Temporal Knowledge Graph, and fused with vector retrieval to produce grounded, timestamp-cited answers. Inference runs entirely on-device by default; cloud models are supported as an explicit opt-in, not a requirement.

---

## Features

| Capability | Description |
| :--- | :--- |
| 4-Route Agentic Router | Classifies queries into Narrative Summarization, Local Forensic Search, Global Master Intelligence, and Conversational Dialogue |
| Global Master Intelligence | Cross-video entity tracking via a macro-graph, enabling pattern recognition across isolated sessions |
| Temporal Persistence | Chronological reasoning that bridges frame gaps and maintains state continuity between sensor logs |
| Recursive Map-Reduce Summarizer | Collapses hours of video into coherent reports without hitting LLM context limits |
| Neural Concurrency Locking | Prevents state corruption during simultaneous ingestion and query operations |
| Local-First Execution | Vision (Moondream) and reasoning (Llama 3 via Ollama) run on-device by default; no data leaves the machine unless a cloud model is explicitly configured |

---

## Installation

### Prerequisites

| Requirement | Version |
| :--- | :--- |
| Python | 3.11+ |
| CUDA | 12.1+ |
| [Ollama](https://ollama.com) | Latest (running) |
| Node.js | v18+ (for web portal) |

### Option A — Install from PyPI (recommended)

```bash
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
pip install vidchain
```

### Option B — Install from source (for development)

```bash
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
git clone https://github.com/rahulsiiitm/videochain-python
cd videochain-python
pip install -e .
```

### Model weights (required either way)

```bash
ollama pull moondream   # Vision Language Model
ollama pull llama3      # Language Model for reasoning & routing
```

> **CPU Fallback:** If no CUDA device is detected, VidChain automatically degrades to CPU mode — no code changes required.

---

## Configuration

| Variable | Required | Description |
| :--- | :--- | :--- |
| `GEMINI_API_KEY` | Only if using a Gemini model via `--llm gemini/...` | API key for Google's Gemini models, used through LiteLLM |
| `db_path` (constructor arg, not env var) | Yes | Local directory where ChromaDB vectors and Temporal Knowledge Graphs are stored |

No other environment variables are required for the default local-only configuration. Any LiteLLM-compatible provider can be substituted for `--llm` or `--vlm`; check the [LiteLLM provider docs](https://docs.litellm.ai/docs/providers) for the corresponding key name if using a provider other than Gemini or Ollama.

---

## Quick Start

```python
from vidchain import VidChain

vc = VidChain(db_path="./forensic_vault")

# Ingest video (runs full default pipeline)
video_id = vc.ingest(video_source="interview_01.mp4")

# Query
response = vc.ask("What is the main topic of discussion?", video_id=video_id)
print(response)

# Summarize
summary = vc.summarize_video(video_id=video_id, mode="concise")
print(summary)
```

---

## CLI Reference

### `vidchain-serve`

Launches the FastAPI backend and Next.js dashboard.

```bash
vidchain-serve
```

- API available at `http://localhost:8000`
- Dashboard opens at `http://localhost:8000`

### `vidchain-analyze`

Headless video ingestion from the terminal.

```bash
vidchain-analyze path/to/video.mp4 --vlm moondream
```

| Flag | Description |
| :--- | :--- |
| `--vlm <model>` | Vision model to use (default: `moondream`, local) |
| `--llm <model>` | Reasoning model to use (default: `ollama/llama3`, local) |
| `--fast` | Replaces VLM with YOLO for high-speed detection (ideal for long CCTV footage) |
| `--emotion` | Injects DeepFace emotion analysis node |
| `--action` | Injects MobileNetV3 action classification node |

**Model substitution:** VidChain uses LiteLLM, so any compatible model can be swapped in, including cloud models if higher reasoning quality is preferred over on-device execution:

```bash
# Local (default)
vidchain-analyze video.mp4 --llm "ollama/llama3"

# Cloud (opt-in, requires API key export)
export GEMINI_API_KEY="your_api_key"
vidchain-analyze video.mp4 --llm "gemini/gemini-2.5-flash"

# Custom VLM
vidchain-analyze video.mp4 --vlm "llava:7b"
```

---

## SDK: Modular Sensor Matrix

VidChain uses a LangChain-inspired composable pipeline. Each `Node` handles one sensing modality; chains are assembled per use case.

### Available Nodes

| Node | Modality | Description |
| :--- | :--- | :--- |
| `AdaptiveKeyframeNode` | Logic | Gaussian-differential sampling — drops redundant frames to reduce compute load |
| `LlavaNode` | Visual | Scene semantics, descriptive captions, and situational context |
| `YoloNode` | Visual | High-speed discrete object detection (lightweight fallback for `LlavaNode`) |
| `WhisperNode` | Audio | Speech transcription and acoustic anomaly detection (e.g., shouts) |
| `OcrNode` | Text | Digital trace extraction — license plates, screens, documents |
| `TrackerNode` | Motion | Persistent object tracking (IoU) and camera motion estimation (Optical Flow) |
| `EmotionNode` | Behavioral | Facial sentiment analysis |
| `ActionNode` | Behavioral | Human activity classification via MobileNetV3 |

### Custom Pipeline Example

```python
from vidchain import VidChain
from vidchain.pipeline import VideoChain
from vidchain.nodes import AdaptiveKeyframeNode, LlavaNode, OcrNode, TrackerNode, YoloNode

vc = VidChain(db_path="./forensic_vault")

surveillance_chain = VideoChain(nodes=[
    AdaptiveKeyframeNode(change_threshold=1.5),  # High sensitivity
    YoloNode(),                                  # Structured detections
    TrackerNode(),
    LlavaNode(model_name="moondream"),           # Semantic scene description
    OcrNode(),
])

video_id = vc.ingest(
    video_source="gate_camera_04.mp4",
    chain=surveillance_chain
)

response = vc.ask(
    "Were there any vehicles with visible license plates after 14:00?",
    video_id=video_id
)
print(response)
```

---

## REST API

Exposed when running `vidchain-serve`.

| Method | Endpoint | Description |
| :--- | :--- | :--- |
| `GET` | `/api/health` | System status and list of ingested video IDs |
| `POST` | `/api/sessions` | Create a new isolated neural session |
| `POST` | `/api/ingest` | Submit a video file path for background processing |
| `POST` | `/api/query` | Run a natural language query through the Agentic Router |
| `GET` | `/api/media-stream` | Serve local video securely for frontend playback |

---

## Architecture

### Isolated GraphRAG

Each ingested video generates a dedicated Temporal Knowledge Graph (`.pkl`). The RAG engine retrieves semantically relevant chunks from ChromaDB and fuses them with structured graph data (co-occurrences, tracking IDs, timestamps). Memory boundaries are strictly enforced — no cross-video context bleed.

### The Neural Lens

Every query response is paired with a Base64-encoded visual snapshot extracted directly from the referenced timestamp, providing visual grounding for AI-generated claims.

---

## Troubleshooting

| Symptom | Likely Cause | Fix |
| :--- | :--- | :--- |
| `vidchain-serve` fails to start | Ollama not running | Start Ollama before launching VidChain (`ollama serve`) |
| Ingestion runs but very slowly | No CUDA device detected, running on CPU fallback | Confirm `nvidia-smi` shows a GPU; reinstall the CUDA-enabled Torch build from [Installation](#installation) |
| `--llm gemini/...` fails with an auth error | `GEMINI_API_KEY` not exported | `export GEMINI_API_KEY="your_api_key"` before running the command |
| Dashboard loads but shows no videos | Wrong `db_path` between ingest and query calls | Ensure `VidChain(db_path=...)` points to the same directory across sessions |

---

## Contributing

Issues and pull requests are welcome via [GitHub Issues](https://github.com/rahulsiiitm/videochain-python/issues). For substantial changes, open an issue first to discuss scope before submitting a PR.

---

## License

MIT — See [LICENSE](LICENSE) for details.

**Author:** Rahul Sharma — IIIT Manipur
[Portfolio](https://rahul.aishtrex.com) · [GitHub](https://github.com/rahulsiiitm)

<div align="center">

*Star this repo if you find it useful — it helps the IRIS Engine grow.*

</div>
