Metadata-Version: 2.4
Name: deepdive-research
Version: 0.3.2
Summary: Local-first, citation-honest, reproducible AI research agent. Span-grounded citations, trace + replay, multi-format export, MCP server. Runs on Ollama.
Author: Open Intelligence Labs contributors
License: MIT
Project-URL: Homepage, https://github.com/openintelligence-labs/deepdive
Project-URL: Repository, https://github.com/openintelligence-labs/deepdive
Keywords: ai,research,agent,ollama,local-first,perplexity
Classifier: Development Status :: 3 - Alpha
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: actants>=0.5.0
Requires-Dist: fastapi>=0.110
Requires-Dist: uvicorn[standard]>=0.27
Requires-Dist: httpx>=0.27
Requires-Dist: selectolax>=0.3.21
Requires-Dist: pydantic>=2.6
Requires-Dist: pydantic-settings>=2.2
Requires-Dist: structlog>=24.1
Requires-Dist: sse-starlette>=2.0
Requires-Dist: ddgs>=7.0
Requires-Dist: click>=8.1
Requires-Dist: rich>=13.7
Provides-Extra: mcp
Requires-Dist: mcp<2,>=1.0; extra == "mcp"
Provides-Extra: corpus
Requires-Dist: sqlite-vec>=0.1.6; extra == "corpus"
Requires-Dist: pypdf>=4.0; extra == "corpus"
Provides-Extra: all
Requires-Dist: mcp<2,>=1.0; extra == "all"
Requires-Dist: sqlite-vec>=0.1.6; extra == "all"
Requires-Dist: pypdf>=4.0; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.23; extra == "dev"
Requires-Dist: ruff<0.17,>=0.16; extra == "dev"
Requires-Dist: mcp<2,>=1.0; extra == "dev"
Requires-Dist: sqlite-vec>=0.1.6; extra == "dev"
Requires-Dist: pypdf>=4.0; extra == "dev"
Dynamic: license-file

# DeepDive

[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
[![Python](https://img.shields.io/badge/python-3.12%2B-blue)](https://www.python.org/)
[![Tests](https://img.shields.io/badge/tests-126%20passing-brightgreen)]()
[![Powered by actants](https://img.shields.io/badge/powered%20by-actants-7c3aed)](https://github.com/openintelligence-labs/actants)

A local-first AI research agent. Asks a question, searches the web (or your own documents), reads the sources, and writes a cited report. Every claim is checked against a verbatim excerpt from its source — claims the validator can't verify are dropped.

```bash
pip install deepdive-research
ollama pull llama3.2
deepdive research "When did Apollo 11 land on the Moon?" -o report.md
```

No API key required. Runs on Ollama by default.

---

## Features

### Span-grounded citations

The claim extractor returns each claim alongside the verbatim excerpt that supports it. A post-extraction validator re-checks every excerpt against the fetched source page via substring match. Claims that don't ground get dropped from the report.

The Markdown report ends with an Evidence appendix listing every claim and its verbatim source excerpt:

```markdown
**1.** Apollo 11 landed on the Moon on July 20, 1969.
   - [✓] [Apollo 11 - Wikipedia](https://en.wikipedia.org/wiki/Apollo_11)
     > Apollo 11 (July 16-24, 1969) was the American spaceflight that
     > first landed humans on the Moon, and the fifth crewed mission of
     > NASA's Apollo program.
```

### Trace and replay

Every research run can record a trace of every LLM call, search, and scrape. Replay reconstructs the report offline, byte-for-byte.

```bash
deepdive research "..." -o report.md         # auto-records report.md.trace.jsonl
deepdive replay report.md.trace.jsonl        # produces an identical report, no network
deepdive trace verify report.md.trace.jsonl  # re-validates every excerpt
deepdive inspect report.md.trace.jsonl       # event counts
```

### Source restriction

Restrict search to your own allow/block lists of hostnames. Subdomains match; wildcards like `*.gov` are supported.

```bash
deepdive research "..." --allow-domains arxiv.org,nih.gov
deepdive research "..." --block-domains medium.com,*.substack.com
```

### Local corpus and offline mode

Index your own documents into a sqlite-vec database, then research them with no outbound network calls.

```bash
deepdive index ~/Documents/papers -o ~/.local/share/deepdive/index.db
deepdive research "..." --corpus ~/.local/share/deepdive/index.db --offline
```

In `--offline` mode the LLM endpoint must be loopback (Ollama is fine; cloud APIs raise) and non-loopback URLs are dropped at the scraper level.

### Multi-format export

```bash
deepdive research "..." -o report.md                      # markdown (default)
deepdive research "..." -o paper.tex --export latex       # +references.bib
deepdive research "..." -o data.json --export json
deepdive research "..." -o note.md --export obsidian      # YAML + [[wikilinks]]
deepdive research "..." -o page.md --export notion
deepdive research "..." -o refs.bib --export bibtex
```

### Use as an MCP server

Expose DeepDive's `research` tool to any MCP-compatible client.

```bash
deepdive serve-mcp                         # stdio
deepdive serve-mcp --http --port 8765      # Streamable HTTP
```

---

## Install

```bash
pip install "deepdive-research[all]"     # everything: MCP server, corpus indexing
pip install "deepdive-research[mcp]"     # MCP server only
pip install "deepdive-research[corpus]"  # local corpus indexing only

ollama pull llama3.2           # tool-capable LLM (default)
ollama pull nomic-embed-text   # embeddings (only needed for --corpus)
```

---

## Architecture

DeepDive is built on [actants](https://github.com/openintelligence-labs/actants). The pipeline:

```
question
   │
   ▼  query generation              (LLM)
search backend                       (DuckDuckGo / SearxNG / local corpus)
   │                                 │  optional: --allow / --block filter
   ▼  scraper                        (web / corpus / offline)
claim extractor                      (LLM, returns excerpts)
   │
   ▼  span-grounding validator       (drops claims without verbatim excerpts)
cross-reference
   │
   ▼  report builder                 (LLM)
Markdown / LaTeX / BibTeX / JSON / Obsidian / Notion
```

Every external call is wrapped in optional recorders, which is how trace and replay work.

---

## What's not in scope

- Hosted SaaS or paid tier — DeepDive is open-source and runs on your hardware
- Multi-modal output (charts, infographics)
- Conversational follow-up on a single report — single-shot research only
- Browser automation
- Code execution as a research step

---

## Status

- **Version:** 0.3.1
- **License:** MIT
- **Python:** 3.12+
- **Framework:** [actants ≥ 0.5.0](https://github.com/openintelligence-labs/actants)

Part of [Open Intelligence Labs](https://github.com/openintelligence-labs).
