Metadata-Version: 2.4
Name: litdiscover
Version: 2.0.0
Summary: Queue-driven, adaptive literature discovery engine — bidirectional citation traversal, LLM screening, thematic synthesis
Author-email: David Goh <davidcagoh@gmail.com>
License-Expression: MIT
Keywords: literature-review,citation-graph,systematic-review,llm,research-tools
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: supabase>=2.4
Requires-Dist: httpx>=0.27
Requires-Dist: tenacity>=8.2
Requires-Dist: rich>=13
Requires-Dist: tomli>=2.0
Requires-Dist: openai>=1.30
Requires-Dist: pymupdf>=1.24
Requires-Dist: typer>=0.12
Requires-Dist: anyio>=4.3
Requires-Dist: numpy>=1.24
Dynamic: license-file

# LitDiscover

**Queue-driven, adaptive literature discovery.** Give it seed papers and inclusion criteria; LitDiscover traverses the citation graph bidirectionally, surfaces semantically similar papers from outside the graph, screens candidates with an LLM, and writes a thematic narrative review with numbered citations.

This is the active development repository. The empirical paper validating the approach ([*Robust Literature Discovery from Minimal Seeds*](https://github.com/davidcagoh/robust-literature-discovery)) is maintained separately.

> **Looking for the older v1 pipeline?** See [automated-lit-reviews](https://github.com/davidcagoh/automated-lit-reviews) (archived).

---

## Quick start

```bash
# 1. Install
git clone https://github.com/davidcagoh/automated-lit-reviews-v2.git
cd automated-lit-reviews-v2
pip install -e .
cp .env.example .env   # fill in keys (see Environment variables)

# 2. Apply DB schema (once, in Supabase SQL editor)
#    paste contents of supabase/schema.sql

# 3. Initialise a project from a folder of seed PDFs
litdiscover init projects/my-review/project.toml

# 4. Run to stability
litdiscover run projects/my-review/project.toml

# 5. Extract structured fields, then synthesise
litdiscover extract my-review
litdiscover synthesize my-review
```

Output: `projects/my-review/my-review_review_numbered.md` — a thematic narrative review with sequential `[N]` citations and a `## References` section.

---

## Environment variables

```
SUPABASE_URL=https://<project>.supabase.co
SUPABASE_SERVICE_KEY=<service-role key>

SEMANTIC_SCHOLAR_API_KEY=<key>   # required for S2 traversal + embeddings
GEMINI_API_KEY=<key>             # screening, extraction, synthesis (default)
GROQ_API_KEY=<key>               # alternative screening backend (free, fast)
```

---

## CLI reference

```bash
litdiscover init   <project.toml>    # create DB record, ingest seeds
litdiscover run    <project.toml>    # main loop: traverse → screen → repeat
litdiscover status <slug>            # corpus stats
litdiscover reset  <slug>            # reset paper queue, keep criteria history
litdiscover extract   <slug>         # structured extraction (post-stability)
litdiscover synthesize <slug>        # thematic narrative review
```

---

## Architecture

LitDiscover is a queue-driven state machine (`litdiscover/core/loop.py`). All state is persisted in Supabase, making runs fully restartable.

```
SEED papers
    │
    ▼
SEARCH (S2 keyword query)
    │
    ▼
SCREEN (LLM batch, up to 20 papers)
    │
  included?
    │── yes → TRAVERSE (forward + backward citation graph)
    │          PDF worker (background reference extraction)
    │          SEARCH (escape-hatch keyword queries when yield is low)
    │          → new candidates added to queue
    │
    └── cycle yield < threshold for 2 rounds → STOP
    │
    ▼
EXTRACT  (structured fields per included paper, Gemini)
    │
    ▼
SYNTHESIZE (k-means cluster → per-theme narrative → numbered bibliography)
```

**Key design choices:**
- Backward traversal is never hub-filtered (survey papers with many references are the most valuable traversal targets).
- Forward traversal applies a Pareto-80 hub filter by default to avoid flooding the queue with citing papers of landmark works.
- An escape-hatch keyword search fires when cycle yield drops below threshold, recovering papers not citation-connected to the seed cluster.
- Background PDF workers run concurrently with screening, extracting reference lists to feed the next traversal cycle.

---

## Relationship to `robust-literature-discovery`

| Repository | Role |
|---|---|
| **`automated-lit-reviews-v2`** (this repo) | Deployed engine: queue loop, DB integrations, LLM screening, extraction, synthesis |
| [**`robust-literature-discovery`**](https://github.com/davidcagoh/robust-literature-discovery) | Empirical paper: APS benchmark, publication figures, manuscript (*Robust Literature Discovery from Minimal Seeds*) |

The paper repository is self-contained — you can reproduce all benchmark results without the full application.

---

## Tests

```bash
python -m pytest tests/ -v
```

All tests are mocked (no live DB or API connections required).

---

## Requirements

- Python ≥ 3.11
- A [Supabase](https://supabase.com) project (free tier works)
- Gemini API key (default backend) or Groq API key (free screening alternative)
- Semantic Scholar API key (for traversal and embeddings)
