Metadata-Version: 2.4
Name: epilens
Version: 0.1.0
Summary: Local-first, evidence-grounded RAG workflows for scientific literature analysis
Author-email: Vincenzo Perri <vincenzo.perri@isi.it>
Maintainer-email: Vincenzo Perri <vincenzo.perri@isi.it>
License-Expression: AGPL-3.0-only
Project-URL: Homepage, https://github.com/VinsRR/EpiLens
Project-URL: Documentation, https://github.com/VinsRR/EpiLens#readme
Project-URL: Repository, https://github.com/VinsRR/EpiLens.git
Project-URL: Issues, https://github.com/VinsRR/EpiLens/issues
Project-URL: Changelog, https://github.com/VinsRR/EpiLens/blob/main/CHANGELOG.md
Keywords: epidemiology,information extraction,large language models,literature review,retrieval-augmented generation,scientific literature
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE.txt
Requires-Dist: pydantic<3,>=2.7
Requires-Dist: typer<1,>=0.12
Requires-Dist: requests<3,>=2.31
Requires-Dist: numpy<3,>=1.24
Requires-Dist: python-dotenv<2,>=1.0
Requires-Dist: fastembed<1,>=0.3
Requires-Dist: pdfminer.six>=20221105
Requires-Dist: filelock<4,>=3.13
Provides-Extra: gemini
Requires-Dist: google-genai<2,>=1.0; extra == "gemini"
Provides-Extra: openai
Requires-Dist: openai<3,>=1.40; extra == "openai"
Provides-Extra: openrouter
Requires-Dist: openai<3,>=1.40; extra == "openrouter"
Provides-Extra: anthropic
Requires-Dist: anthropic<1,>=0.40; extra == "anthropic"
Provides-Extra: providers
Requires-Dist: epilens[anthropic,gemini,openai]; extra == "providers"
Provides-Extra: ui
Requires-Dist: streamlit<2,>=1.36; extra == "ui"
Requires-Dist: pandas<3,>=2.0; extra == "ui"
Provides-Extra: server
Requires-Dist: fastapi<1,>=0.115; extra == "server"
Requires-Dist: uvicorn<1,>=0.30; extra == "server"
Requires-Dist: python-multipart<1,>=0.0.9; extra == "server"
Provides-Extra: qdrant
Requires-Dist: qdrant-client~=1.13.3; extra == "qdrant"
Provides-Extra: mongo
Requires-Dist: pymongo<5,>=4.6; extra == "mongo"
Provides-Extra: grobid
Requires-Dist: grobid-client-python<1,>=0.0.8; extra == "grobid"
Requires-Dist: lxml<7,>=4.9; extra == "grobid"
Provides-Extra: faiss
Requires-Dist: faiss-cpu<2,>=1.8; extra == "faiss"
Provides-Extra: local-ml
Requires-Dist: torch<3,>=2.2; extra == "local-ml"
Requires-Dist: transformers<6,>=4.40; extra == "local-ml"
Requires-Dist: sentence-transformers<6,>=3; extra == "local-ml"
Requires-Dist: tqdm<5,>=4.66; extra == "local-ml"
Requires-Dist: unstructured[pdf]<1,>=0.15; extra == "local-ml"
Requires-Dist: rapidfuzz<4,>=3; extra == "local-ml"
Provides-Extra: all
Requires-Dist: epilens[server]; extra == "all"
Requires-Dist: epilens[ui]; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest==9.0.3; extra == "dev"
Requires-Dist: ruff==0.15.11; extra == "dev"
Requires-Dist: ty==0.0.35; extra == "dev"
Requires-Dist: tox==4.54.0; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: twine; extra == "dev"
Dynamic: license-file

# EpiLens

EpiLens is a local-first Python toolkit for evidence-grounded analysis of
scientific literature. It parses and indexes papers on your machine, retrieves
the passages relevant to a question, and can use a local or hosted language
model to produce answers and structured outputs with inspectable provenance.

It is designed for epidemiological and public-health literature, but its
retrieval, classification, and extraction workflows can be adapted to other
scientific domains. The distribution name, import package, and command are all
`epilens`.

## Install

EpiLens requires Python 3.10 or newer:

```bash
python -m pip install epilens
epilens --version
```

The base install includes lightweight PDF parsing and local semantic retrieval.
It does not install every model-provider SDK, a database server, or the larger
ML parsing stack. Add only what you plan to use; the [extras table](#optional-extras)
shows the choices.

## A Five-Minute First Run

This path starts with no API key, database, Docker service, or configuration
file. Replace `paper.pdf` with one of your own papers.

### 1. Inspect a paper

```bash
epilens inspect paper.pdf
```

This uses the included `pdfminer` parser. It verifies that the document is
readable and reports the detected sections and metadata. It works offline and
does not download a model.

### 2. Retrieve relevant evidence

```bash
epilens explore "What data sources were used?" --path paper.pdf --quality fast
```

The first semantic-retrieval command downloads the small
`sentence-transformers/all-MiniLM-L6-v2` embedding model (about 90 MB) and
caches it for later runs. No LLM key is needed; the command returns the most
relevant source passages.

### 3. Generate an answer when you are ready

Without a configured LLM, `ask` still returns the retrieved passages instead of
failing:

```bash
epilens ask "What data sources were used?" --path paper.pdf --quality fast
```

To generate a synthesized answer, install one provider and run the guided
setup. For example, with Gemini:

```bash
python -m pip install "epilens[gemini]"
epilens quickstart
epilens ask "What data sources were used?" --path paper.pdf --quality fast
```

`quickstart` writes the selected provider and key to a local, ignored `.env`
file. It does not contact optional services unless you pass `--probe`. You can
also use local Ollama without a provider SDK or API key.

### 4. Keep a reusable literature workspace

For more than one command or paper, create a workspace:

```bash
epilens init my-review
epilens index ./papers --workspace my-review
epilens papers --workspace my-review
epilens ask "Which papers mention GenBank?" --workspace my-review
```

The workspace is an ordinary directory containing `epilens.toml`, copied
papers, a local vector index, metadata, task definitions, outputs, and logs.
Run commands from inside it, or pass `--workspace` from anywhere.

## What EpiLens Does

- Parses PDFs, text, and Markdown with a deterministic lightweight default.
- Indexes one paper or a corpus into a local file store without a database
  server.
- Retrieves semantically relevant passages and keeps their paper/section
  provenance.
- Answers questions over retrieved evidence using Gemini, OpenAI, OpenRouter,
  Anthropic, or Ollama.
- Runs schema-validated classification across analyst-defined label axes.
- Extracts structured items such as data sources, supplementary links, and key
  references.
- Lets researchers define new classifiers and extractors as JSON task specs,
  without changing Python code.
- Scales to optional FAISS or Qdrant vector stores, MongoDB metadata, GROBID
  parsing, a FastAPI service, and a Streamlit Studio.

## Mental Model

EpiLens follows the three-stage workflow described in the accompanying paper:

1. **Ingest and index:** parse papers, split them into chunks, embed the chunks,
   and store them locally.
2. **Retrieve:** find the most relevant chunks for a query or fixed workflow
   template, optionally filtering by paper or section.
3. **Generate and validate:** pass only the selected evidence to an LLM,
   validate structured outputs against a schema, and retain provenance.

The LLM is a controlled linguistic step, not a knowledge database. Corpus
storage, retrieval, task logic, and validation remain local; only the selected
evidence is sent to a remote provider when you choose one.

## Main Commands

| Command | Purpose | Needs an LLM? |
| --- | --- | --- |
| `epilens inspect FILE` | Parse and summarize a document | No |
| `epilens explore QUERY --path PATH` | Retrieve matching passages | No |
| `epilens ask QUERY --path PATH` | Retrieve and optionally synthesize | Optional |
| `epilens init DIR` | Create a reusable workspace | No |
| `epilens index PATH` | Add papers to a local index | No |
| `epilens papers` | List indexed papers | No |
| `epilens classify --file FILE` | Apply a classification workflow | Yes |
| `epilens precision-miner --file FILE` | Extract structured items | Yes |
| `epilens tasks` | List built-in and user-defined tasks | No |
| `epilens doctor` | Check local essentials without network probes | No |
| `epilens studio` | Start the API and browser UI together | Depends on action |

Run `epilens COMMAND --help` for examples and advanced controls. `--quality
fast|balanced|accurate` is the simplest way to tune chunking and retrieval;
individual settings remain available for experienced users.

## Optional Extras

Install extras with `python -m pip install "epilens[EXTRA]"`.

| Extra | Adds |
| --- | --- |
| `gemini` | Google Gemini SDK |
| `openai` | OpenAI SDK |
| `openrouter` | OpenAI-compatible SDK used for OpenRouter |
| `anthropic` | Anthropic SDK |
| `providers` | All hosted-provider SDKs |
| `server` | FastAPI, Uvicorn, and uploads |
| `ui` | Streamlit and pandas |
| `all` | Local file-backed API + Studio UI (`server,ui`) |
| `faiss` | Local FAISS vector backend |
| `qdrant` | Qdrant client; a Qdrant server is still required |
| `mongo` | MongoDB client; a MongoDB server is still required |
| `grobid` | GROBID client and XML parser; a GROBID server is still required |
| `local-ml` | Torch-based embeddings and Unstructured `hi_res` PDF parsing |
| `dev` | Tests, lint, type checking, and build/release tools |

Examples:

```bash
python -m pip install "epilens[openai]"      # one hosted provider
python -m pip install "epilens[all,gemini]" # Studio plus Gemini
python -m pip install "epilens[grobid]"     # richer academic-PDF parsing
```

## Configuration

EpiLens loads environment variables from your shell and from the nearest `.env`
file. You do not need one for the local-first workflow. For generated answers,
`epilens quickstart` creates a minimal `.env` containing only the provider you
choose. Manual configuration looks like:

```dotenv
EPILENS_LLM_PROVIDER=gemini
EPILENS_LLM_MODEL=gemini-2.5-flash
GEMINI_API_KEY=your-key-here

# Optional local overrides
EPILENS_EMBED_PROVIDER=auto
EPILENS_DEVICE=cpu
EPILENS_OUTPUT_FORMAT=human
```

Other provider credentials are `OPENAI_API_KEY`, `OPENROUTER_API_KEY`, and
`ANTHROPIC_API_KEY`; Ollama uses `OLLAMA_HOST`. The inert, beginner-safe
template is [`.env.example`](https://github.com/VinsRR/EpiLens/blob/main/.env.example).
Docker and external-service settings are kept separately in
[`.env.docker.example`](https://github.com/VinsRR/EpiLens/blob/main/.env.docker.example).

Secrets belong in `.env` or the deployment secret store, never in a workspace
file, notebook, issue, or commit.

## Structured Workflows

The built-in workflow families are:

- `PaperClassifier`, for evidence-backed labels such as paper type,
  geographical coverage, data type, and reported data accessibility.
- `PrecisionMiner`, for variable-length structured extraction such as data
  sources, supplementary material, and key references.

A workflow fixes two task-specific artifacts across every paper: retrieval and
prompt instructions, plus a Pydantic/JSON output schema. Reusing the same
template and validating the same schema makes corpus-level outputs comparable
and machine-readable.

Create a task interactively:

```bash
epilens tasks new --kind classifier --interactive
epilens tasks new --kind miner --interactive
```

Place the resulting JSON file in a workspace's `tasks/` directory for automatic
discovery, pass it directly with `--task-file`, or expose a directory through
`EPILENS_TASKS_DIR`.

## Python API

Start at the public package surface:

```python
from epilens import EpiLensRuntime, PaperClassifier, PrecisionMiner

runtime = EpiLensRuntime()
health = runtime.health()
print(health.checks)
```

For a small, fully local parsing example:

```python
from epilens.rag.ingestion.document_loader import DocumentLoaderFactory

loader = DocumentLoaderFactory.get_loader("pdfminer")
sections, metadata, references = loader.load("paper.pdf")

print(metadata.title)
print(sections[0].content[:500])
```

The ordered notebooks progressively introduce schemas, indexing, retrieval,
PDF ingestion, structured extraction/classification, the runtime facade, and
workspaces. They use helper files and sample PDFs that are intentionally not
bundled in the wheel, so clone the repository before running them:

```bash
git clone https://github.com/VinsRR/EpiLens.git
cd EpiLens
python -m pip install -e ".[dev]"
jupyter lab notebooks/
```

See the
[notebook guide](https://github.com/VinsRR/EpiLens/tree/main/notebooks) for the
recommended order.

## Studio and API

The easiest browser path uses local file-backed workspaces and requires no
Qdrant or MongoDB:

```bash
python -m pip install "epilens[all]"
epilens studio
```

Studio starts FastAPI and Streamlit together on `127.0.0.1`, manages both
processes, and opens the workspace UI. To run the parts separately:

```bash
epilens serve --port 8000
epilens-ui
```

The API includes health, retrieval, classification, precision-mining, and
workspace-scoped routes. Interactive API documentation is available at
`http://127.0.0.1:8000/docs` while the server is running.

For a shared deployment with Qdrant and GROBID, use the repository's
[Docker Compose configuration](https://github.com/VinsRR/EpiLens/blob/main/docker-compose.yml):

```bash
cp .env.docker.example .env
# Add a provider key to .env if generated answers are required.
docker compose up --build
```

MongoDB remains external and is only needed for the unscoped
corpus-backed routes; the workspace-scoped Studio path is file-backed.

## Development

```bash
git clone https://github.com/VinsRR/EpiLens.git
cd EpiLens
python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e ".[dev,all,qdrant,mongo,faiss,providers]"
pytest -q
ruff check src/epilens tests
python scripts/clean_release_artifacts.py
python -m build
python -m twine check dist/*
```

CI tests the base package across Python 3.10-3.14 on Linux and representative
macOS and Windows versions, exercises optional API/UI/store/provider paths,
validates both distribution artifacts, and installs the wheel in an isolated
environment outside the source tree.

Releases use PyPI Trusted Publishing. Maintainers configure the GitHub `pypi`
environment once, update the version and changelog, then push a matching tag
such as `v0.1.0`; the release workflow rejects mismatched tags before upload.

## Citation

EpiLens was developed by Vincenzo Perri at ISI Foundation. Software citation
metadata is in
[`CITATION.cff`](https://github.com/VinsRR/EpiLens/blob/main/CITATION.cff).
The accompanying manuscript is still a draft, so its final journal reference
and DOI can be added when available.

## License

EpiLens is distributed under the
[GNU Affero General Public License v3.0](https://github.com/VinsRR/EpiLens/blob/main/LICENSE.txt)
(`AGPL-3.0-only`).
