Metadata-Version: 2.4
Name: paperminertoolkit
Version: 1.0.0
Summary: A package for extracting structured information from scientific papers with configurable language and vision models.
Author-email: Oskar Soulas <oskar.soulas@gmail.com>, Louie <louies@hotmail.co.uk>
License-Expression: MIT
Project-URL: Homepage, https://github.com/SMTG-Bham/PaperMinerToolkit
Project-URL: Repository, https://github.com/SMTG-Bham/PaperMinerToolkit
Project-URL: Issues, https://github.com/SMTG-Bham/PaperMinerToolkit/issues
Keywords: research,scientific-papers,materials,text-mining,llm
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Chemistry
Classifier: Topic :: Text Processing
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: click
Requires-Dist: headroom-ai[image,ml]<0.36,>=0.35
Requires-Dist: openai
Requires-Dist: numpy
Requires-Dist: pandas
Requires-Dist: pillow
Requires-Dist: pymupdf
Requires-Dist: pypdf
Requires-Dist: requests
Requires-Dist: regex
Requires-Dist: scipy
Requires-Dist: joblib
Requires-Dist: matplotlib
Requires-Dist: scikit-learn
Requires-Dist: tiktoken>=0.13.0
Requires-Dist: tqdm
Requires-Dist: transformers
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Requires-Dist: pytest-cov; extra == "test"
Requires-Dist: ruff; extra == "test"
Provides-Extra: docs
Requires-Dist: myst-nb<2,>=1.2; extra == "docs"
Requires-Dist: pydata-sphinx-theme<0.21,>=0.16; extra == "docs"
Requires-Dist: sphinx<10,>=8; extra == "docs"
Requires-Dist: sphinx-click<7,>=6; extra == "docs"
Requires-Dist: sphinxcontrib-mermaid<2,>=1; extra == "docs"
Dynamic: license-file

<p align="center">
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="assets/Paper_Miner_Toolkit_banner_dark.svg">
    <img src="assets/Paper_Miner_Toolkit_banner_light.svg" alt="PaperMinerToolkit banner" width="640">
  </picture>
</p>

<p align="center">
  <a href="https://github.com/SMTG-Bham/PaperMinerToolkit/actions/workflows/tests.yml"><img src="https://github.com/SMTG-Bham/PaperMinerToolkit/actions/workflows/tests.yml/badge.svg?branch=main" alt="Tests"></a>
  <a href="https://codecov.io/gh/SMTG-Bham/PaperMinerToolkit"><img src="https://codecov.io/gh/SMTG-Bham/PaperMinerToolkit/branch/main/graph/badge.svg" alt="Coverage"></a>
  <a href="https://pypi.org/project/paperminertoolkit/"><img src="https://img.shields.io/pypi/v/paperminertoolkit?logo=pypi&amp;logoColor=white" alt="PyPI version"></a>
  <a href="https://www.python.org/"><img src="https://img.shields.io/badge/Python-3.11%2B-3776AB?logo=python&amp;logoColor=white" alt="Python 3.11 or newer"></a>
  <a href="https://paperminertoolkit.readthedocs.io/en/latest/"><img src="https://img.shields.io/badge/Docs-Read%20the%20Docs-8CA1AF?logo=readthedocs&amp;logoColor=white" alt="Documentation"></a>
</p>

# PaperMinerToolkit

PaperMinerToolkit builds scientific-paper corpora and extracts structured, recipe-defined data with configurable text and vision models. It searches Elsevier/Scopus, CORE, OpenAlex, PubMed, arXiv, medRxiv, bioRxiv, and chemRxiv; supplements paper metadata from Crossref, OpenAlex, PubMed, arXiv, medRxiv, bioRxiv, and chemRxiv, and imports an author's works from Crossref; downloads abstracts, full text, and PDFs; supports persistent regex and LDA topic filters; and stores source content and pipeline state in SQLite.

The complete user guide, CLI reference, Python API, HPC instructions, and rendered notebooks live in the [documentation source](https://paperminertoolkit.readthedocs.io/en/latest/). The repository is ready for Read the Docs; a hosted link will be added after the project is imported.

## Installation

PaperMinerToolkit requires Python 3.11 or newer:

```bash
git clone https://github.com/SMTG-Bham/PaperMinerToolkit.git
cd PaperMinerToolkit
python -m pip install -e .
```

## Quickstart

Configure a text model and any search/download credentials you need, then run a small workflow:

```bash
pmt config model text --provider openai --model YOUR_TEXT_MODEL

pmt search "lithium solid electrolyte" papers.db \
  --source openalex --count 25
pmt enrich papers.db
pmt download papers.db --format abstract

pmt scrape papers.db sse \
  --mode abstract \
  --output temp_scraped_materials.csv

pmt store papers.db \
  temp_scraped_materials.csv \
  materials.csv \
  sse \
  --assume-yes
```

Use `pmt corpus stats papers.db` to inspect stored content and `pmt status papers.db` to inspect pipeline progress.

## Documentation

Install and build the Sphinx site locally:

```bash
python -m pip install -e '.[docs]'
make -C docs html
```

Open `docs/_build/html/index.html`. Notebook templates for OpenAI, Anthropic, local Qwen/vLLM, LDA model selection, temporal trends, and hybrid filtering are under `docs/examples/`.

## Testing

```bash
python -m pip install -e '.[test]'
ruff check paperminertoolkit tests
pytest
```

PaperMinerToolkit is currently alpha software. Keep the corpus database, recipe, model configuration, intermediate extraction CSV, and final results together so a workflow can be reviewed and reproduced.
