Metadata-Version: 2.4
Name: generesearcher
Version: 0.1.5
Summary: Gene-centered biomedical literature retrieval and analysis pipeline
License-Expression: BSD-2-Clause
Requires-Python: <3.13,>=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: biopython<2,>=1.83
Requires-Dist: click<9,>=8
Requires-Dist: numpy<2,>=1.26
Requires-Dist: openai<3,>=1.30
Requires-Dist: pandas<4,>=2.2
Requires-Dist: pymilvus[milvus_lite]==3.0.0
Requires-Dist: milvus-lite==2.5.1
Requires-Dist: setuptools<81,>=68
Requires-Dist: python-dotenv<2,>=1.0
Requires-Dist: PyYAML<7,>=6
Requires-Dist: requests<3,>=2.31
Requires-Dist: scispacy<0.7,>=0.6
Requires-Dist: spacy<3.8,>=3.7
Requires-Dist: tqdm<5,>=4.66
Provides-Extra: dashscope
Requires-Dist: dashscope<2,>=1.20; extra == "dashscope"
Provides-Extra: local-embedding
Requires-Dist: torch>=2.2; extra == "local-embedding"
Requires-Dist: transformers<5,>=4.40; extra == "local-embedding"
Provides-Extra: dev
Requires-Dist: ruff<1,>=0.6; extra == "dev"
Dynamic: license-file

# GeneResearcher

GeneResearcher searches biomedical literature for gene/function/background relationships, retrieves article text, filters evidence, synthesizes results, and writes a Markdown report.

## Choose how to use it

| Use case | Install | Run |
| --- | --- | --- |
| Use a built package | Install the wheel, then run `python -m generesearcher install-spacy-model`. | Import `ResearchConfig` and `run_analysis` in your Python code. |
| Run or modify the source | From the project source directory, run `python -m pip install -e .`, then `python -m generesearcher install-spacy-model`. | Create a study YAML file and run `python -m generesearcher run --config study.yaml --save-to ./results/study`. |

Both routes use the same `ResearchConfig` and pipeline. Users provide their own `.env` credentials. The SciSpaCy language model is a separate download, installed with `install-spacy-model` after the package dependencies are installed. Install the published package from PyPI with `python -m pip install generesearcher`; add the `dashscope` extra when using DashScope embeddings.

## How a study is organized

- The normal Python API does not require a project folder. Each run uses an internal temporary workspace for Milvus Lite and intermediate files; the workspace is removed when its `AnalysisResult` is cleaned up or released.
- If you want to keep the analysis, pass a `project_folder` or call `save_analysis()`. One saved project folder represents one research background and can contain multiple function analyses, each under its own `analysis_<function>` directory.
- Re-running the same function in a chosen project folder clears that function's Milvus database and generated exports before starting. The Python API emits a warning if a database already exists; the pipeline also logs the directory it clears.
- CSV and TXT files are display exports. Pipeline recovery and later steps use the database as their source of truth.

The analysis results are returned to Python in memory. Milvus Lite still needs a local database file while the pipeline is running, so GeneResearcher keeps that implementation detail in the temporary workspace unless you choose a persistent project folder.

The six pipeline stages run in this order:

1. Retrieve articles and their available text.
2. Embed article text and recall relevant sentences.
3. Filter recalled evidence for relevance.
4. Synthesize per-article and cross-article results.
5. Build gene-level and study-level reporting results.
6. Generate the final Markdown report.

During Step 1, GeneResearcher tries PMC full-text sources in this order: BioC, PMC OAI-PMH, then Entrez JATS. BioC requests are serialized and make at most three attempts with exponential backoff; server-provided `Retry-After` and browser refresh delays are honored. If BioC returns `no healthy upstream`, GeneResearcher skips the remaining BioC retries and immediately tries OAI-PMH. Other transient BioC failures fall back to OAI-PMH after the retry attempts are exhausted. OAI-PMH requests are also serialized and limited to fewer than three requests per second, as required by PMC. OAI-PMH provides JATS full text only for articles whose reuse rights allow it; when it returns no full text, GeneResearcher continues to Entrez JATS. PMC asks scripts making more than 100 OAI-PMH requests to run outside weekday peak hours (5:00 a.m.–9:00 p.m. U.S. Eastern time).

## Install and run from source

Use Python 3.10–3.12 in an isolated environment. SciSpaCy 0.6.x requires NumPy 1.x, while many existing data-science environments now use NumPy 2.x. Installing GeneResearcher into a shared environment can therefore downgrade NumPy and conflict with its other packages. A fresh virtual environment keeps those dependencies separate. SciSpaCy 0.6.x does not support Python 3.13. The clean installation has been verified on macOS arm64 with Python 3.11. From this directory:

```bash
python -m venv .venv
# macOS/Linux
source .venv/bin/activate
# Windows PowerShell: .venv\Scripts\Activate.ps1
python -m pip install -e .
python -m generesearcher install-spacy-model
cp .env.example .env
```

In Jupyter or Colab, installing packages into a running kernel can leave an older NumPy binary loaded in memory even after pip replaces the files. Restart the kernel after such an installation. The supplied Colab notebook uses a separate virtual environment and subprocess so it does not alter the notebook kernel's packages.

Edit `.env` with the required service values. Do not commit it. The default `.env` is read from the current working directory; `--env-file` can point to another file.

Step 2 uses a SciSpaCy model to split article text into sentences. The default is `en_core_sci_lg`; you may instead select `en_core_sci_sm`, `en_core_sci_md`, or `en_core_sci_scibert`. The four choices are SciSpaCy's general biomedical pipelines; GeneResearcher does not use the separate NER-only models. After installing GeneResearcher, download and validate the default model with:

```bash
python -m generesearcher install-spacy-model
```

To use another model, install it and set the same name in `study.yaml` under `scispacy_model`:

```bash
python -m generesearcher install-spacy-model --model en_core_sci_sm
```

```yaml
scispacy_model: en_core_sci_sm
```

The installer downloads the selected model archive from the [official SciSpaCy model list](https://github.com/allenai/scispacy/blob/main/docs/index.md). It does not retain the archive in pip's cache. The default `en_core_sci_lg` model uses about 620 MB after installation; choose `en_core_sci_sm` if you need a smaller model. GeneResearcher checks required dependencies and loads the selected model before it starts a run or clears an existing analysis database; a missing or incompatible model produces an actionable error. Model names are also saved in `params.json` with the analysis parameters.

The default install supports the configured Gitee-compatible embedding endpoint. DashScope and local Transformers are optional extras. Install the matching extra with the package:

```bash
# Source checkout
python -m pip install -e '.[dashscope]'

# Or a built wheel
python -m pip install 'path/to/generesearcher-0.1.5-py3-none-any.whl[dashscope]'
```

For a local Transformers embedding model, use `.[local-embedding]` instead and set `EMBEDDING_PROVIDER=local` plus `EMBEDDING_MODEL_NAME` in `.env`. PyTorch can be a large download, so install that extra only when using local embeddings.

Copy and edit the example study file, then run:

```bash
cp study.example.yaml study.yaml
python -m generesearcher run --config study.yaml --save-to ./results/example_study
```

Omit `--save-to` to run in temporary storage. The command-line process removes temporary results when it exits; use `--save-to` or a `project_folder` in the YAML file when you want to keep the database and exported files.

You can specify an environment file and see detailed logs with:

```bash
python -m generesearcher run --config study.yaml --env-file .env --log-level DEBUG
python -m generesearcher run --config study.yaml --quiet
```

The log levels follow one rule across the package:

- `DEBUG`: cache decisions, retry attempts, and low-level diagnostics.
- `INFO`: stage starts and completions, durations, paths, and record counts.
- `WARNING`: recoverable omissions, fallbacks, and partial results.
- `ERROR`: failures that stop a stage or the full run.

The default is `INFO`. Use `--log-level DEBUG` (or `--verbose`) for detail, `--log-level WARNING` to reduce routine progress messages, `--quiet` to show errors only, or `--log-level OFF` to disable logs. Fatal run failures include a traceback at the Python API boundary. Progress bars appear at `INFO` or `DEBUG` in interactive terminals and notebook kernels. The supplied Colab notebook streams output so progress bars update in place. Application logs do not include prompts or article text. The Python warning about clearing an existing analysis database remains visible even when logs are disabled.

Relative `project_folder` paths in YAML and relative `--save-to` paths are resolved from the YAML file's directory. The CLI looks for `.env` beside the YAML file, then falls back to `.env` in the current working directory.

## Use as an installed Python package

To create an installable wheel from this checkout:

```bash
python -m pip install build
python -m build
python -m pip install dist/generesearcher-0.1.5-py3-none-any.whl
python -m generesearcher install-spacy-model
```

Users can install the current release directly from PyPI:

Create and activate a fresh environment using the commands above before running these install commands; this keeps GeneResearcher's NumPy/SciSpaCy stack separate from other projects.

```bash
python -m pip install 'generesearcher==0.1.5'
# For DashScope embeddings:
python -m pip install 'generesearcher[dashscope]==0.1.5'
```

People working from the source checkout can use `pip install .` or `pip install -e .`.

Synchronous Python:

```python
import logging
import os
from generesearcher import ResearchConfig, run_analysis

# Python API users control package log output with Python's logging settings.
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(name)s: %(message)s")
# To suppress GeneResearcher logs in this process, use:
# logging.getLogger("generesearcher").setLevel(logging.CRITICAL + 1)

# Service credentials can be set in this process. Values already present in
# the process environment take precedence over values loaded from .env.
os.environ["ENTREZ_EMAIL"] = "your-email@example.org"
os.environ["EMBEDDING_PROVIDER"] = "dashscope"
os.environ["EMBEDDING_API_KEY"] = "your-embedding-api-key"
os.environ["LLM_MODEL_NAME"] = "your-model-name"
os.environ["LLM_API_KEY"] = "your-llm-api-key"
os.environ["LLM_BASE_URL"] = "https://your-llm-endpoint/v1"

config = ResearchConfig(
    genes=["EP300", "CREBBP"],
    function_name="histone acetylation",
    research_background_template=(
        "What is the role of {gene} in {function_name}, especially in cancer development?"
    ),
    keywords=["cancer", "tumor"],
    gene_aliases={"EP300": ["p300"], "CREBBP": ["CBP"]},
    scispacy_model="en_core_sci_lg",
)
result = run_analysis(config)
print(result.report_path)
# Optional: persist the database and exports under a chosen project folder.
# config.project_folder = "./results/example_study"  # set before run_analysis()
# Or save this in-memory result afterwards:
# from generesearcher import save_analysis
# saved_directory = save_analysis(result, "./results/example_study")
```

The values above are examples; do not commit real credentials in a Python script. You can instead set the variables in the shell, a deployment secret store, or a local `.env` file. GeneResearcher reads process environment variables directly and does not overwrite them when it loads `.env`. The `.env` file is searched in the current working directory unless you pass `env_file` explicitly.

For an already-running async application, use `await run_analysis_async(config, env_file=".env")`.

### Save and restore an analysis

The usual Python workflow needs no project folder: `run_analysis()` returns an `AnalysisResult` and keeps Milvus Lite's working database in a temporary directory. If you want to keep the database and exports, either provide `project_folder` in `ResearchConfig` before the run or call `save_analysis()` afterwards:

```python
from generesearcher import load_analysis, save_analysis

# Save the completed function analysis under a project folder.
# The same folder can hold other function analyses for this research background.
saved_directory = save_analysis(result, "./results/example_study")

# Restore the parameters and result tables into memory later.
restored_result = load_analysis(saved_directory)
print(restored_result.analysis_data.raw_papers_df.head())
print(restored_result.report_path)
```

`save_analysis()` copies the function's complete analysis directory, including its Milvus Lite database, parameters, and generated files. It does not serialize through CSV. The destination is the project root; it may contain other functions from the same research background, but the same function directory must not already exist. The helper returns the saved `analysis_<function>` directory. `load_analysis()` restores from the database and does not call literature, embedding, or language-model services. It returns the saved result for inspection and downstream Python use; it does not rerun the pipeline. Call `result.cleanup()` when you no longer need an unsaved result; it removes that result's temporary files. Results loaded from a saved directory are not temporary. In the CLI, use `--save-to PATH` to keep outputs; a CLI run without a save path removes its temporary files when the command exits.

`ResearchConfig`, `params.json`, and Milvus use the same canonical field names. PMID is the exclusion identifier; PMCID is used only as the retrieval/full-text identifier and as the rare fallback when an article has no PMID.

## Configuration fields

The YAML file and `ResearchConfig` accept the same fields:

| Field | Required | Meaning |
| --- | --- | --- |
| `genes` | Yes | One or more gene symbols |
| `function_name` | Yes | The function or pathway analyzed in this run |
| `research_background_template` | Yes | Background text; `{gene}` and `{function_name}` are replaced during retrieval and analysis |
| `project_folder` | No | Optional persistent root shared by runs for the same research background; omitted runs use a temporary workspace |
| `keywords` | No | Literature search terms |
| `function_query` | No | Optional search wording for the function |
| `exclude_pmids` | No | PMID values to exclude |
| `publication_date_before` | No | Inclusive search cutoff in the format supported by the search code |
| `gene_aliases` | No | Additional search aliases keyed by gene symbol |
| `max_papers_per_gene` | No | Maximum search results per gene; defaults to 10 |
| `scispacy_model` | No | SciSpaCy sentence-processing model; defaults to `en_core_sci_lg` |

Credentials and provider settings belong in environment variables or `.env`, not in the study YAML. Existing process environment variables take precedence over `.env` values. Use `EMBEDDING_API_KEY` for the embedding provider key, including DashScope.

## Naming and persisted data

Public Python entry points, configuration fields, persisted JSON keys, and Milvus fields use descriptive `snake_case` names. The project uses the canonical first-version format and does not translate alternate spellings. Research background templates use `{gene}` and `{function_name}` placeholders.

Only the `generesearcher` package is included in built distributions. Notebook checkpoints, bytecode caches, and historical duplicate scripts at the repository root are not runtime package modules.

## License

GeneResearcher is distributed under the BSD-2-Clause License. See [LICENSE](LICENSE) for the full terms.
