Metadata-Version: 2.4
Name: generesearcher
Version: 0.1.0
Summary: Gene-centered biomedical literature retrieval and analysis pipeline
License-Expression: BSD-2-Clause
Requires-Python: <3.13,>=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: biopython<2,>=1.83
Requires-Dist: click<9,>=8
Requires-Dist: numpy<3,>=1.26
Requires-Dist: openai<3,>=1.30
Requires-Dist: pandas<4,>=2.2
Requires-Dist: pymilvus[milvus_lite]==3.0.0
Requires-Dist: milvus-lite==2.5.1
Requires-Dist: setuptools<81,>=68
Requires-Dist: python-dotenv<2,>=1.0
Requires-Dist: PyYAML<7,>=6
Requires-Dist: requests<3,>=2.31
Requires-Dist: scispacy<0.7,>=0.6
Requires-Dist: spacy<3.8,>=3.7
Requires-Dist: tqdm<5,>=4.66
Provides-Extra: dashscope
Requires-Dist: dashscope<2,>=1.20; extra == "dashscope"
Provides-Extra: local-embedding
Requires-Dist: torch>=2.2; extra == "local-embedding"
Requires-Dist: transformers<5,>=4.40; extra == "local-embedding"
Provides-Extra: dev
Requires-Dist: ruff<1,>=0.6; extra == "dev"
Dynamic: license-file

# GeneResearcher

GeneResearcher searches biomedical literature for gene/function/background relationships, retrieves article text, filters evidence, synthesizes results, and writes a Markdown report.

## Choose how to use it

| Use case | Install | Run |
| --- | --- | --- |
| Use a built package | Install the wheel, then run `python -m generesearcher install-spacy-model`. | Import `ResearchConfig` and `run_analysis` in your Python code. |
| Run or modify the source | From the project source directory, run `python -m pip install -e .`, then `python -m generesearcher install-spacy-model`. | Create a study YAML file and run `python -m generesearcher run --config study.yaml`. |

Both routes use the same `ResearchConfig` and pipeline. Users provide their own `.env` credentials. The SciSpaCy language model is a separate download, installed with `install-spacy-model` after the package dependencies are installed. The package has build metadata and a working wheel; PyPI installation is available after the project publishes a release.

## How a study is organized

- One `project_folder` represents one research background.
- A project can contain multiple function analyses. Each run analyzes one `function_name` for one or more `genes` and writes to its own `analysis_<function>` directory.
- Re-running the same function clears that function directory's Milvus database and generated exports before starting. The Python API emits a warning if a database already exists; the pipeline also logs the directory it clears.
- CSV and TXT files are display exports. Pipeline recovery and later steps use the database as their source of truth.

The six pipeline stages run in this order:

1. Retrieve articles and their available text.
2. Embed article text and recall relevant sentences.
3. Filter recalled evidence for relevance.
4. Synthesize per-article and cross-article results.
5. Build gene-level and study-level reporting results.
6. Generate the final Markdown report.

## Install and run from source

Use Python 3.10–3.12 in an isolated environment. SciSpaCy 0.6.x does not support Python 3.13. The clean installation has been verified on macOS arm64 with Python 3.11. From this directory:

```bash
python -m pip install -e .
python -m generesearcher install-spacy-model
cp .env.example .env
```

Edit `.env` with the required service values. Do not commit it. The default `.env` is read from the current working directory; `--env-file` can point to another file.

Step 2 uses a SciSpaCy model to split article text into sentences. The default is `en_core_sci_lg`; you may instead select `en_core_sci_sm`, `en_core_sci_md`, or `en_core_sci_scibert`. The four choices are SciSpaCy's general biomedical pipelines; GeneResearcher does not use the separate NER-only models. After installing GeneResearcher, download and validate the default model with:

```bash
python -m generesearcher install-spacy-model
```

To use another model, install it and set the same name in `study.yaml` under `scispacy_model`:

```bash
python -m generesearcher install-spacy-model --model en_core_sci_sm
```

```yaml
scispacy_model: en_core_sci_sm
```

The installer downloads the selected model archive from the [official SciSpaCy model list](https://github.com/allenai/scispacy/blob/main/docs/index.md). It does not retain the archive in pip's cache. The default `en_core_sci_lg` model uses about 620 MB after installation; choose `en_core_sci_sm` if you need a smaller model. GeneResearcher checks required dependencies and loads the selected model before it starts a run or clears an existing analysis database; a missing or incompatible model produces an actionable error. Model names are also saved in `params.json` with the analysis parameters.

The default install supports the configured Gitee-compatible embedding endpoint. DashScope and local Transformers are optional extras. Install the matching extra with the package:

```bash
# Source checkout
python -m pip install -e '.[dashscope]'

# Or a built wheel
python -m pip install 'path/to/generesearcher-0.1.0-py3-none-any.whl[dashscope]'
```

For a local Transformers embedding model, use `.[local-embedding]` instead and set `EMBEDDING_PROVIDER=local` plus `EMBEDDING_MODEL_NAME` in `.env`. PyTorch can be a large download, so install that extra only when using local embeddings.

Copy and edit the example study file, then run:

```bash
cp study.example.yaml study.yaml
python -m generesearcher run --config study.yaml
```

You can specify an environment file and see detailed logs with:

```bash
python -m generesearcher run --config study.yaml --env-file .env --log-level DEBUG
python -m generesearcher run --config study.yaml --quiet
```

The log levels follow one rule across the package:

- `DEBUG`: cache decisions, retry attempts, and low-level diagnostics.
- `INFO`: stage starts and completions, durations, paths, and record counts.
- `WARNING`: recoverable omissions, fallbacks, and partial results.
- `ERROR`: failures that stop a stage or the full run.

The default is `INFO`. Use `--log-level DEBUG` (or `--verbose`) for detail, `--log-level WARNING` to reduce routine progress messages, `--quiet` to show errors only, or `--log-level OFF` to disable logs. Fatal run failures include a traceback at the Python API boundary. Progress bars are disabled when the level is above `INFO`. Application logs do not include prompts or article text. The Python warning about clearing an existing analysis database remains visible even when logs are disabled.

Relative `project_folder` paths in YAML are resolved from the YAML file's directory. The CLI looks for `.env` beside the YAML file, then falls back to `.env` in the current working directory.

## Use as an installed Python package

To create an installable wheel from this checkout:

```bash
python -m pip install build
python -m build
python -m pip install dist/generesearcher-0.1.0-py3-none-any.whl
python -m generesearcher install-spacy-model
```

PyPI publishing is a separate maintainer release step. Until then, users can install a supplied wheel directly as shown above; people working from the source checkout can use `pip install .` or `pip install -e .`.

Synchronous Python:

```python
import logging
import os
from generesearcher import ResearchConfig, run_analysis

# Python API users control package log output with Python's logging settings.
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(name)s: %(message)s")
# To suppress GeneResearcher logs in this process, use:
# logging.getLogger("generesearcher").setLevel(logging.CRITICAL + 1)

# Service credentials can be set in this process. Values already present in
# the process environment take precedence over values loaded from .env.
os.environ["ENTREZ_EMAIL"] = "your-email@example.org"
os.environ["EMBEDDING_PROVIDER"] = "dashscope"
os.environ["EMBEDDING_API_KEY"] = "your-embedding-api-key"
os.environ["LLM_MODEL_NAME"] = "your-model-name"
os.environ["LLM_API_KEY"] = "your-llm-api-key"
os.environ["LLM_BASE_URL"] = "https://your-llm-endpoint/v1"

config = ResearchConfig(
    genes=["EP300", "CREBBP"],
    function_name="histone acetylation",
    research_background_template=(
        "What is the role of {gene} in {function_name}, especially in cancer development?"
    ),
    project_folder="./results/example_study",
    keywords=["cancer", "tumor"],
    gene_aliases={"EP300": ["p300"], "CREBBP": ["CBP"]},
    scispacy_model="en_core_sci_lg",
)
result = run_analysis(config)
print(result.report_path)
```

The values above are examples; do not commit real credentials in a Python script. You can instead set the variables in the shell, a deployment secret store, or a local `.env` file. GeneResearcher reads process environment variables directly and does not overwrite them when it loads `.env`. The `.env` file is searched in the current working directory unless you pass `env_file` explicitly.

For an already-running async application, use `await run_analysis_async(config, env_file=".env")`.

### Save and restore an analysis

The usual workflow needs no explicit save or load call. `run_analysis()` returns an `AnalysisResult` in memory. If you want to keep a separate copy or restore a previous result later, use the optional helpers:

```python
from generesearcher import load_analysis, save_analysis

# Save the completed function analysis to a new directory.
saved_directory = save_analysis(result, "./backups/ep300_histone_acetylation")

# Restore the parameters and result tables into memory later.
restored_result = load_analysis(saved_directory)
print(restored_result.analysis_data.raw_papers_df.head())
print(restored_result.report_path)
```

`save_analysis()` copies the function's complete analysis directory, including its Milvus Lite database, parameters, and generated files. It does not serialize through CSV. The destination directory must not already exist. `load_analysis()` restores from the database and does not call literature, embedding, or language-model services. It returns the saved result for inspection and downstream Python use; it does not rerun the pipeline. Keep the saved directory separate from the active `project_folder`, because a new full run clears the active function analysis before starting.

`ResearchConfig`, `params.json`, and Milvus use the same canonical field names. PMID is the exclusion identifier; PMCID is used only as the retrieval/full-text identifier and as the rare fallback when an article has no PMID.

## Configuration fields

The YAML file and `ResearchConfig` accept the same fields:

| Field | Required | Meaning |
| --- | --- | --- |
| `genes` | Yes | One or more gene symbols |
| `function_name` | Yes | The function or pathway analyzed in this run |
| `research_background_template` | Yes | Background text; `{gene}` and `{function_name}` are replaced during retrieval and analysis |
| `project_folder` | No | Root folder shared by runs for the same research background |
| `keywords` | No | Literature search terms |
| `function_query` | No | Optional search wording for the function |
| `exclude_pmids` | No | PMID values to exclude |
| `publication_date_before` | No | Inclusive search cutoff in the format supported by the search code |
| `gene_aliases` | No | Additional search aliases keyed by gene symbol |
| `max_papers_per_gene` | No | Maximum search results per gene; defaults to 10 |
| `scispacy_model` | No | SciSpaCy sentence-processing model; defaults to `en_core_sci_lg` |

Credentials and provider settings belong in environment variables or `.env`, not in the study YAML. Existing process environment variables take precedence over `.env` values. Use `EMBEDDING_API_KEY` for the embedding provider key, including DashScope.

## Naming and persisted data

Public Python entry points, configuration fields, persisted JSON keys, and Milvus fields use descriptive `snake_case` names. The project uses the canonical first-version format and does not translate alternate spellings. Research background templates use `{gene}` and `{function_name}` placeholders.

Only the `generesearcher` package is included in built distributions. Notebook checkpoints, bytecode caches, and historical duplicate scripts at the repository root are not runtime package modules.

## License

GeneResearcher is distributed under the BSD-2-Clause License. See [LICENSE](LICENSE) for the full terms.
