Metadata-Version: 2.4
Name: refget
Version: 0.12.0
Summary: GA4GH refget - reference sequence and sequence collection tools
Project-URL: Homepage, https://github.com/refgenie/refget
Author: Michal Stolarczyk
Author-email: Nathan Sheffield <nathan@code.databio.org>
License-Expression: BSD-2-Clause
License-File: LICENSE.txt
Keywords: assembly,bioinformatics,genome,reference,sequence
Classifier: Development Status :: 4 - Beta
Classifier: License :: OSI Approved :: BSD License
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Python: >=3.10
Requires-Dist: gtars>=0.9.0
Requires-Dist: jsonschema
Requires-Dist: pyyaml
Requires-Dist: requests
Requires-Dist: tomli-w
Requires-Dist: typer>=0.9.0
Provides-Extra: db
Requires-Dist: psycopg2-binary; extra == 'db'
Requires-Dist: sqlmodel; extra == 'db'
Provides-Extra: seqcolapi
Requires-Dist: fastapi; extra == 'seqcolapi'
Requires-Dist: uvicorn>=0.30.0; extra == 'seqcolapi'
Provides-Extra: seqcolapi-db
Requires-Dist: fastapi; extra == 'seqcolapi-db'
Requires-Dist: psycopg2-binary; extra == 'seqcolapi-db'
Requires-Dist: sqlmodel; extra == 'seqcolapi-db'
Requires-Dist: uvicorn>=0.30.0; extra == 'seqcolapi-db'
Provides-Extra: test
Requires-Dist: fastapi; extra == 'test'
Requires-Dist: httpx; extra == 'test'
Requires-Dist: pytest; extra == 'test'
Requires-Dist: pytest-cov>=6.0.0; extra == 'test'
Requires-Dist: sqlmodel; extra == 'test'
Description-Content-Type: text/markdown

# Refget

![Run pytests](https://github.com/pepkit/looper/workflows/Run%20pytests/badge.svg)

User-facing documentation is hosted at [refgenie.org/refget](https://refgenie.org/refget/).

This repository includes:

1. `/refget`: The `refget` Python package, which provides a Python interface to both remote and local use of refget standards. It has clients and functions for both refget sequences and refget sequence collections (seqcol).
2. `/refget/seqcolapi`: Sequence collections API software, a FastAPI wrapper built on top of the `refget` package. It provides a bare-bones Sequence Collections API service. It ships in the `refget` wheel, but its dependencies do not — see [Installation](#installation). Nothing on the plain `import refget` path imports this subpackage, so a base install never pays for fastapi/uvicorn/sqlmodel/psycopg2.
3. `/seqcolapi`: A thin compatibility shim re-exporting `refget.seqcolapi`, kept so existing deployments that run `uvicorn seqcolapi.main:app` (or `:store_app`) keep working. **This shim is deliberately excluded from the wheel** — it is importable only from a checkout of this repository (or from a deployment that `COPY`s the directory), never from `pip install refget`. Every instruction below therefore uses `refget.seqcolapi.main`, which works everywhere.
4. `/deployment`: Server configurations for demo instances and public deployed instances. There are also github workflows (in `.github/workflows`) that deploy the demo server instance from this repository.
5. `/test_fasta` and `/test_api`: Dummy data and a compliance test, to test external implementations of the Refget Sequence Collections API.
6. `/frontend`: a React seqcolapi front-end.


## Installation

The base install is deliberately light — a client library and CLI, no web
server and no ORM. Everything heavier is an extra, and the extras compose:

| Install | Adds | Use it for |
| --- | --- | --- |
| `pip install refget` | — | The Python library and the `refget` CLI: digests, `RefgetStore`, API clients, compliance. No fastapi, no sqlalchemy. |
| `pip install 'refget[db]'` | sqlmodel, psycopg2-binary | The SQLModel layer: `refget.models`, `refget.agents.RefgetDBAgent`, `refget admin`. A library capability — you can want the ORM without wanting a server. |
| `pip install 'refget[seqcolapi]'` | fastapi, uvicorn | **Serving a RefgetStore.** `uvicorn refget.seqcolapi.main:store_app`, `refget store serve`, `refget.seqcolapi.create_seqcol_app`. No database of any kind. |
| `pip install 'refget[seqcolapi-db]'` | both of the above | **The PostgreSQL-backed service** (`uvicorn refget.seqcolapi.main:app`), i.e. what runs seqcolapi.databio.org. |

The importable module path is `refget.seqcolapi.main`, not `seqcolapi.main`. The
bare `seqcolapi` package in this repository is a compatibility shim that is
**not** shipped in the wheel; `uvicorn seqcolapi.main:store_app` resolves only
from a checkout of this repository, and raises
`ModuleNotFoundError: No module named 'seqcolapi'` in a pip-installed
environment.

The two service extras correspond to the two deployment modes described under
[Development and deployment: Backend](#development-and-deployment-backend). The
store-backed mode is the common case and the cheaper one; it needs no database
dependencies at all.

These boundaries are enforced by module structure, not convention: the
database code lives in `refget/models.py`, `refget/agents.py` and
`refget/seqcolapi/dbapp.py`, and nothing else imports them at module level. The
router's response bodies live in `refget/response_models.py` (plain pydantic)
precisely so that serving the API does not require an ORM. Importing a module
without its extra raises an error naming the extra to install, rather than a
bare `ModuleNotFoundError`. `tests/local/test_import_gating.py` is the tripwire.


## Deploy to AWS ECS

To deploy the public demo instance, you can either:

1. **Create a GitHub release** - This triggers the `deploy_release_software.yml` workflow, which builds and pushes the Docker image to DockerHub. After that completes, it automatically triggers `deploy_primary.yml` to deploy to AWS ECS.

2. **Manual dispatch** - You can manually trigger either workflow from the GitHub Actions tab.

This builds seqcolapi, pushes to DockerHub, and deploys to ECS.

## Testing

### Unit tests

```bash
pytest
```

### Integration tests (requires Docker)

Integration tests run against an ephemeral PostgreSQL database in Docker:

```bash
./scripts/test-integration.sh
```

This starts the test database, runs tests, and cleans up automatically.

## Development and deployment: Backend

### Store-backed (no database)

The store-backed seqcolapi uses a RefgetStore (local files) instead of PostgreSQL. This is the simplest way to run the API, and it needs only `pip install 'refget[seqcolapi]'` — fastapi and uvicorn, no sqlmodel, no sqlalchemy, no psycopg2.

For safe concurrent serving, the store is fully loaded and converted to a read-only store (`RefgetStore.into_readonly()`) before serving, so HTTP reads borrow immutably across request threads. The `refget store serve` CLI does this by default; pass `--lazy` to serve directly from the mutable, lazy-loading store instead (single-reader-oriented, not recommended for concurrent production serving).

#### Quick start

*Requires a checkout of this repository* (the script and the demo FASTA files
live here):

```console
bash deployment/store_demo_up.sh
```

This will:
- Build a local RefgetStore from test FASTA files
- Run the store-backed seqcolapi with uvicorn
- Block the terminal until you press Ctrl+C, which cleans up

No Docker or database required.

#### Step-by-step

1. Build a store from FASTA files. `data_loaders/` is not part of the wheel, so
   this step needs a checkout of this repository:

```console
python data_loaders/demo_build_store.py test_fasta /tmp/refget_demo_store
```

   From a pip install, build a store with the CLI instead:

```console
refget store init -p /tmp/refget_demo_store
refget store add -p /tmp/refget_demo_store <your.fa>
```

2. Start the store-backed API. This works anywhere `refget[seqcolapi]` is
   installed:

```console
REFGET_STORE_PATH=/tmp/refget_demo_store uvicorn refget.seqcolapi.main:store_app --reload --port 8100
```

#### Remote store

To run against a remote (S3) store:

```console
REFGET_STORE_URL=https://example.com/store uvicorn refget.seqcolapi.main:store_app --port 8100
```

### DB-backed (PostgreSQL)

If you need a database-backed instance (e.g., for mutable data, advanced queries), use the DB-backed workflow. This one needs `pip install 'refget[seqcolapi-db]'`. In a moment I'll show you how to do these steps individually, but if you're in a hurry, the easy way to get a development API running for testing is to just use my very simple shell script like this (no data persistence, just loads demo data):

```console
bash deployment/demo_up.sh
```

This will:
- populate env vars
- launch postgres container with docker
- run the refget service with uvicorn
- load up the demo data
- block the terminal until you press Ctrl+C, which will shut down all services.

### Step-by-step process (DB-backed)

Alternatively, if you want to run each step separately to see what's really going on, start here.

#### Setting up a database connection

First configure a database connection through environment variables. Choose one of these:

```
source deployment/local_demo/local_demo.env # local demo (see below to create the database using docker)
source deployment/seqcolapi.databio.org/production.env # connect to production database
```

If you're using the `local_demo`, then use docker to launch a local postgres database service like this:

```
docker run --rm --name refget-postgres -p 127.0.0.1:5432:5432 \
  -e POSTGRES_PASSWORD \
  -e POSTGRES_USER \
  -e POSTGRES_DB \
  -e POSTGRES_HOST \
  postgres:17.0
```

If you need to load test data into your server, then you have to install [gtars](https://docs.bedbase.org/gtars/) (with `pip install gtars`), a Python package for computing GA4GH digests. You can then load test data like this:

```
PYTHONPATH=. python data_loaders/load_demo_seqcols.py
```

or, with the CLI (`refget add-fasta` was replaced by `refget admin load`):

```
refget admin load --pep test_fasta/test_fasta_metadata.csv --fa-root test_fasta
```

#### Running the seqcolapi API backend

Run the demo `seqcolapi` service like this:

```
uvicorn refget.seqcolapi.main:app --reload --port 8100
```

#### Running with docker

To build the docker file, first build the image from a checkout of this
repository (the build context is the `seqcolapi/` compatibility shim, which
exists only here):

```
docker build -f deployment/dockerhub/Dockerfile -t databio/seqcolapi seqcolapi
```

To run in container:

```
source deployment/seqcolapi.databio.org/production.env
docker run --rm -p 8000:80 --name seqcolapi \
  --env "POSTGRES_USER" \
  --env "POSTGRES_DB" \
  --env "POSTGRES_PASSWORD" \
  --env "POSTGRES_HOST" \
  databio/seqcolapi
```

#### Deploying container to dockerhub

Use the github action in this repo which deploys on release, or through manual dispatch.

## Running the frontend

Once you have a backend running, you can run a frontend to interact with it

### Local client with local server

```
cd frontend
npm i
VITE_API_BASE="http://localhost:8100" npm run dev
```

### Local client with production server

```
cd frontend
npm i
VITE_API_BASE="https://seqcolapi.databio.org" npm run dev
```

### Development with local WASM

The `/digest` feature uses [@databio/gtars](https://www.npmjs.com/package/@databio/gtars) for WASM-based FASTA processing. To use a local gtars-wasm build instead of the npm package:

```
LOCAL_GTARS=../../gtars/gtars-wasm/pkg npm run dev
```

The `LOCAL_GTARS` env var should point to the `pkg/` directory of a built gtars-wasm package (run `wasm-pack build --target web` in gtars-wasm to build it).

### gtars WASM API Reference

The streaming API handles files of any size:

```javascript
import * as gtars from '@databio/gtars';
await gtars.default();  // Initialize WASM

// Streaming API (for large files)
const handle = gtars.fastaHasherNew();
gtars.fastaHasherUpdate(handle, chunk);  // Feed Uint8Array chunks
const result = gtars.fastaHasherFinish(handle);  // Get SeqColResult

// Batch API (for small files)
const result = gtars.digestSeqcol(fastaBytes);
```

Result object:
```typescript
interface SeqColResult {
  digest: string;           // Collection digest (SHA512t24u)
  names_digest: string;
  sequences_digest: string;
  lengths_digest: string;
  n_sequences: number;
  sequences: Array<{
    name: string;
    length: number;
    alphabet: string;       // dna2bit, dna3bit, etc.
    sha512t24u: string;
    md5: string;
    description?: string;
  }>;
}
```

### Deploying

1. Ensure the [refget](https://github.com/refgenie/refget/) package master branch is as you want it.
2. Deploy the updated [secqolapi](https://github.com/refgenie/seqcolapi/) app to dockerhub (using manual dispatch, or deploy on github release).
3. Finally, deploy the instance with manual dispatch using the included GitHub action.

## Developer notes

### Models

The database objects and attributes are represented as SQLModel objects in `refget/models.py` (requires `refget[db]`). To add a new attribute:

1. create a new model. This will create a table for that model, etc.
2. change the function that creates the objects, to populate the new attribute.

HTTP response bodies are *not* defined there. They are plain pydantic models in
`refget/response_models.py`, so that `refget.router` — and therefore the
store-backed service — can be imported without an ORM. Put a new response
schema there unless it genuinely maps to a database table.

## Example of loading reference fasta datasets:

Needs `pip install 'refget[seqcolapi-db]'` and a configured PostgreSQL
connection (see [DB-backed (PostgreSQL)](#db-backed-postgresql)):

```
refget admin load --pep ref_fasta.csv --fa-root $BRICKYARD/datasets_downloaded/pangenome_fasta/reference_fasta
```
