Metadata-Version: 2.4
Name: scrapyrus
Version: 0.0.1
Summary: Add short description here
Maintainer-email: Dominic Kempf <your@email.com>
License-Expression: MIT
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Requires-Python: >=3.13
Description-Content-Type: text/markdown
License-File: LICENSE.md
Requires-Dist: beautifulsoup4
Requires-Dist: cltk[stanza]>=2.5.1
Requires-Dist: click
Requires-Dist: saxonche
Requires-Dist: pydantic>=2
Requires-Dist: psycopg[binary]>=3
Requires-Dist: requests
Requires-Dist: termgraph>=0.7.6
Requires-Dist: tqdm
Provides-Extra: tests
Requires-Dist: pytest; extra == "tests"
Requires-Dist: pytest-cov; extra == "tests"
Provides-Extra: docs
Requires-Dist: sphinx; extra == "docs"
Requires-Dist: sphinx_mdinclude; extra == "docs"
Requires-Dist: sphinx_rtd_theme; extra == "docs"
Dynamic: license-file

# Welcome to scrapyrus

[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![GitHub Workflow Status](https://img.shields.io/github/actions/workflow/status/ssciwr/scrapyrus/ci.yml?branch=main)](https://github.com/ssciwr/scrapyrus/actions/workflows/ci.yml)
[![codecov](https://codecov.io/gh/ssciwr/scrapyrus/branch/main/graph/badge.svg)](https://codecov.io/gh/ssciwr/scrapyrus)

## Installation

The Python package `scrapyrus` can be installed from PyPI:

```
python -m pip install scrapyrus
```

## Development installation

If you want to contribute to the development of `scrapyrus`, we recommend
the following editable installation from this repository:

```
git clone git@github.com:ssciwr/scrapyrus.git
cd scrapyrus
python -m pip install --editable .[tests]
```

Having done so, the test suite can be run using `pytest`:

```
python -m pytest
```

## Docker Compose

The repository includes a Docker Compose setup with PostgreSQL and a profiled
`scrapyrus` execution container. The PostgreSQL service uses a pgvector-enabled
image because `scrapyrus embeddings` commands require the `vector` extension:

```
docker compose up -d postgres
```

The `scrapyrus` container mounts this repository at `/workspace` and sets
`SCRAPYRUS_DATABASE_URL` to the PostgreSQL service. It also mounts the local
`./exchange` directory at `/exchange` for moving data between the host and the
container. Start it as a one-off shell when you need to run the CLI:

```
docker compose run --build --rm scrapyrus bash
scrapyrus metadata ingest
scrapyrus transcriptions ingest
scrapyrus embeddings ingest \
    --inference-server-url <url> --model-name <model> --api-key <key>
```

Embedding ingestion reads the XML rows created by `transcriptions ingest`, so
both commands must target the same PostgreSQL database and must run in that
order.

The `scrapyrus` service is intentionally not started by the default `up`
command. Some Snap-packaged Docker installations fail to stop idle long-lived
containers during rebuilds with `cannot stop container: ... permission denied`;
using one-off `run --rm` containers avoids that host-side recreate path.

If `idp.data` is somewhere else inside the mounted workspace, pass it explicitly:

```
scrapyrus --idp-data /workspace/idp.data metadata ingest
```

To export the metadata database tables as CSV files:

```
scrapyrus metadata dump
```

If an older checkout already started the Compose stack with the stock PostgreSQL
image, recreate the database service after updating:

```
docker compose up -d --force-recreate postgres
```

## Acknowledgments

This repository was set up using the [SSC Cookiecutter for Python Packages](https://github.com/ssciwr/cookiecutter-python-package).
