Metadata-Version: 2.4
Name: das-setup-helper
Version: 0.2.4
Summary: Public helper to setup crawlers and manage acquisitions in WIHP (EUROSTAT)
Author: estat-wihp
License-Expression: EUPL-1.2
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: boto3<2,>=1.42
Requires-Dist: requests<3,>=2.33
Requires-Dist: pyjwt<3,>=2.11
Requires-Dist: pyyaml<7,>=6.0
Requires-Dist: python-dotenv<2,>=1.1
Requires-Dist: pandas<4,>=3.0
Requires-Dist: aws-secretsmanager-caching<2,>=1.1
Provides-Extra: dev
Requires-Dist: isort<7,>=6.0; extra == "dev"
Requires-Dist: black<27,>=26.1; extra == "dev"
Requires-Dist: pytest<10,>=9.0; extra == "dev"
Requires-Dist: pytest-cov<8,>=7.0; extra == "dev"
Requires-Dist: pytest-asyncio<2,>=1.2; extra == "dev"
Requires-Dist: pre-commit<5,>=4.5; extra == "dev"
Requires-Dist: sphinx<10,>=9.0; extra == "dev"
Requires-Dist: sphinx-rtd-theme<4,>=3.0; extra == "dev"
Requires-Dist: sphinx-autodoc-typehints<4,>=3.9; extra == "dev"
Dynamic: license-file

# DAS Setup Helper

A Python CLI tool to set up crawlers, manage acquisitions, and extract structured data in WIHP (EUROSTAT).
It provides four pipelines: creating crawlers, triggering acquisitions, stopping acquisitions, and extracting data from JSON acquisitions to processed CSV files.

---

## Requirements

- Python >= 3.12
- [uv](https://docs.astral.sh/uv/) — fast Python package and project manager

Install uv (once, globally):
```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
```

---

## Installation

### From PyPI (recommended)

```bash
# Create a project directory
mkdir das-setup-helper && cd das-setup-helper

# Install the package in an isolated environment
uv add das-setup-helper
```

All CLI commands (`das-setup-crawler`, `das-create-acq`, `das-stop-acq`, `das-extract-data`) are then available via `uv run`.

### From source (contributors)

```bash
# Clone the repository
git clone <repo-url>
cd das-setup-helper

# Install all dependencies including dev tools
make sync
# or directly:
uv sync --extra dev --no-install-project
```

The source workflow installs dependencies without installing the repository as
an editable package. This prevents development commits after a numbered
`.devN` tag from being interpreted as a new package version by `setuptools_scm`.
The Make targets run the pipeline modules directly from the working tree.

---

## Configuration

Copy the template and fill in your values:
```bash
cp env.sh.template env.sh
# edit env.sh with your settings
```

Key variables:

| Variable | Description |
|---|---|
| `STORAGE_BACKEND` | `local` (default) or `s3` |
| `USE_SECRET_MANAGER` | `false` (default, uses env vars) or `true` (uses AWS Secrets Manager) |
| `AWS_SECRET_MANAGER_NAME` | Secret name in Secrets Manager — only needed if `USE_SECRET_MANAGER=true` |
| `S3_AWS_WIHP_CONFIG_DIRECTORY` | Path to the config directory (local path or S3 prefix) |
| `S3_AWS_WIHP_INPUT_SOURCES_TO_CREATE` | Path to the input CSV file with sources to create |
| `S3_AWS_WHIP_READY_FOR_CRAWLING_DIRECTORY` | Directory where crawler outputs are written |
| `S3_AWS_STAGING_DIRECTORY` | Root directory for staging JSON content files (default `staging`) |
| `S3_AWS_PROCESSED_DIRECTORY` | Root directory for processed CSV output files (default `processed`) |
| `OVERWRITE_EXISTING` | `false` (default, skip existing CSVs) or `true` (re-process and overwrite) |
| `WIHP_API_URL` | WIHP API base URL |
| `WIHP_TOKEN_URL` | WIHP OAuth token endpoint |
| `WIHP_API_KEY` | WIHP API key |
| `WIHP_GROUP` | WIHP group (e.g. `/MNE`) |

### Local storage

When `STORAGE_BACKEND=local`, paths are local filesystem paths.
The `local_storage/` directory mirrors the S3 structure:

```
local_storage/
├── config/                   # input files
├── ready_for_crawling/       # crawler outputs
├── ready_to_export/
├── done/
├── staging/                  # input JSON acquisition files
└── processed/                # extracted CSV output files and _metadata/
```

The only input file required to run `das-setup-crawler`:
```
local_storage/MNE/config/dashelper/sources_to_create.csv
```

Format (semicolon-separated):
```csv
mne_name;wikipedia_url
Airbus;https://en.wikipedia.org/wiki/Airbus
TotalEnergies;https://en.wikipedia.org/wiki/TotalEnergies
```

---

## Running the pipelines

Load your environment variables then run a pipeline:

```bash
source env.sh

# 1. Create a crawler from the input CSV
make run-crawler

# 2. Trigger acquisitions for existing crawlers
make run-acq

# 3. Stop running acquisitions
make run-stop

# 4. Extract data from staging JSON files to processed CSV files
make run-extract
# or via CLI entry point:
uv run das-extract-data [--force]
```

When the published package is installed with `uv add das-setup-helper`, the
equivalent commands are `uv run das-setup-crawler`, `uv run das-create-acq`,
`uv run das-stop-acq`, and `uv run das-extract-data`.

Pipeline order: **setup-crawler → create-acq → stop-acq → extract-data**

---

## Development

```bash
# Run tests
make test

# Format code
make lint

# Refresh dependency lock (without upgrading)
make lock

# Upgrade all dependencies
make lock-upgrade
```

Available `make` targets:
```bash
make info
```

---

## Deployment (prod/staging)

In production (MWAA / Lambda), set:
```bash
USE_SECRET_MANAGER=true
AWS_SECRET_MANAGER_NAME=prod/DASHelper
STORAGE_BACKEND=s3
```

All other variables are loaded automatically from AWS Secrets Manager.
The IAM role must have `secretsmanager:GetSecretValue` permission on the secret.

---

## Release

The complete contributor and release-manager workflow is documented in
[Release process](docs/release-process.md).

The package version is derived from Git tags and follows Python package versioning conventions (PEP 440):

- Development tags use `vX.Y.Z.devN`.
- Release candidate tags use `vX.Y.Z.rcN`.
- Final releases use `vX.Y.Z`.

GitLab CI release flow:

- On `development`, run `tag-dev-patch`, `tag-dev-minor`, `tag-dev-major`, or
	`tag-dev-same`. The resulting development tag is published to the internal
	GitLab Python registry.
- Merge `development` into `test` without squashing, then run `tag-rc`. The
	resulting release candidate is published to TestPyPI.
- Merge `test` into `main` without squashing, then run `tag-release`. The final
	tag is published to PyPI and creates a GitLab Release with notes generated
	from the commits since the previous final tag.

Versions remain entirely tag-driven through `setuptools_scm`. Commit messages do
not determine the next package version.

### Commit convention

Commit subjects and merge request titles follow Conventional Commits:

```text
feat(api): add source filtering
fix: handle an expired token
refactor!: remove the legacy connector
```

Supported types are `feat`, `fix`, `perf`, `refactor`, `docs`, `test`, `build`,
`ci`, `chore`, and `revert`. Install the local `commit-msg` check with:

```bash
make pre-commit-install
```

The CI validates merge request titles and commits independently of local hooks.
Feature merge requests should be squashed using their Conventional Commit title.
Promotion merge requests from `development` to `test` and from `test` to `main`
must not be squashed, so the final release history retains the commits used to
generate release notes.

SonarQube runs in the `quality` stage for merge requests and for the `development` and `main` branches.

To allow the release-tag job to push tags from CI, define this GitLab CI/CD variable:

- `GIT_PUSH_TOKEN`

Also define the repository tokens used for publishing:

- `TEST_PYPI_TOKEN`
- `PYPI_TOKEN`

For SonarQube, also define:

- `SONAR_TOKEN`

## License

This project is distributed under the European Union Public Licence v1.2
(EUPL-1.2). The full licence text is available in [LICENSE](LICENSE) and on the
European Commission website:
https://commission.europa.eu/about/departments-and-executive-agencies/digital-services/open-source-strategy-history/european-union-public-licence_en
