Metadata-Version: 2.4
Name: data-product-portal
Version: 0.8.0
Summary: Data Product Portal core
Author: Stijn Janssens
Author-email: stijn.janssens@dataminded.com
Requires-Python: >=3.13,<4.0
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Dist: alembic (>=1.20.0,<2.0.0)
Requires-Dist: asgi-correlation-id (>=5.0.1,<6.0.0)
Requires-Dist: asyncpg (>=0.31,<0.33)
Requires-Dist: authlib (==1.8.0)
Requires-Dist: boto3 (>=1.43.93,<2.0.0)
Requires-Dist: cachetools (>=7.1.8,<8.0.0)
Requires-Dist: casbin_sqlalchemy_adapter (>=1.4.0,<2.0.0)
Requires-Dist: cryptography (>=50.0.1,<51.0.0)
Requires-Dist: emailgen (>=0.1.1,<0.2.0)
Requires-Dist: fastapi (==0.141.1)
Requires-Dist: fastembed (>=0.8.0,<0.9.0)
Requires-Dist: fastmcp (==3.4.8)
Requires-Dist: greenlet (>=3.5.5,<4.0.0)
Requires-Dist: gunicorn (>=26.2.0,<27.0.0)
Requires-Dist: h11 (>=0.16.0,<0.17.0)
Requires-Dist: httpx (>=0.28.1,<0.29.0)
Requires-Dist: jinja2 (>=3.1.6,<4.0.0)
Requires-Dist: opentelemetry-api (>=1.39.1,<2.0.0)
Requires-Dist: opentelemetry-distro (>=0.64b0,<0.65)
Requires-Dist: opentelemetry-exporter-otlp (>=1.39.1,<2.0.0)
Requires-Dist: opentelemetry-instrumentation-fastapi (>=0.64b0,<0.65)
Requires-Dist: opentelemetry-instrumentation-sqlalchemy (>=0.64b0,<0.65)
Requires-Dist: opentelemetry-sdk (>=1.39.1,<2.0.0)
Requires-Dist: pgvector (>=0.5.0,<0.6.0)
Requires-Dist: posthog (>=7.53.0,<8.0.0)
Requires-Dist: psycopg2-binary (>=2.9.13,<3.0.0)
Requires-Dist: pycasbin (>=2.8.0,<3.0.0)
Requires-Dist: pydantic-settings (>=2.15.0,<3.0.0)
Requires-Dist: pydantic[email] (>=2.13.5,<3.0.0)
Requires-Dist: pyjwt (>=2.14.0,<3.0.0)
Requires-Dist: python-dotenv (>=1.2.3,<2.0.0)
Requires-Dist: pytz (>=2026.3,<2027.0)
Requires-Dist: pyyaml (>=6.0.3,<7.0.0)
Requires-Dist: requests (>=2.34.2,<3.0.0)
Requires-Dist: shortuuid (>=1.0.13,<2.0.0)
Requires-Dist: sqlalchemy-easy-softdelete (>=0.9.1,<0.10.0)
Requires-Dist: sqlalchemy-utils (>=0.42.1,<0.43.0)
Requires-Dist: sqlalchemy[asyncio] (>=2.0.52,<3.0.0)
Requires-Dist: typer (>=0.27.2,<0.28.0)
Requires-Dist: uvicorn (>=0.52.4,<0.55.0)
Description-Content-Type: text/markdown

[![FastAPI][FastAPI]][FastAPI-url]
[![Python][Python]][Python-url]
[![Postgres][Postgres]][Postgres-url]
[![Docker][Docker]][Docker-url]

# Backend Getting Started

## Prerequisites

### Python

- Install [Python 3.13](https://www.python.org/downloads) on your machine.

### Poetry

- Install [Poetry](https://python-poetry.org/docs/#installation) on your machine.
- Make sure it is available on your PATH.

### Configuration (.env file)

Both for local execution and local development, you need to specify some configuration arguments.

You can use the following `.env` file to get started:

```dotenv
POSTGRES_PASSWORD=abc123
POSTGRES_PORT=5432
POSTGRES_USER=postgres
POSTGRES_DB=data-product-portal
POSTGRES_SERVER=localhost

CORS_ALLOWED_ORIGINS=http://localhost:3000,http://localhost:5050,http://localhost:8080
AWS_DEFAULT_REGION=eu-west-1
LOGGING_DIRECTORY=./tmp/logs
HOST=http://localhost:3000/
SLOW_QUERY_LOGGING_ENABLED=True
OPENTELEMETRY_TRACES_ENABLED=True
```

## Running the project

### Installation

In order to install all project dependencies using poetry,
open the 'backend' directory as a workspace in your favorite IDE and execute the command below.

```sh
poetry install
```

### Starting required services

Required services can be started using docker compose:

```bash
docker compose up -d postgresql mailhog jaeger
```

### Populating the database

In order to initialize your database with the correct structure, execute the command below from the backend folder.

```sh
poetry run python -m app.db_tool init "sample_data.sql"
```

This populates the database with sample data from [sample_data.sql](sample_data.sql) .
If you want to use your custom data, you should create a .sql file similar to
the [sample_data.sql](sample_data.sql) file and reference that one in the command above.

### Running the backend

In order to run the project in development mode, after initializing the database, execute the command below.

```sh
poetry run python -m app.local_startup
```

:tada:

The backend is now running on port 5050.

## Enabling the MCP server
The MCP server can work (locally or remotely) via Claude Desktop.
Install Claude Desktop and run `poetry run fastmcp install claude-desktop app/remote_proxy.py -e $(pwd) --env-file ./.env --env DISABLED_AWS='true'` in the `backend` folder. Make sure the `HOST` in your .env file is either referring to your localhost backend or the remote deployed backend.

## Development utilities

### Including the virtual environment in your IDE

- Execute the command below to display the location of your virtual environment.

```sh
poetry env info --path
```

- You can use this location to pass in your IDE to be able to run the source code from your IDE and to resolve all imports.

### Data

A small CLI tool is delivered to enable database setup / migrations.
The CLI tool fetches database configuration from the `.env` file in the current folder.
Make sure your database connection variables are set up correctly.
Execute `python -m app.db_tool --help` from the `backend` folder to receive the help text from this tool.

#### Force Database recreation

If you want to completely destroy and recreate the database when performing the commands listed below,
you should specify the '--force' option.

```sh
python -m app.db_tool init --force "sample_data.sql"
```

### Database migrations

Whenever you develop functionality that would include a database structure change,
you will need to [create an alembic migration script](https://alembic.sqlalchemy.org/en/latest/tutorial.html#create-a-migration-script) for it.
This comes down to running `python -m app.db_tool revision "{your message}"` in the `/backend` directory.

In order to apply this migration using the database CLI tool you need to execute the command below.

```sh
python -m app.db_tool migrate
```

## Testing

### Integration testing

To run the integration tests, execute the following commands:

```sh
poetry install --all-groups
docker compose up -d postgresql
poetry run pytest -v
```

It will install test dependencies, boot up the test database and run the tests.

### Load testing (Locust)

Run a baseline headless test against local backend (10 users, spawn 5/s, 1m):

```sh
poetry run poe loadtest --headless -H http://localhost:5050 -u 10 -r 5 -t 1m
```

For more info see `backend/loadtest/locustfile.py`

<!-- MARKDOWN LINKS & IMAGES -->
<!-- https://www.markdownguide.org/basic-syntax/#reference-style-links -->

[FastAPI]: https://img.shields.io/badge/FastAPI-005571?style=for-the-badge&logo=fastapi

[FastAPI-url]: https://fastapi.tiangolo.com

[Docker]: https://img.shields.io/badge/docker-%230db7ed.svg?style=for-the-badge&logo=docker&logoColor=white

[Docker-url]: https://www.docker.com

[Postgres]:https://img.shields.io/badge/postgres-%23316192.svg?style=for-the-badge&logo=postgresql&logoColor=white

[Postgres-url]:https://www.postgresql.org

[Python]:https://img.shields.io/badge/python-3670A0?style=for-the-badge&logo=python&logoColor=ffdd54

[Python-url]:https://www.python.org

