Metadata-Version: 2.4
Name: shared-libs-py
Version: 1.0.2
Summary: Shared Python packages
Author-email: Aleksandrovic Palma <palma.aleksandrovic@gmail.com>
License-Expression: MIT
Project-URL: Repository, https://github.com/aleepl/shared-libs-py.git
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Requires-Python: ~=3.11.0
Description-Content-Type: text/markdown
Requires-Dist: azure-appconfiguration==1.7.1
Requires-Dist: azure-identity==1.23.0
Requires-Dist: azure-keyvault==4.2.0
Requires-Dist: azure-storage-blob==12.25.1
Requires-Dist: azure-storage-file-datalake==12.20.0
Requires-Dist: pydantic-settings==2.9.1
Requires-Dist: pydantic==2.11.5
Requires-Dist: aiofiles==24.1.0
Requires-Dist: sqlalchemy==2.0.41
Requires-Dist: pandas==2.3.0

# Data engineering shared python libs
This project is used by the data engineering team to share common libraries between multiple projects

## Table of Contents
1. [Authors](#1-authors)
2. [Project Structure](#2-project-structure)
3. [Getting Started](#3-getting-started)
4. [Usage](#4-usage)
5. [Development Setup](#5-development-setup)
6. [Database Migrations](#6-database-migrations)
7. [Documentation](#7-documentation)

## 1. Authors
**Principal maintainers**: data.engineering

**Team**: Data Engineering

## 2. Project Structure
```
shared-libs-py/

    ├── .tests/         <- local notebooks, data, etc.
    |
    ├── .vscode/        <- vscode config files
    |
    ├── devops/
    |   ├── ci
    |   |   └── build.yaml
    |   |   └── pr.yaml
    |   ├── cd
    |   |   └── deployment.yaml
    |
    ├── src/
    |   └──shared/
    │       ├── azure/           <- several azure libaries
    │       ├── db/              <- various config files related to project
    │       └── utils/           <- utility modules
    │
    ├── tests/
    │   ├── unit/
    │
    ├── Dockerfile          <- basic Dockerfile template
    ├── .dockerignore
    │
    ├── .gitignore
    ├── Makefile            <- environement creation and basic checks
    ├── pyproject.toml      <- basic standards config
    ├── README.md           <- project README template
    └── uv.lock             <- uv dependencies locker
```

## 3. Getting Started
### Prerequisites
- Make
- Python 3.11
- Docker & Docker Compose
- Liquibase (for database migrations)
### Installation
Clone this repository
   ```
   git clone <repository-url>
   cd shared-libs-py
   ```

Set create env and install libs
   ```
   make env
   ```

### Docker Setup
Start the services using Docker Compose:

```
docker-compose up -d
```

## 4. Usage
Below you will see the instructions to install the package under two different methods depending on your project's package manager.

## 4.1 Using UV

1. Go to: https://dev.azure.com/oneocean/
2. Click on "User Settings" -> "Personal access tokens".
3. Use "+ New Token" button to generate new token. It will only require read access to "Packaging"
4. Create 2 environment variables:
```
export UV_INDEX_ONEOCEAN_FEED_USERNAME=pylib
export UV_INDEX_ONEOCEAN_FEED_PASSWORD=[YOUR_TOKEN_HERE]
```
5. Add the package to pyproject.toml by using the following command:
```
uv add shared-libs-py --index oneocean-feed="https://pkgs.dev.azure.com/oneocean/Data/_packaging/OneOcean/pypi/simple/"
```
## 4.1 Using pip
1. Go to: https://dev.azure.com/oneocean/
2. Click on "User Settings" -> "Personal access tokens".
3. Use "+ New Token" button to generate new token. It will only require read access to "Packaging"
4. Create a pip.config file in the following path ~/.pip/pip.conf.
5. Add the index into the pip.conf by using the following configuration in it.
```
[global]
index-url = https://pypi.org/simple
extra-index-url = https://pylib:[YOUR_TOKEN_HERE]@pkgs.dev.azure.com/oneocean/Data/_packaging/OneOcean/pypi/simple/
```


## 5. Development Setup
### PR checks
This project uses the following linters and formatters:

- **Black** - Code formatting ([docs](https://github.com/psf/black))
- **isort** - Import sorting ([docs](https://pycqa.github.io/isort/))
- **Ruff** - Fast Python linter ([docs](https://docs.astral.sh/ruff/))
- **Pylint** - Static code analysis ([docs](https://pylint.readthedocs.io/))
- **Bandit** - Security-focused linter ([docs](https://bandit.readthedocs.io/))

To run all checks:
```
make checks
```

To automatically fix issues where possible:
```
make fix
```

### CI
- Pr: Once a PR is created, checks like black, isort, ruff, etc. are automatically executed. The PR cannot be completed until at least 1 reviewer approves it.
- Build: Create a docker image and push it to  acr. Cache is used to speed up the process. Once the image is created, dependency tracker is executed in order to scan for vulnerabilites and security issues.

### CD
- Deployment: When the build is completed, the task will publish the python package into the company's atifacts feed.

## 6. Documentation
