Metadata-Version: 2.3
Name: containerio
Version: 1.0.0
Summary: Azure Blob Storage handler with unified local/cloud interface
Author: Alliance SwissPass
Requires-Dist: polars>=1.30.0
Requires-Dist: azure-storage-blob>=12.26.0
Requires-Dist: azure-identity>=1.25.0
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: rich>=13.0.0
Requires-Python: >=3.11
Project-URL: Repository, https://codefloe.com/Alliance-SwissPass/py-containerio
Description-Content-Type: text/markdown

# py-containerio

Shared Azure Blob Storage handler for internal and collaboration projects of [Alliance SwissPass](https://www.allianceswisspass.ch/).
Provides a unified interface for reading and writing data to both local filesystem and Azure Blob Storage, allowing seamless switching between local development and cloud environments.

## Features

- **Unified API**: Same code works for local files and Azure Blob Storage
- **Environment-based switching**: Use `STORAGE_ENV` to switch between local/staging/prod
- **[Polars](https://pola.rs/) integration**: Native support for Polars DataFrames and LazyFrames
- **Multiple auth methods**: Device code flow (default), client credentials (CI/CD), or SAS tokens
- **CLI tools**: Unified `io` command for listing, downloading, uploading, deleting blobs and generating SAS tokens
- **Memory-efficient**: Streaming support with `scan_*` and `sink_*` methods

## Install containerio

### From PyPI

The package is available on PyPI: [https://pypi.org/project/containerio/](https://pypi.org/project/containerio/)

```bash
pip install containerio
```

Or with uv:

```bash
uv add containerio
uv sync
```

## Use containerio as a package dependency

### From PyPI (recommended)

Add to your project's `pyproject.toml`:

```toml
[project]
dependencies = [
    "containerio",
]
```

Then run:

```bash
uv sync
```

### From Forgejo Package Registry

If you need the Forgejo version, configure the custom index:

```toml
[project]
dependencies = [
    "containerio",
]

[tool.uv.sources]
containerio = { index = "forgejo" }

[[tool.uv.index]]
name = "forgejo"
url = "https://codefloe.com/api/packages/Alliance-SwissPass/pypi/simple"
```

Then run:

```bash
uv sync
```

### From Git

For development or testing, install directly from the repository:

```toml
[project]
dependencies = [
    "containerio",
]

[tool.uv.sources]
containerio = { git = "https://codefloe.com/Alliance-SwissPass/py-containerio.git", tag = "v1.0.0" }
```

## Usage in Other Projects

When installed as a dependency, containerio integrates seamlessly with your project:

- **CLI tools are available** - Run `uv run io` directly from your project
- **`.env` files live in your project** - The storage module reads `.env` and `.env.staging` from your current working directory, not from the containerio package
- **No configuration in containerio needed** - Just add the dependency and configure in your project

Example workflow in your project:

```bash
# Your project directory
cd my-project

# Create your .env.staging file here
cp /path/to/.env.example .env.staging
# Edit .env.staging with your credentials

# List blobs in both containers
uv run io ls --env staging

# List blobs in local mode (./data/ directory)
uv run io ls

# Generate SAS tokens (updates .env.staging)
uv run io generate-sas --env staging --update-env
```

## Example Usage

```python
import os
from containerio import StorageHandler

# Set environment (defaults to "local" which uses ./data/ directory)
os.environ["STORAGE_ENV"] = "staging"
# or "prod"

# Read operations (uses read container)
handler = StorageHandler(container="ro")

# Read files
df = handler.read_csv_blob("path/to/file.csv")
df = handler.read_excel_blob("path/to/file.xlsx")
df, metadata = handler.read_parquet_blob("path/to/file.parquet")

# Lazy scan (memory-efficient for large files)
lf, metadata = handler.scan_parquet_blob("large_file.parquet")
lf = handler.scan_csv_blob("large_file.csv")

# Write operations (uses write container)
handler_rw = StorageHandler(container="rw")

handler_rw.write_csv_blob("output.csv", df)
handler_rw.write_parquet_blob("output.parquet", df)
handler_rw.write_excel_blob("output.xlsx", df)

# Stream LazyFrame directly to storage
handler_rw.sink_parquet_blob("output.parquet", lf)

# Upload any file (JSON, YAML, images, etc.)
handler_rw.upload_blob("config.json", "./local/config.json")

# List blobs
blobs = handler.list_blobs()    # Returns List[BlobInfo]
handler.download_blob("file")   # Download to local ./data/
handler_rw.delete_blob("file")  # Delete a blob or folder
handler_rw.move_blob("old.csv", "archive/old.csv")  # Move/rename
```

## Configuration

### Environment Variables

Set `STORAGE_ENV` to control the storage backend:

| Value | Behavior |
|-------|----------|
| `local` (default) | Uses local filesystem (`./data/` directory) |
| `staging` | Uses Azure staging storage (reads from `.env.staging`) |
| `prod` | Uses Azure production storage (reads from `.env`) |

### Azure Configuration

For Azure storage, create a `.env` file (for prod) or `.env.staging` (for staging).

**Important:** Restrict permissions on your `.env` file so only you can access it:

```bash
chmod 600 .env
```

#### Example `.env` (production)

```bash
AZURE_STORAGE_ACCOUNT=<azure_storage_account>
AZURE_CONTAINER_NAME_READ_PROD=<azure_container_name_read_prod>
AZURE_CONTAINER_NAME_WRITE_PROD=<azure_container_name_write_prod>
AZURE_TENANT_ID=<azure_tenant_id>
AZURE_CLIENT_ID=<azure_client_id>

# SAS tokens (optional, if not using device code auth)
AZURE_STORAGE_SAS_TOKEN_READ=<azure_storage_sas_token_read>
AZURE_STORAGE_SAS_TOKEN_WRITE=<azure_storage_sas_token_write>
```

#### Example `.env.staging`

```bash
AZURE_STORAGE_ACCOUNT=<azure_storage_account>
AZURE_CONTAINER_NAME_READ_STAGING=<azure_container_name_read_staging>
AZURE_CONTAINER_NAME_WRITE_STAGING=<azure_container_name_write_staging>
AZURE_TENANT_ID=<azure_tenant_id>
AZURE_CLIENT_ID=<azure_client_id>

# SAS tokens (optional, if not using device code auth)
AZURE_STORAGE_SAS_TOKEN_READ=<azure_storage_sas_token_read>
AZURE_STORAGE_SAS_TOKEN_WRITE=<azure_storage_sas_token_write>
```

### Authentication

Use `io auth` to manage authentication explicitly.
Device-code login and status auto-discover the environment from your `.env` / `.env.staging` file, so `--env` is optional.
SAS token and client credentials login require explicit `--env`:

```bash
# Device code login (default) — auto-discovers env from .env file
uv run io auth

# Device code login with explicit env
uv run io auth --env staging

# Check auth status across all methods
uv run io auth status

# SAS token login — requires --env (wrong env = wrong token)
uv run io auth login --method sas-token --env staging

# Client credentials — requires --env (for CI/CD pipelines)
uv run io auth login --method client-credentials --env prod

# Clear cached device code tokens
uv run io auth clear

# Force re-authentication (clears cache first)
uv run io auth login --force
```

The chosen auth method is saved to the session. Subsequent commands (`ls`, `download`, etc.) automatically use it.

### Authentication Methods

1. **Device Code Flow** (recommended for development)
   - Interactive browser-based authentication
   - Tokens are cached locally for 90 days
   - No credentials stored in files

2. **Client Credentials**
   - For CI/CD pipelines
   - Use `io auth login --method client-credentials` to authenticate
   - Requires:
     - `AZURE_TENANT_ID`: Service principal's tenant ID
     - `AZURE_CLIENT_ID_CONTAINER`: Service principal's client ID
     - `AZURE_CLIENT_SECRET_CONTAINER_PROD`: Service principal's client secret for production
     - `AZURE_CLIENT_SECRET_CONTAINER_STAGING`: Service principal's client secret for staging

3. **SAS Token**
   - Use pre-generated tokens with limited validity
   - Good for sharing temporary access

## CLI Tools

The package provides a unified `io` command with subcommands for all storage operations. All commands accept `--env prod|staging` to override `STORAGE_ENV`.

| Command | Usage | Description |
|---------|-------|-------------|
| `auth` | `uv run io auth [login\|status\|clear] [--method M] [--force]` | Manage authentication |
| `ls` | `uv run io ls [--container ro\|rw\|ro-rw]` | List blobs in storage |
| `download` | `uv run io download <blob_name> [--container ro\|rw]` | Download a blob to local `data/` directory |
| `upload` | `uv run io upload <blob_name> <file_path>` | Upload a local file to the write container |
| `rm` | `uv run io rm <path>` | Delete a blob or folder from the write container |
| `mv` | `uv run io mv <source> <dest> [--overwrite]` | Move (rename) a blob within the write container |
| `generate-sas` | `uv run io generate-sas [--container ro\|rw\|ro-rw] [--days N] [--update-env]` | Generate Azure Storage SAS tokens |

### Examples

```bash
# Authenticate with device code flow (auto-discovers env)
uv run io auth

# Check authentication status (auto-discovers env)
uv run io auth status

# List blobs in local mode (./data/)
uv run io ls

# List both read and write containers (default --container ro-rw)
uv run io ls --env staging

# Download a blob to the local data/ directory
uv run io download path/to/file.csv --env staging

# Upload a local file to the write container
uv run io upload path/to/blob.csv ./local/file.csv --env staging

# Delete a blob or folder from the write container
uv run io rm path/to/blob.csv --env staging
uv run io rm old-folder/ --env staging

# Move a blob within the write container
uv run io mv data/old.csv archive/old.csv --env staging

# Overwrite an existing destination
uv run io mv data/old.csv data/existing.csv --env staging --overwrite

# Generate SAS tokens and update .env.staging file
uv run io generate-sas --env staging --update-env

# Generate only read container token, valid for 3 days
uv run io generate-sas --env prod --container ro --days 3
```

### Session

When doing multiple operations on the same container, use `session` to save defaults so you don't have to repeat `--env` and `--container` on every command.
Both `--env` and `--container` are required to start a session.

```bash
# Start a session — saves env and container
uv run io session --env staging --container rw

# Show active session
uv run io session

# Now these use the saved defaults
uv run io ls
uv run io rm old-folder/
uv run io mv data/old.csv archive/old.csv

# Explicit flags still override the session
uv run io ls --container ro

# End the session
uv run io session clear
```

The session is stored in `.containerio/session.json` in the current directory. It is project-local and git-ignored.

The session is also available programmatically via the `Session` class:

```python
from containerio.session import Session

session = Session()
print(session.auth_method)  # e.g. 'client-credentials'
print(session.env)          # e.g. 'staging'
print(session.is_active)    # True if any session data exists
```

Options:

- `--env`: `prod` or `staging` (default: staging for sas; uses `STORAGE_ENV` for other commands)
- `--container`: `ro`, `rw`, or `ro-rw` (default: ro-rw)
- `--days`: Token validity in days, max 7 (default: 1)
- `--update-env`: Write tokens to .env file instead of printing

**Note:** SAS tokens are generated using a *user delegation key* which has a maximum validity of **7 days** (Azure limitation). See [Microsoft documentation](https://learn.microsoft.com/en-us/rest/api/storageservices/get-user-delegation-key#request-body) for details.

## API Reference

### StorageHandler

Main class for storage operations.

```python
StorageHandler(
    container: Literal["ro", "rw"] = "ro",
    auth_type: Optional[str] = None,
    session: Optional[Session] = None,
)
```

- `auth_type=None`: Falls back to session auth method, then device code flow
- `auth_type="device-code"`: Device code flow (interactive)
- `auth_type="sas-token"`: SAS token authentication
- `auth_type="client-credentials"`: Client credentials (service principal)
- `session=None`: Creates a new `Session` (skipped in local mode). Pass an existing `Session` instance for dependency injection.
- `container="ro"`: Uses the read container (`AZURE_CONTAINER_NAME_READ_{ENV}`)
- `container="rw"`: Uses the write container (`AZURE_CONTAINER_NAME_WRITE_{ENV}`)

#### Read Methods

| Method | Description |
|--------|-------------|
| `read_parquet_blob(blob_name)` | Read parquet file, returns `(DataFrame, metadata)` |
| `read_csv_blob(blob_name, **kwargs)` | Read CSV file into DataFrame |
| `read_excel_blob(blob_name, **kwargs)` | Read Excel file into DataFrame |
| `scan_parquet_blob(blob_name)` | Lazy scan parquet, returns `(LazyFrame, metadata)` |
| `scan_csv_blob(blob_name, **kwargs)` | Lazy scan CSV file |

#### Write Methods

| Method | Description |
|--------|-------------|
| `write_parquet_blob(blob_name, df)` | Write DataFrame to parquet |
| `write_csv_blob(blob_name, df, **kwargs)` | Write DataFrame to CSV |
| `write_excel_blob(blob_name, df, **kwargs)` | Write DataFrame to Excel |
| `sink_parquet_blob(blob_name, lf)` | Stream LazyFrame to parquet |

#### Utility Methods

| Method | Description |
|--------|-------------|
| `upload_blob(blob_name, file_path, progress_hook)` | Upload a local file to storage |
| `delete_blob(blob_name)` | Delete a blob, file, or folder recursively. Returns `List[str]` of deleted paths |
| `move_blob(source, destination, overwrite)` | Move (rename) a blob within the same container |
| `download_blob(blob_name, progress_hook)` | Download blob to local `data/` directory. Returns `Optional[Path]` |
| `list_blobs()` | List all blobs, returns `List[BlobInfo]` |

### AzureStorageConfig

Configuration dataclass for Azure storage settings.

```python
from containerio import AzureStorageConfig

# Load from environment
config = AzureStorageConfig.from_environment()  # Uses STORAGE_ENV
config = AzureStorageConfig.from_environment(environment="staging")

# Access configuration
print(config.storage_account)
print(config.container_name_read)
```

### Environment

Enum for storage environments.

```python
from containerio import Environment

# Parse from string
env = Environment.from_string("prod")  # Environment.PROD

# Use enum directly
if env == Environment.LOCAL:
    print("Using local storage")
```

## Development

```bash
# Clone and install
git clone ssh://git@codefloe.com/Alliance-SwissPass/py-containerio.git
cd py-containerio
uv sync --group dev

# Run tests
uv run pytest

# Lint and format
uv run ruff check
uv run ruff format
```

## Version Management with git-sv

This project uses [git-sv](https://github.com/thegeeklab/git-sv) for semantic versioning and changelog generation.

The `.gitsv/config.yaml` file defines how versions are bumped based on commit messages.

1. **Commit messages**: Follow [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) format
2. **Version bump**: The CI pipeline automatically runs `git-sv bump` based on commit history
3. **Changelog**: The CI pipeline generates the changelog using `git-sv changelog` and pins it as a Git issue
4. **Release**: The CI pipeline automatically handles tagging and publishing

The CI pipeline will automatically:
- Bump the version based on commit history
- Generate and pin the changelog as a Git issue
- Build and publish the package to Forgejo
- Create a release with the changelog notes

## Publishing

### Via CI Pipeline (recommended)

The repository includes a Crow CI pipeline that automatically builds, tests, and publishes the package when a git tag is pushed:

2. Commit and push your changes
3. Create and push a git tag (e.g., `git tag v1.0.0 && git push --tags`)

The pipeline will:
- Run linting and tests
- Update the version in `pyproject.toml` automatically
- Generate changelog from git history using `git-sv`
- Build and publish the package to Forgejo and PyPI
- Create a release with changelog notes
- Push version updates back to the repository

**Note:** For CI publishing, ensure the `UV_PUBLISH_TOKEN` secret is configured in your Crow CI pipeline settings.

## Security Best Practices

1. **Never commit credentials**: Always add `.env*` to your `.gitignore` file
2. **Restrict file permissions**: Use `chmod 600 .env` to limit access to your credentials
3. **Use short-lived tokens**: SAS tokens have a maximum validity of 7 days
4. **Rotate secrets regularly**: Especially for production environments and CI/CD pipelines
5. **Use device code flow**: For development to avoid storing credentials in files
6. **Environment isolation**: Use separate credentials for staging and production
7. **Limit token scope**: When generating SAS tokens, specify the minimum required permissions

## References

- [Delegate access with shared access signature](https://learn.microsoft.com/en-us/rest/api/storageservices/delegate-access-with-shared-access-signature)
- [Create user delegation SAS](https://learn.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas)
- [Get user delegation key](https://learn.microsoft.com/en-us/rest/api/storageservices/get-user-delegation-key)

## License

Alliance SwissPass <https://allianceswisspass.ch>
