Metadata-Version: 2.4
Name: lotos
Version: 0.1.0
Summary: Self-hosted data ingestion framework — extract, transform, and load data from anywhere.
Author: yob
License: ARTEFACT
Requires-Python: >=3.11
Requires-Dist: httpx>=0.27
Requires-Dist: polars>=1.0.0
Requires-Dist: pydantic-settings>=2.0
Requires-Dist: pydantic>=2.0
Requires-Dist: pymysql
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.0
Requires-Dist: sqlalchemy>=2.0
Requires-Dist: structlog>=24.0
Requires-Dist: typer[all]>=0.12
Provides-Extra: all
Requires-Dist: azure-identity>=1.0; extra == 'all'
Requires-Dist: azure-storage-blob>=12.0; extra == 'all'
Requires-Dist: azure-storage-file-datalake>=12.0; extra == 'all'
Requires-Dist: psycopg2-binary>=2.9; extra == 'all'
Requires-Dist: pymysql>=1.1; extra == 'all'
Requires-Dist: pyodbc>=5.0; extra == 'all'
Provides-Extra: azure
Requires-Dist: azure-identity>=1.0; extra == 'azure'
Requires-Dist: azure-storage-blob>=12.0; extra == 'azure'
Requires-Dist: azure-storage-file-datalake>=12.0; extra == 'azure'
Provides-Extra: dev
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Provides-Extra: mssql
Requires-Dist: pyodbc>=5.0; extra == 'mssql'
Provides-Extra: mysql
Requires-Dist: pymysql>=1.1; extra == 'mysql'
Provides-Extra: postgres
Requires-Dist: psycopg2-binary>=2.9; extra == 'postgres'
Description-Content-Type: text/markdown

# 🪷 Lotos

**Self-hosted data ingestion framework** — extract, transform, and load data from anywhere.

Lotos is a lightweight, YAML-driven ETL framework built with Python. Define pipelines as configuration, not code.

---

## Features

- **YAML-based pipelines** — declarative, version-controlled, easy to read
- **Pluggable architecture** — connectors, transforms, and sinks are auto-discovered via a registry
- **Source connectors** — SQL databases, REST APIs, local/remote files (CSV, JSON, Parquet)
- **Transform chain** — select, rename, cast, filter, deduplicate, flatten, SQL transforms, computed columns, custom expressions, and schema validation
- **Destination sinks** — local files, Azure Blob Storage, Azure Data Lake Storage Gen2
- **Secret resolution** — reference environment variables or `.env` values with `${SECRET:KEY}` syntax
- **Incremental loading** — watermark-based extraction for processing only new/changed rows
- **Structured logging** — powered by `structlog` with JSON or console output
- **CLI interface** — run, validate, inspect, and scaffold pipelines from the terminal

---

## Installation

```bash
pip install -e .
```

With optional dependencies:

```bash
pip install -e ".[azure]"       # Azure Blob / ADLS Gen2
pip install -e ".[postgres]"    # PostgreSQL
pip install -e ".[mysql]"       # MySQL
pip install -e ".[all]"         # Everything
pip install -e ".[dev]"         # Dev tools (pytest, ruff)
```

> **Requires Python 3.11+**

---

## Quick Start

### 1. Generate a pipeline template

```bash
lotos init my_pipeline --source sql
```

### 2. Edit the YAML

```yaml
pipeline:
  name: my_pipeline
  description: "Extract customers and clean data"
  version: "1.0"

source:
  connector: sql
  config:
    connection_string: "mysql+pymysql://user:pass@localhost:3306/mydb"
    query: "SELECT * FROM customers"

transforms:
  - type: select_columns
    config:
      columns: [id, name, email, created_at]

  - type: deduplicate
    config:
      subset: [id]
      keep: last

sink:
  connector: file
  config:
    path: "output/customers.parquet"
    format: parquet
  write_mode: overwrite
```

### 3. Run it

```bash
lotos run pipelines/my_pipeline.yaml --log-level INFO
```

---

## CLI Commands

| Command | Description |
|---|---|
| `lotos run <file>` | Run a pipeline |
| `lotos validate <file>` | Validate YAML without running |
| `lotos inspect <file>` | Show pipeline details |
| `lotos init <name>` | Generate a pipeline template |
| `lotos list connectors` | List available source connectors |
| `lotos list transforms` | List available transforms |
| `lotos list sinks` | List available destination sinks |
| `lotos list all` | List everything |

### Run options

```
--output, -o       Save result to file (csv/json/parquet)
--format, -f       Output format (default: parquet)
--dry-run          Extract + transform only, skip sink
--log-level, -l    DEBUG, INFO, WARNING, ERROR
--log-format       console or json
```

---

## Connectors

| Name | Type | Description |
|---|---|---|
| `sql` | Source | Any SQLAlchemy-compatible database |
| `rest_api` | Source | REST APIs with pagination & auth |
| `file` | Source | CSV, JSON, Parquet (local or Azure Blob) |
| `file` | Sink | Write to local files or Azure Blob |
| `adls_gen2` | Sink | Azure Data Lake Storage Gen2 / Blob Storage |

### ADLS Gen2 Sink

The `adls_gen2` sink supports both Azure storage modes via the `hierarchical_namespace` option:

```yaml
sink:
  connector: adls_gen2
  config:
    account_name: "mystorageaccount"
    container_name: "mycontainer"
    directory_path: "data/output"
    file_name: "results"
    format: parquet
    hierarchical_namespace: false  # false = Blob API, true = Data Lake (DFS) API
    account_key: "${SECRET:ADLS_KEY}"
  write_mode: overwrite
```

---

## Transforms

| Name | Description |
|---|---|
| `select_columns` | Keep only specified columns |
| `rename_columns` | Rename columns via a mapping |
| `cast_types` | Cast column data types |
| `filter_rows` | Filter rows by conditions |
| `deduplicate` | Remove duplicate rows |
| `flatten` | Flatten nested structs/lists |
| `sql` | Run SQL queries (Polars SQL — no DB needed) |
| `computed` | Add computed columns with expressions |
| `custom` | Apply a custom Python expression |
| `validate_schema` | Validate column types and constraints |

---

## Project Structure

```
lotos/
├── cli.py                  # CLI (Typer)
├── config/                 # Settings, logging, secrets
├── connectors/             # Source connectors (SQL, REST, file)
├── core/                   # Pipeline engine, models, registry
├── sinks/                  # Destination sinks (file, ADLS Gen2)
├── transforms/             # Transform plugins
└── monitoring/             # Monitoring (placeholder)
pipelines/                  # Pipeline YAML definitions
tests/                      # Test suite
```

---

## Development

```bash
pip install -e ".[dev]"
pytest
ruff check .
```

---

## License

Artefact
