Metadata-Version: 2.4
Name: easy_data_loader
Version: 0.1.1
Summary: Data transfer utilities between files and databases
Author-email: Bojoi Gabriel <bojoigabriel@gmail.com>
Requires-Python: >=3.13
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: click>=8.3.0
Requires-Dist: openpyxl>=3.1.5
Requires-Dist: pandas>=2.3.3
Requires-Dist: pyarrow>=22.0.0
Requires-Dist: pydantic>=2.12.5
Requires-Dist: pydantic-settings>=2.12.0
Requires-Dist: pyodbc>=5.2.0
Requires-Dist: python-dotenv>=1.1.1
Requires-Dist: sqlalchemy>=2.0.43
Dynamic: license-file

# Easy Data Loader 🚀

**Easy Data Loader** is a flexible, modular Python library designed to streamline ETL (Extract, Transform, Load) processes between various data sources (CSV, Excel, Parquet) and SQL databases (MSSQL, PostgreSQL, and others).

## ✨ Key Features
- **Declarative Configuration**: Manage connections and pipelines through simple python files and `.env` resources.
- **Integrated CLI**: Initialize a standardized project structure with a single command.
- **Custom Transformation Hooks**: Inject your own Pandas transformation logic directly into the pipeline execution.
- **Performance Optimized**: Built-in support for chunked loading and writing to handle large datasets efficiently.
- **Extensible Architecture**: Uses a Factory Pattern for database connectors, making it easy to support new drivers.

---

## 📦 Installation

Install directly via `pip` or `uv`:

```bash
pip install easy_data_loader
```

## 🚀 Getting Started

1. Initialize a new project structure to generate template configurations:
   ```bash
   easy-loader init
   ```
2. Review the generated `config/` folders for sample resources and pipelines.
3. Run all discovered pipelines across the active configurations:
   ```bash
   easy-loader run_all
   ```

## 🏗️ Architecture & Process Flow

The system is designed with modularity in mind. Object dependencies and their standard instantiation lifecycle are executed through the `Configuration` singleton and the `CONNECTOR_FACTORY`. 

```mermaid
graph TD
    %% Main Execution Flow
    CLI[CLI Application] -->|Instantiates| Pipeline[Pipeline: Load, Procedure, Orchestrator]
    
    %% Configuration and Definitions
    Pipeline -->|Requests Definition & Resources| Config[Configuration Singleton]
    Config -->|Reads| PipelineDef[Pipeline Definitions .py]
    Config -->|Reads| ResourcesEnv[Resource Configs .env]
    
    %% Instantiation of Operations
    Pipeline -->|Uses ResourceConfig| Factory[CONNECTOR_FACTORY]
    Factory -->|Creates| DBConn[DatabaseConnector: SqlServer, SQLite]
    DBConn -->|Provides Engine to| DBOps[DatabaseOperations]
    
    Pipeline -->|Uses FileSettings| FileOps[FileOperations]
    
    %% Pipeline dependencies
    Pipeline -->|Contains| DBOps
    Pipeline -->|Contains| FileOps
    
    %% Audit
    Pipeline -->|Uses| AuditOps[Audit DatabaseOperations]
```

