Metadata-Version: 2.1
Name: dbengine
Version: 1.0.1
Summary: A Python package providing a unified interface for database operations, supporting Parquet, SQLite, and PostgreSQL backends.
Author-Email: Tome Magalhaes Gouveia <tome.pmg@gmail.com>
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Requires-Dist: pandas>=2.0.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: psycopg2-binary>=2.9.0
Requires-Dist: pyarrow>=12.0.0
Requires-Dist: sqlalchemy==2.0.43
Requires-Dist: whoosh>=2.7.4
Requires-Dist: black
Requires-Dist: flake8
Requires-Dist: isort
Requires-Dist: ipykernel
Requires-Dist: pre-commit
Requires-Dist: pytest>=7.0.0
Requires-Dist: pytest-cov>=4.0.0
Description-Content-Type: text/markdown

# DBEngine

A unified, production-ready database interface for Python that provides seamless access to SQLite, Parquet, and PostgreSQL databases through a consistent pandas-based API.

## Features

- 🚀 **Multi-Database Support**: Work with SQLite, Parquet files, and PostgreSQL databases using the same API
- 🔧 **Production-Ready**: Built-in configuration management, error handling, and security features
- 🛡️ **Security First**: Environment-based configuration, security validation, and secure credential handling
- 🎯 **Pandas Integration**: Native pandas DataFrame/Series support for all operations
- ⚡ **High Performance**: Connection pooling, batching, and optimised data handling
- 🐳 **PostgreSQL Server Management**: Programmatically start/stop PostgreSQL servers using Docker for development and testing
- 🧪 **Comprehensive Testing**: Full test suite with validation and integration tests
- 📦 **Easy Setup**: Simple installation and configuration with sensible defaults

## Quick Start

### Installation

```bash
pip install dbengine
```

### Basic Usage

```python
from dbengine import create_database
import pandas as pd

# Create a SQLite database
db = create_database('sqlite', path='my_data.db')

# Create sample data
data = pd.DataFrame({
    'id': [1, 2, 3],
    'name': ['Alice', 'Bob', 'Charlie'],
    'age': [25, 30, 35]
})

# Write data
db.write(table_name='users', data)

# Read data
users = db.query(table_name='users')
print(users)

# Query data
young_users = db.query(criteria={'age': 25}, table_name='users')
print(young_users)

```


### PostgreSQL Server Management

```python

## Supported Databasespython
from dbengine import create_postgres_server, PostgreSQLDatabase

# Start a PostgreSQL server for development/testing
with create_postgres_server(port=5433) as server:
    # Create database connection
    db = PostgreSQLDatabase(**server.get_connection_params())

    # Use the database normally
    data = pd.DataFrame({'id': [1, 2], 'name': ['Alice', 'Bob']})
    db.write(table_name='users', item=data)

    # Database and server automatically cleaned up
```

## Supported Databases

### SQLite
- **Use Case**: Local development, testing, lightweight applications
- **Features**: File-based, serverless, ACID transactions
- **Configuration**: Simple file path specification

### Parquet
- **Use Case**: Data analytics, archival, big data processing
- **Features**: Columnar storage, compression, fast analytics
- **Configuration**: Directory path with compression options

### PostgreSQL
- **Use Case**: Production applications, multi-user systems
- **Features**: Full ACID compliance, connection pooling, advanced SQL
- **Configuration**: Host, port, credentials, SSL support
- **Server Management**: Docker-based server lifecycle management for development/testing

## Configuration

DBEngine supports configuration files for different environments with YAML format.

### Environment-Based Configuration

Create configuration files for different environments:

```yaml
databases:
  postgresql:
    host: "prod-db.example.com"
    port: 5432
    database: "myapp_prod"
    user: "prod_user"
    password: "secure_password"  # Set directly in config

logging:
  level: "WARNING"
  handlers: ["file", "syslog"]
```

**Note**: All configuration values must be set in the YAML configuration files.

## Advanced Usage

### PostgreSQL Server Management

DBEngine includes utilities to programmatically start and stop PostgreSQL servers using Docker, making it perfect for development workflows and testing:

```python
from dbengine import PostgreSQLServerManager, PostgreSQLDatabase

# Manual server lifecycle management
server = PostgreSQLServerManager(port=5433, database='my_test_db')
server.start()

try:
    # Create database connection
    db = PostgreSQLDatabase(**server.get_connection_params())

    # Perform database operations
    data = pd.DataFrame({'id': [1, 2, 3], 'value': ['a', 'b', 'c']})
    db.write(table_name='test_table', item=data)

    result = db.query(table_name='test_table')
    print(result)

finally:
    db.close()
    server.stop()

# Context manager (recommended)
with PostgreSQLServerManager(port=5434) as server:
    db = PostgreSQLDatabase(**server.get_connection_params())
    # Server automatically stopped when exiting context
```

#### Server Management Features:
- **Docker Integration**: Automatic container lifecycle management
- **Port Configuration**: Avoid conflicts with existing PostgreSQL instances
- **Custom Databases**: Create servers with specific database names and credentials
- **Health Checks**: Automatic server readiness detection
- **Multiple Servers**: Run multiple isolated PostgreSQL instances simultaneously

## Examples

See the `notebooks/` directory for comprehensive usage examples.

## Testing

Run the comprehensive test suite:

```bash
# Run all tests
pytest

# Run with coverage
pytest --cov=src --cov-report=html
```

**Note**: PostgreSQL server management tests require Docker to be running. Tests will be automatically skipped if Docker is not available.

## Development

### Setup Development Environment

```bash
git clone https://github.com/tomemgouveia/dbengine.git
cd dbengine
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install -e .
```

### Prerequisites

For PostgreSQL server management features, you'll need:
- **Docker**: Required for PostgreSQL server management utilities

```bash
# Install Docker (macOS)
brew install docker

# Install Docker (Ubuntu/Debian)
sudo apt-get update && sudo apt-get install docker.io

# Start Docker daemon
sudo systemctl start docker  # Linux
# or use Docker Desktop on macOS/Windows
```

### Code Quality

The project uses automated code quality tools:

```bash
# Format code
black src/ tests/

# Check imports
isort src/ tests/

# Lint code
flake8 src/ tests/
```

## Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests for new functionality
5. Ensure all tests pass
6. Submit a pull request

## License

MIT License - see [LICENSE](LICENSE) file for details.
