Metadata-Version: 2.4
Name: data-connectors-ai
Version: 0.1.3
Summary: Production-ready data connector libraries for AI applications and data processing pipelines
Author-email: Nagarjun Rajendran <nagarjunr@live.in>
License-Expression: MIT
Project-URL: Homepage, https://github.com/nagarjunr/data-connectors-for-ai-agents
Project-URL: Repository, https://github.com/nagarjunr/data-connectors-for-ai-agents
Project-URL: Issues, https://github.com/nagarjunr/data-connectors-for-ai-agents/issues
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: boto3>=1.26.0
Requires-Dist: google-cloud-storage>=2.10.0
Requires-Dist: psycopg2-binary>=2.9.0
Requires-Dist: sqlalchemy>=2.0.0
Requires-Dist: requests>=2.31.0
Requires-Dist: python-dateutil>=2.8.0
Requires-Dist: pika>=1.3.0
Provides-Extra: s3
Requires-Dist: boto3>=1.26.0; extra == "s3"
Provides-Extra: gcp
Requires-Dist: google-cloud-storage>=2.10.0; extra == "gcp"
Provides-Extra: postgres
Requires-Dist: psycopg2-binary>=2.9.0; extra == "postgres"
Requires-Dist: sqlalchemy>=2.0.0; extra == "postgres"
Provides-Extra: sharepoint
Requires-Dist: requests>=2.31.0; extra == "sharepoint"
Requires-Dist: msal>=1.20.0; extra == "sharepoint"
Provides-Extra: onedrive
Requires-Dist: requests>=2.31.0; extra == "onedrive"
Requires-Dist: msal>=1.20.0; extra == "onedrive"
Provides-Extra: mcp
Requires-Dist: mcp>=1.0.0; extra == "mcp"
Provides-Extra: all
Requires-Dist: boto3>=1.26.0; extra == "all"
Requires-Dist: google-cloud-storage>=2.10.0; extra == "all"
Requires-Dist: psycopg2-binary>=2.9.0; extra == "all"
Requires-Dist: sqlalchemy>=2.0.0; extra == "all"
Requires-Dist: requests>=2.31.0; extra == "all"
Requires-Dist: msal>=1.20.0; extra == "all"
Requires-Dist: pika>=1.3.0; extra == "all"
Requires-Dist: mcp>=1.0.0; extra == "all"
Dynamic: license-file

# Data Connectors

<!-- mcp-name: io.github.nagarjunr/data-connectors-postgres -->
<!-- mcp-name: io.github.nagarjunr/data-connectors-sharepoint -->
<!-- mcp-name: io.github.nagarjunr/data-connectors-onedrive -->

Production-ready, reusable data connector libraries for AI applications and data processing pipelines.

![Python](https://img.shields.io/badge/python-3.12%2B-blue.svg)
![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)

**Key Features:**
- PostgreSQL, S3-compatible, GCP Storage, SharePoint, OneDrive connectors
- Enterprise-ready (proxy support, CA certificates)
- Type-hinted and well-documented
- Explicit configuration injection (no hidden state)
- Safe for batch jobs, CI/CD, containers, Airflow

---

## 🚀 Quick Start (5 Minutes)

### 1. Install Package

```bash
# Local development (editable)
cd /path/to/your-project
pip install -e /path/to/data-connectors

# Production (git SSH)
pip install git+ssh://git@github.com/nagarjunr/data-connectors-for-ai-agents.git@v0.1.0

# Install specific connectors only
pip install -e /path/to/data-connectors[postgres,s3]
```

### 2. Create Connector Module

**File:** `src/connectors/__init__.py`

```python
"""Centralized connector initialization (singleton pattern)."""
from typing import Optional
import os
from dotenv import load_dotenv

# Import connectors you need
from modules.postgres import PostgresClient
from modules.s3_bucket import S3Client
from modules.gcp_bucket import GCPBucketClient
from modules.sharepoint import SharePointClient
from modules.onedrive import OneDriveClient

load_dotenv()

# Singleton instances
_pg_client: Optional[PostgresClient] = None
_s3_client: Optional[S3Client] = None
_gcp_client: Optional[GCPBucketClient] = None
_sharepoint_client: Optional[SharePointClient] = None
_onedrive_client: Optional[OneDriveClient] = None


def get_postgres_client() -> PostgresClient:
    """Get PostgreSQL client."""
    global _pg_client
    if _pg_client is None:
        _pg_client = PostgresClient(
            host=os.getenv("POSTGRES_HOST", "localhost"),
            port=int(os.getenv("POSTGRES_PORT", "5432")),
            database=os.getenv("POSTGRES_DATABASE", "your_database"),
            user=os.getenv("POSTGRES_USER"),
            password=os.getenv("POSTGRES_PASSWORD"),
        )
    return _pg_client


def get_s3_client() -> S3Client:
    """Get S3-compatible client."""
    global _s3_client
    if _s3_client is None:
        _s3_client = S3Client(
            endpoint_url=os.getenv("S3_ENDPOINT"),
            access_key=os.getenv("AWS_ACCESS_KEY_ID"),
            secret_key=os.getenv("AWS_SECRET_ACCESS_KEY"),
            region=os.getenv("AWS_DEFAULT_REGION", "us-east-1"),
        )
    return _s3_client


def get_gcp_client() -> GCPBucketClient:
    """Get GCP Storage client."""
    global _gcp_client
    if _gcp_client is None:
        _gcp_client = GCPBucketClient(
            bucket_name=os.getenv("GCP_BUCKET_NAME"),
        )
    return _gcp_client


def get_sharepoint_client() -> SharePointClient:
    """Get SharePoint client."""
    global _sharepoint_client
    if _sharepoint_client is None:
        _sharepoint_client = SharePointClient(
            site_id=os.getenv("SITE_ID"),
            client_id=os.getenv("CLIENT_ID"),
            client_secret=os.getenv("CLIENT_SECRET"),
            tenant_id=os.getenv("TENANT_ID"),
        )
    return _sharepoint_client


def get_onedrive_client() -> OneDriveClient:
    """Get OneDrive client."""
    global _onedrive_client
    if _onedrive_client is None:
        _onedrive_client = OneDriveClient(
            client_id=os.getenv("CLIENT_ID"),
            client_secret=os.getenv("CLIENT_SECRET"),
            tenant_id=os.getenv("TENANT_ID"),
            directory=os.getenv("ONEDRIVE_DIRECTORY"),  # Optional
        )
    return _onedrive_client


def close_all_connections():
    """Close all connections on shutdown."""
    global _pg_client, _s3_client, _gcp_client, _sharepoint_client, _onedrive_client
    for client in [_pg_client, _s3_client, _gcp_client, _sharepoint_client, _onedrive_client]:
        if client:
            client.close()
```

### 3. Add Environment Variables

**File:** `.env`

```bash
# PostgreSQL
POSTGRES_HOST=localhost
POSTGRES_PORT=5432
POSTGRES_DATABASE=your_database
POSTGRES_USER=your_user
POSTGRES_PASSWORD=your_password

# S3-Compatible Object Storage
AWS_ACCESS_KEY_ID=your_access_key
AWS_SECRET_ACCESS_KEY=your_secret_key
S3_ENDPOINT=https://your-s3-endpoint.example.com:9021
S3_NAMESPACE=your-namespace  # Optional, only needed for some S3-compatible providers
AWS_DEFAULT_REGION=us-east-1

# GCP Storage
GCP_BUCKET_NAME=your-gcp-bucket-name

# Azure AD / Microsoft Graph (for SharePoint & OneDrive)
CLIENT_ID=your_client_id
CLIENT_SECRET=your_client_secret
TENANT_ID=your-tenant-id

# SharePoint Specific
SITE_ID=your-site-id

# OneDrive Specific
ONEDRIVE_DIRECTORY=Documents  # Optional: default directory
```

### 4. Use in Your Code

```python
from src.connectors import (
    get_postgres_client,
    get_s3_client,
    get_gcp_client,
    get_sharepoint_client,
    get_onedrive_client,
    close_all_connections,
)

# PostgreSQL
pg = get_postgres_client()
results = pg.query("SELECT * FROM users LIMIT 10")

# S3-compatible
s3 = get_s3_client()
s3.upload_file("local.txt", "bucket-name", "remote/path/file.txt")
files = s3.list_objects("bucket-name", prefix="documents/")

# GCP Storage
gcp = get_gcp_client()
gcp.upload("/tmp/report.pdf", "reports/monthly.pdf")

# SharePoint
sp = get_sharepoint_client()
sp.download("Shared Documents/file.pdf", "/tmp/file.pdf")

# OneDrive
od = get_onedrive_client()
od.list("Documents")

# Cleanup on shutdown (FastAPI example)
# @app.on_event("shutdown")
# async def shutdown():
#     close_all_connections()
```

---

## 📦 Available Connectors

| Connector | Purpose | Environment Variables | Documentation |
|-----------|---------|----------------------|---------------|
| **PostgreSQL** | Database connectivity with SQLAlchemy ORM | `POSTGRES_HOST`, `POSTGRES_PORT`, `POSTGRES_DATABASE`, `POSTGRES_USER`, `POSTGRES_PASSWORD` | [README](modules/postgres/README.md) |
| **S3 Bucket** | S3-compatible object storage (MinIO, AWS, etc.) | `S3_ENDPOINT`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_DEFAULT_REGION` | [README](modules/s3_bucket/README.md) |
| **GCP Storage** | Google Cloud Storage | `GCP_BUCKET_NAME` | [README](modules/gcp_bucket/README.md) |
| **SharePoint** | SharePoint document libraries via Microsoft Graph | `CLIENT_ID`, `CLIENT_SECRET`, `TENANT_ID`, `SITE_ID` | [README](modules/sharepoint/README.md) |
| **OneDrive** | OneDrive file storage via Microsoft Graph | `CLIENT_ID`, `CLIENT_SECRET`, `TENANT_ID`, `SITE_ID`, `ONEDRIVE_DIRECTORY` | [README](modules/onedrive/README.md) |
| **RabbitMQ** | Send/receive with a durable quorum queue, auto-reconnect | `CONNECTION_STRING` or `RABBITMQ_HOST`, `RABBITMQ_PORT`, `RABBITMQ_USERNAME`, `RABBITMQ_PASSWORD` | [README](modules/rabbitmq/README.md) |

---

## 📚 Documentation

### For Users
- **[Deployment Guide](docs/DEPLOYMENT.md)** - Complete deployment guide from local dev to OpenShift/Kubernetes

### For Contributors
- **[Development Guide](docs/DEVELOPMENT.md)** - Development setup, code quality standards, and contributing guidelines

### Quick Development Commands

```bash
# Format code
make format

# Lint and auto-fix
make lint-fix

# Type check
make typecheck

# Run all checks
make check
```

---

## 🎯 Common Use Cases

### Use Case 1: Load Documents from S3

```python
from src.connectors import get_s3_client
from pathlib import Path

def load_documents():
    s3 = get_s3_client()
    bucket = os.getenv("S3_BUCKET_NAME")
    files = s3.list_objects(bucket, prefix="documents/")

    documents = []
    for file_key in files:
        local_path = f"/tmp/{Path(file_key).name}"
        s3.download_file(bucket, file_key, local_path)
        with open(local_path, 'r') as f:
            documents.append(f.read())

    return documents
```

### Use Case 2: Save Results to PostgreSQL

```python
from src.connectors import get_postgres_client

def save_result(data: dict):
    pg = get_postgres_client()

    query = """
        INSERT INTO results (category, answer, status)
        VALUES (%s, %s, %s)
        RETURNING id
    """
    result_id = pg.execute(query, (data['category'], data['answer'], data['status']))
    return result_id
```

### Use Case 3: Backup to GCP

```python
from src.connectors import get_gcp_client
from pathlib import Path

def backup_files(source_dir: str):
    gcp = get_gcp_client()

    for file_path in Path(source_dir).rglob("*"):
        if file_path.is_file():
            gcp_path = f"backups/{file_path.name}"
            gcp.upload(str(file_path), gcp_path)
```

---

## 🏗️ Repository Structure

```
data-connectors/
├── README.md              # This file - Quick start and overview
├── docs/
│   ├── INTEGRATION.md     # Integration guide for using connectors
│   ├── DEPLOYMENT.md      # Complete deployment guide (local to OpenShift)
│   └── DEVELOPMENT.md     # Development setup and guidelines
├── pyproject.toml         # Python project configuration
├── requirements-dev.txt   # Development dependencies
├── Makefile               # Development commands (format, lint, typecheck)
├── examples/              # Runnable reference examples
│   ├── s3_bucket/
│   ├── sharepoint/
│   ├── onedrive/
│   ├── gcp_bucket/
│   └── postgres/
└── modules/
    ├── _common/           # Shared base classes and auth helpers
    ├── s3_bucket/           # S3-compatible connector
    ├── sharepoint/       # SharePoint connector
    ├── onedrive/         # OneDrive connector
    ├── gcp_bucket/       # GCP bucket connector
    └── postgres/         # PostgreSQL connector
```

---

## 🔐 OpenShift Quick Setup

**Note:** Deploy keys may be disabled depending on your Git host policy. Use Personal Access Tokens instead.

### Step 1: Create GitHub Personal Access Token

1. Go to: [GitHub Settings - Personal Access Tokens](https://github.com/settings/tokens)
2. Click **"Generate new token (classic)"**
3. Select `repo` scope (Full control of private repositories)
4. Set expiration (90 days recommended)
5. **Copy token immediately** - you won't see it again

### Step 2: Create Secret in OpenShift

```bash
oc login --server=https://api.your-openshift-cluster.example.com:6443 --token=<your-token>
oc project your-project

oc create secret generic data-connectors-github-pat \
  --from-literal=username=<your-github-username> \
  --from-literal=password=ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxx \
  --type=kubernetes.io/basic-auth

oc annotate secret data-connectors-github-pat \
  'build.openshift.io/source-secret-match-uri-1=https://github.com/*'
```

For complete OpenShift deployment instructions, see [Deployment Guide](docs/DEPLOYMENT.md#production-deployment-openshift).

---

## 🐛 Troubleshooting

### Import Error
```bash
# Problem: ModuleNotFoundError: No module named 'modules'
# Solution:
pip install -e /path/to/data-connectors
pip list | grep data-connectors
```

### Connection Error
```bash
# Problem: Connection refused to PostgreSQL/S3
# Solution:
# 1. Check .env variables
# 2. Verify services are running
# 3. Test connection:
psql -h localhost -U user -d db_name
```

### SSL Certificate Error
```bash
# Problem: SSLError: certificate verify failed
# Solution:
export SSL_CERT_FILE=/path/to/ca-bundle.pem
export REQUESTS_CA_BUNDLE=$SSL_CERT_FILE
```

For more troubleshooting help, see the [Deployment Guide](docs/DEPLOYMENT.md#troubleshooting).

---

## 🎓 Design Principles

- ✅ **Explicit configuration**: All settings injected via constructor parameters
- ✅ **No hidden state**: No environment variable loading inside modules
- ✅ **Deterministic behavior**: Same input = same output, always
- ✅ **Easy testing**: Pure functions, dependency injection, no side effects
- ✅ **Production-ready**: Designed for batch jobs, CI/CD, containers, Airflow
- ✅ **Type-safe**: Full type hints for better IDE support and error detection

---

## 🆘 Getting Help

- **Issues:** [GitHub Issues](https://github.com/nagarjunr/data-connectors-for-ai-agents/issues)
- **Integration Guide:** [docs/INTEGRATION.md](docs/INTEGRATION.md)
- **Deployment Guide:** [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md)
- **Development Guide:** [docs/DEVELOPMENT.md](docs/DEVELOPMENT.md)

---

## 📋 Checklist for New Projects

- [ ] Install data-connectors package
- [ ] Create `src/connectors/__init__.py`
- [ ] Add environment variables to `.env`
- [ ] Test connections locally
- [ ] Add to `requirements.txt` / `requirements.prod.txt`
- [ ] For OpenShift: Create PAT and secrets
- [ ] Test deployment in staging
- [ ] Deploy to production

---

## License

MIT — see [LICENSE](LICENSE).
