Metadata-Version: 2.4
Name: django-connectors
Version: 0.1.0
Summary: A Django framework for connecting applications to third-party systems, with support for authentication, polling, webhooks, incremental sync, and data ingestion via dlt.
License-Expression: MIT
Project-URL: Homepage, https://github.com/gaussian/django-connectors
Project-URL: Issues, https://github.com/gaussian/django-connectors/issues
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: Django>=6.0
Requires-Dist: dlt[sqlalchemy]>=1.30
Provides-Extra: mysql
Requires-Dist: PyMySQL>=1.1; extra == "mysql"
Provides-Extra: postgres
Requires-Dist: psycopg2-binary>=2.9; extra == "postgres"
Provides-Extra: drf
Requires-Dist: djangorestframework>=3.15; extra == "drf"
Provides-Extra: celery
Requires-Dist: celery>=5.4; extra == "celery"
Provides-Extra: allauth
Requires-Dist: django-allauth>=65.0; extra == "allauth"
Provides-Extra: secrets
Requires-Dist: cryptography>=43.0; extra == "secrets"
Provides-Extra: sql
Requires-Dist: dlt[sql-database]>=1.30; extra == "sql"
Provides-Extra: csv
Requires-Dist: pandas>=2.2; extra == "csv"
Provides-Extra: parquet
Requires-Dist: dlt[parquet]>=1.30; extra == "parquet"
Provides-Extra: s3
Requires-Dist: dlt[s3]>=1.30; extra == "s3"
Provides-Extra: gs
Requires-Dist: dlt[gs]>=1.30; extra == "gs"
Provides-Extra: az
Requires-Dist: dlt[az]>=1.30; extra == "az"
Provides-Extra: google
Requires-Dist: google-auth>=2.35; extra == "google"
Provides-Extra: microsoft
Requires-Dist: azure-identity>=1.19; extra == "microsoft"
Requires-Dist: openpyxl>=3.1; extra == "microsoft"
Dynamic: license-file

# django-connectors

Connect Django applications to third-party APIs, SaaS platforms, files, databases
and warehouses — with pluggable authentication, [dlt](https://dlthub.com/)-powered
synchronization, webhooks, and customer-configurable data projections.

> **Status: v0.1.** The service layer is the intended integration surface. The
> DRF API and the provider connectors are provisional and may change.

## What it does

```
  external system  →  Connection / Binding  →  dlt  →  landing tables
                                                            ↓
                                                       Projection
                                                            ↓
                                              your TargetDefinition + writer
                                                            ↓
                                                     your Django models
```

Source data lands first, in a source-shaped form, and is only then mapped into
shapes your application declares. That boundary is what lets a mapping change be
replayed without re-fetching from the provider, and a failed write to your models
be retried without touching the provider at all.

**The library never imports your models.** You declare a *shape* and a function
that persists it; it never learns what that function does.

## Install

```bash
pip install django-connectors
```

```python
INSTALLED_APPS = [
    "django.contrib.contenttypes",  # required: Connection.owner is a GenericForeignKey
    ...
    "django_connectors",
]

DJANGO_CONNECTORS = {
    # A SQLAlchemy DSN, NOT a Django DATABASES alias — it is reached only
    # through dlt, which makes routing an ORM model there impossible.
    # MySQL and PostgreSQL are both supported and both covered by CI.
    "LANDING_URL": "postgresql+psycopg2://user:pw@host:5432/connectors_landing",
    "SOURCES": {"rest": "django_connectors.sources.rest.RestSource"},
}
```

Extras: `mysql`, `postgres`, `drf`, `celery`, `allauth`, `secrets`, `sql`, `csv`,
`parquet`, `s3`, `gs`, `az`, `google`, `microsoft`. Installing one never enables behaviour by itself —
the corresponding source or backend must also be named in the setting.

## Declare a target

In any installed app's `connectors.py` (auto-discovered, like `admin.py`):

```python
from django_connectors import (
    DateTimeField, JSONField, StringField, TargetDefinition, register_target,
)

def event_writer(records, context):
    """Must be idempotent per identity: a failed batch retries the whole run."""
    for record in records:
        if record.operation == "delete":
            Event.objects.filter(external_id=record.identity["external_id"]).delete()
            continue
        Event.objects.update_or_create(
            team_id=context.owner_object_id,
            external_id=record.identity["external_id"],
            defaults=record.values,
        )
    return len(records)

register_target(TargetDefinition(
    key="events",
    fields={
        "external_id": StringField(required=True),
        "occurred_at": DateTimeField(required=True),
        "type": StringField(required=True),
        "payload": JSONField(),
    },
    identity_fields=("external_id",),
    identity_scope="owner",   # required: decides whether two tenants may collide
    writer=event_writer,
))
```

Your customers then map landed columns onto those fields declaratively — no
Python — and the library validates, previews and executes the mapping.

## Try it

`example/` is a runnable Django project demonstrating the whole flow with no
credentials required:

```bash
cd example
python manage.py migrate
python manage.py demo
```

## Sources

Built in: `memory` (a test driver with injectable failure modes), `rest`
(config-driven, over `dlt.sources.rest_api`), `sql` (warehouses and databases),
`filesystem` (JSONL/CSV/Parquet on local disk, S3, GCS or Azure — one file, a
prefix, or a recursive glob). Provider connectors for Gmail, Google Sheets,
Google Drive, Microsoft/Entra files and Excel, and Salesforce ship under
`django_connectors.providers` — see their module docstrings for what is and is
not verified against a live provider.

Writing your own means subclassing `SourceDefinition` and returning a dlt source.

## Documentation

- [docs/quickstart.md](docs/quickstart.md) — end to end in ten minutes
- [docs/architecture.md](docs/architecture.md) — why the pieces are shaped as they are
- [docs/operations.md](docs/operations.md) — deploying on MySQL or PostgreSQL, concurrency, retention
- [docs/api.md](docs/api.md) — the optional REST API, and composing it with your own
- [docs/TESTING.md](docs/TESTING.md) — the conformance suite every source must pass, and how to test a connector against the real thing
- [AGENTS.md](AGENTS.md) — development workflow and test tiers

## Development

```bash
uv sync --all-extras
uv run --all-extras pytest
uv run --all-extras ruff check django_connectors/ tests/ example/
uv run --all-extras ruff format django_connectors/ tests/ example/
```

Test tiers: default (sqlite, no docker), minimal (no extras installed),
server-backed (MySQL **and** PostgreSQL), and an example-project smoke test. See
[AGENTS.md](AGENTS.md); the server tier is not optional polish — it covers
data-loss and portability failures that are invisible on sqlite.

`develop` is the working branch; releases flow `develop` → `main` and publish to
PyPI automatically.

## License

MIT — see [LICENSE](LICENSE).
