Metadata-Version: 2.4
Name: django-vpg
Version: 0.3.0
Classifier: Development Status :: 4 - Beta
Classifier: Framework :: Django
Classifier: Framework :: Django :: 5.2
Classifier: Framework :: Django :: 6.0
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Rust
Classifier: Topic :: Database
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Dist: django>=5.2
Requires-Dist: psycopg>=3.1
Requires-Dist: django-async-backend>=6.0.5 ; extra == 'async'
Provides-Extra: async
License-File: LICENSE
Summary: Rust-powered PostgreSQL driver and Django database backend.
Author-email: David Burke <david@burkesoftware.com>
Requires-Python: >=3.12
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Bug Tracker, https://gitlab.com/glitchtip/django-vpg/-/issues
Project-URL: Changelog, https://gitlab.com/glitchtip/django-vpg/-/blob/main/CHANGELOG.md
Project-URL: Homepage, https://gitlab.com/glitchtip/django-vpg
Project-URL: Repository, https://gitlab.com/glitchtip/django-vpg

# django-vpg

> **Status: Pre-1.0.** The driver core runs in production as
> [GlitchTip]'s database engine, and this package is maintained by the
> GlitchTip team. Everything below 1.0 is the API sandbox — packaging
> and public surface may still change between minor versions.

A Rust-powered PostgreSQL driver and Django database backend.

django-vpg replaces the connection/cursor IO layer of Django's stock
PostgreSQL backend with a Rust core (tokio-postgres + deadpool, via PyO3):
swap one `ENGINE` line, keep your schema, migrations, and ORM code
unchanged.

Part of the **django-v\*** family — [django-vcache], [django-vtasks],
django-vpg — small, fast, Rust-core Django infrastructure maintained by the
[GlitchTip] team. django-vcache and django-vtasks are battle-tested in
production at glitchtip.com; django-vpg is the newest member. (The *v*
originally stood for Valkey; with three packages it now just marks the
family.)

[django-vcache]: https://gitlab.com/glitchtip/django-vcache
[django-vtasks]: https://gitlab.com/glitchtip/django-vtasks
[GlitchTip]: https://glitchtip.com

## Why

- **CPU and RSS.** Protocol parsing, row decoding, and type marshalling
  run in Rust; only final Python values are materialized — intermediate
  protocol buffers and COPY encoding never traverse the Python heap. In
  the GlitchTip team's A/B benchmarks against psycopg (production-shaped
  ingest workloads, replayed locally), the driver holds resident memory
  flat under heavy-tailed payloads where libpq's buffers ratchet
  (2.5 vs 21 MB growth per 10k events), and wins per-op CPU on jsonb
  reads, pipelined batch inserts, and COPY event batches.
- **`fetch_models` fusion.** Row → model-instance materialization can run
  as a single driver call instead of a per-row Python loop.
- **COPY at native speed.** `COPY FROM STDIN` with batch text-format
  encoding in Rust (`encode_copy_chunk`).
- **psycopg-shaped errors.** Full PEP 249 exception hierarchy with
  `sqlstate` and `Diagnostics`, so existing error-handling code ports
  cleanly.
- **Server-side cancellation.** Cancelled queries fire a PostgreSQL
  `CancelRequest` on a fresh connection and the pooled connection returns
  clean — no poisoned pool state, no abandoned server work.
- **Real pooling.** deadpool checkout/return with warmup, wait timeouts,
  and jittered max-lifetime recycling; prepared-statement cache with
  psycopg3-style admission (`prepare_threshold` / `prepared_max`). Behind a
  transaction-mode PgBouncer without `max_prepared_statements`, disable
  prepared statements the psycopg way — `OPTIONS={"prepare_threshold": None}`.
- **libpq-parity TLS.** `sslmode` (`disable`/`allow`/`prefer`/`require`/
  `verify-ca`/`verify-full`), `sslrootcert`, client certificates.

## Install

```sh
pip install django-vpg            # sync ORM + async driver API
pip install "django-vpg[async]"   # + async ORM via django-async-backend (Django 6+)
```

Wheels cover Linux (manylinux + musllinux, x86_64 + aarch64, CPython
3.12-3.14). Other platforms install from the sdist, which needs a Rust
toolchain at build time.

Note: **psycopg (the library) must be installed** — it is a declared
dependency. Django's stock `postgresql` backend, which django-vpg
subclasses for schema/introspection/operations, imports it at load time.
The IO path never uses it.

## Quickstart

```python
DATABASES = {
    "default": {
        "ENGINE": "django_vpg.backend",
        "NAME": "mydb",
        "USER": "postgres",
        "PASSWORD": "...",
        "HOST": "db.example.com",
        "PORT": "5432",
        # Optional driver OPTIONS (libpq-parity naming; psycopg3-compatible
        # pool dict):
        # "OPTIONS": {
        #     "pool": {"max_size": 20, "min_size": 2},   # default max_size: 10
        #     "sslmode": "verify-full",
        #     "sslrootcert": "/path/ca.pem",
        # },
    }
}
```

Supported `OPTIONS` keys: `pool` (psycopg3-compatible dict — `max_size`,
`min_size`, `timeout`, `max_lifetime`, `max_idle`, `name` — or `False` for
a dedicated single-slot connection per Django connection, e.g. under test
runners), `pool_size` (shorthand for `pool={"max_size": N}`; default 10),
`prepare_threshold` (psycopg3 semantics — int = prepare after N uses,
default 5, `0` = prepare on first use, `None` = disable prepared statements),
`prepared_max`, `isolation_level`, `application_name`, `connect_timeout`,
`keepalives`, `keepalives_idle`, `sslmode`, `sslrootcert`, `sslcert`,
`sslkey`, and `server_settings`. `pgbouncer` (bool) is accepted as a legacy
alias for `prepare_threshold=None`; prefer the psycopg spelling, which — unlike
`pgbouncer` — is a real psycopg key, so the same OPTIONS dict also works on the
stock backend. Unknown keys warn once and are dropped.

## The async story (three layers)

1. **Sync ORM — stock Django, zero extra deps.** Works everywhere Django
   works.
2. **Async driver API — real asyncio, works anywhere.** The raw driver
   (`django_vpg._vpg_driver.RustPgDriver`) exposes `await`-able query,
   execute, transaction, and COPY operations with no dependency on any
   async framework beyond asyncio itself; driver awaitables interoperate
   with `gather`, `wait_for`, `shield`, and cancellation.
   (`django_vpg.dbapi` itself is the *sync* DB-API 2.0 shim the Django
   engine builds on.)
3. **Async ORM — via [django-async-backend].** Installing the
   `django-vpg[async]` extra activates `AsyncDatabaseWrapper`, which plugs
   into the `async_connections` framework. Without it, the backend cleanly
   degrades to sync-only (no ImportError).

Unifying Django's `connections` and `async_connections` (transaction
management and isolation across both worlds) is active upstream work in
django-async-backend that this package tracks closely.

[django-async-backend]: https://github.com/Arfey/django-async-backend

## Pooling and session semantics (psycopg is the reference)

Two rules, both deliberately psycopg's, because porting an app should not
mean re-learning connection behaviour:

- **The backend decides whether a transaction is open — never the driver.**
  Every PostgreSQL response ends with a `ReadyForQuery` carrying the
  session's transaction state; libpq caches it as `PQtransactionStatus` and
  psycopg exposes it as `Connection.info.transaction_status`, which
  django-vpg does too. On release the pool applies psycopg_pool's policy:
  idle sends **nothing**, an open or failed block is rolled back, and a
  connection whose state cannot be established is discarded rather than
  reused. (Inferring this from SQL text instead is not sound — it misreads
  `ROLLBACK TRANSACTION TO SAVEPOINT`, `COMMIT AND CHAIN`, `ABORT`,
  comment-led SQL and PL/pgSQL bodies, and it cannot see statements that
  never went through the driver's own statement path.) A pool hook
  re-checks at checkout, so no return path can hand on a connection that is
  still inside a transaction.
- **Session state is not reset between checkouts.** Same as psycopg_pool,
  which ships no reset by default: GUCs set with `SET`, TEMP tables, session
  advisory locks and `LISTEN` registrations survive into whoever checks the
  connection out next. Reset them yourself if your application needs a clean
  session — and note `DISCARD ALL` is the wrong tool, because it deallocates
  prepared statements the driver's per-connection cache still refers to.

One more difference worth knowing if you use the async ORM: in **async
autocommit** each statement borrows a pooled connection and returns it, so
consecutive autocommit statements may run on different backend sessions —
`pg_backend_pid()` is not stable, and TEMP tables, `SET` GUCs and advisory
locks do not persist between them. The sync path pins one session per
connection and matches psycopg. Transactions pin in both. This is a
deliberate trade for throughput (holding no connection between statements
lets a small pool serve many concurrent tasks), and it is safe rather than
merely fast because of the status checks above. It will become a
configurable choice before 1.0.

## The tokio runtime

The driver runs on a fork-safe shared tokio runtime owned by the package
(1 worker thread by default; `VPG_TOKIO_WORKER_THREADS` overrides). Prefork
servers are safe: the runtime is rebuilt per child on first use.

Rust embedders get the same hook django-vcache offers: link `vpg-pyo3` as
an rlib, re-export its pyclasses into your own pymodule, and install a
process-wide runtime with `vpg_core::set_runtime_provider(my_get_runtime)`
so one tokio pool serves every driver in the process. (This is how
GlitchTip's own Rust extension embeds the driver.) The hook returns
`false` if a runtime had already been built — assert on it at module
init to catch wiring mistakes.

## Known limitations

- **`USE_TZ = True` only (Django's default).** `timestamptz` values are
  returned timezone-aware in UTC. The stock psycopg backend returns
  *naive local* datetimes when `USE_TZ = False`; django-vpg does not
  implement that conversion yet, so a `USE_TZ = False` project will hit
  naive/aware comparison errors on DateTimeFields.
- **One statement per `execute()`.** Multi-statement strings
  (`cursor.execute("SET x = 1; SELECT 1")`) raise a syntax error where
  psycopg3 runs them; the extended query protocol is one statement per
  Parse. Django's schema editor and test-runner paths batch through a
  dedicated internal path and are unaffected — this only bites ported
  raw-SQL code, which should split its statements.

## Status

The driver is the only database engine in [GlitchTip]'s backend (as of
July 2026). Pre-1.0 while the standalone packaging settles; the driver
core predates this package and carries its full test suite (including
an in-process PG wire-server harness) with it.

Known naming wart: the vendored tokio-postgres buffer-cap patch reads the
`GT_PG_BUF_CAP_BYTES` env var (512 KiB default retained-buffer cap); it
will be renamed with a compatibility window before 1.0.

## Development

```sh
./scripts/check.sh        # fmt + clippy -D warnings + cargo test + ruff
                          # (includes tests/wire.rs — an in-process scripted
                          #  PG wire server; no live postgres needed)

# Python suite (needs maturin + a live postgres for the marked tests):
python -m venv .venv && source .venv/bin/activate
pip install maturin pytest pytest-asyncio django psycopg django-async-backend
maturin develop --release
VPG_TEST_DSN=postgres://postgres@localhost:5432/postgres python -m pytest tests/
```

The workspace is two crates: `crates/vpg-core` (pure Rust, no pyo3,
`cargo test`-able) and `crates/vpg-pyo3` (thin bindings, `cdylib + rlib`).
`vendor/tokio-postgres` carries a minimal retained-buffer-capacity patch —
see `vendor/tokio-postgres/GT_PATCH.md` for rationale and the re-audit
procedure on version bumps.

## License

MIT

