Metadata-Version: 2.4
Name: hyperscale-crypto
Version: 0.1.0
Summary: Application-tier encrypted Django fields backed by Tink keysets and AWS KMS
Keywords: django,encryption,tink,kms,fields
Author: Andy Caine
Author-email: Andy Caine <andy@hyperscale.consulting>
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Framework :: Django
Classifier: Framework :: Django :: 5.2
Classifier: Framework :: Django :: 6.1
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Security :: Cryptography
Classifier: Typing :: Typed
Requires-Dist: django>=5.2
Requires-Dist: tink[awskms]>=1.16.0
Requires-Python: >=3.14
Project-URL: Homepage, https://github.com/hyperscale-consulting/hyperscale-crypto
Project-URL: Repository, https://github.com/hyperscale-consulting/hyperscale-crypto
Project-URL: Changelog, https://github.com/hyperscale-consulting/hyperscale-crypto/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/hyperscale-consulting/hyperscale-crypto/issues
Description-Content-Type: text/markdown

# hyperscale-crypto

Application-tier encrypted Django fields backed by Tink keysets and AWS KMS.

Install the `hyperscale-crypto` package; import it as `hyperscale.crypto`.

Values are encrypted in Python before they reach the database. Two Tink
keysets hold the data keys: `aead` (AES256-GCM, randomised) and `daead`
(AES256-SIV, deterministic). Both are stored in your repository, wrapped by a
KMS key, and unwrapped in memory at startup.

## Install

```bash
uv add hyperscale-crypto
```

```python
INSTALLED_APPS = [
    # ...
    "hyperscale.crypto",
]
```

Then run `python manage.py migrate`. The app ships migrations for three
tables: `KeysetState` (the active deterministic key), `KeysetCanary` (the
startup check) and `KeyRotationEvent` (the rotation log).

## Settings

```python
import os
from pathlib import Path

BASE_DIR = Path(__file__).resolve().parent.parent

HYPERSCALE_CRYPTO = {
    "mode": os.environ.get("CRYPTO_MODE", "kms"),  # "kms" or "cleartext"
    "kms_key_uri": os.environ.get("CRYPTO_KMS_KEY_URI", ""),
    "keyset_dir": BASE_DIR / "keysets",  # required
    "reunwrap_seconds": 900,
    "revoke_on_reunwrap_failure": False,
    "retention_days": 35,
    "git_sha": os.environ.get("GIT_SHA", ""),
}
```

| Key | Default | Meaning |
|---|---|---|
| `mode` | `"kms"` | `"kms"` reads KMS-wrapped keysets; `"cleartext"` reads the dev files |
| `kms_key_uri` | `""` | `aws-kms://arn:aws:kms:<region>:<account>:key/<id>`; required in `kms` mode |
| `keyset_dir` | none (required) | directory holding the keyset files |
| `reunwrap_seconds` | `900` | how often a running process re-reads and re-unwraps the keysets |
| `revoke_on_reunwrap_failure` | `False` | drop cached keys when a re-unwrap fails (see [Startup](#startup)) |
| `retention_days` | `35` | minimum days between retiring a key and destroying it |
| `git_sha` | `$GIT_SHA` or `""` | recorded on every `KeyRotationEvent` |

`reunwrap_seconds` and `retention_days` must be positive integers and
`revoke_on_reunwrap_failure` a real `bool` (not the string `"false"`). In
`kms` mode, `kms_key_uri` must start with `aws-kms://` (or `fake-kms://` for
tests). `mode="cleartext"` and `fake-kms://` key URIs are only accepted when
the `ENV` environment variable is `development` or `test`. Any
misconfiguration (missing or unknown keys, a bad value, cleartext outside
those environments) raises `ImproperlyConfigured` during Django setup, not on
first use.

## Keysets

```bash
python manage.py keyset_init              # both keysets; --keyset aead|daead|all
```

In `kms` mode this writes `<keyset_dir>/aead.json` and
`<keyset_dir>/daead.json`, each wrapped by the KMS key. Commit them: the
wrapped files are safe to store, and every deploy ships the keys it needs.
Protect the keyset path with branch protection and signed commits, since
whoever can change these files can change which keys the application trusts.

In `cleartext` mode the files are `<keyset_dir>/dev/aead.json` and
`<keyset_dir>/dev/daead.json`, unwrapped, for development and tests only.

`keyset_init` refuses to overwrite a keyset that already exists.

## Fields

```python
from django.db import models

from hyperscale.crypto import fields as ef


class Customer(models.Model):
    owner_id = models.IntegerField()
    notes = ef.EncryptedTextField(blank=True, default="")
    token = ef.EncryptedCharField(
        max_length=64, context=lambda obj: f"owner:{obj.owner_id}"
    )
    ni_number = ef.DeterministicEncryptedCharField(max_length=9, unique=True)
```

| Field | Underlying | Deterministic variant |
|---|---|---|
| `EncryptedCharField` | `CharField` | `DeterministicEncryptedCharField` |
| `EncryptedTextField` | `TextField` | |
| `EncryptedEmailField` | `EmailField` | `DeterministicEncryptedEmailField` |
| `EncryptedDateField` | `DateField` | |
| `EncryptedDecimalField` | `DecimalField` | |
| `EncryptedIntegerField` | `IntegerField` | `DeterministicEncryptedIntegerField` |
| `EncryptedBooleanField` | `BooleanField` | |

- **Storage.** Every field is a `text` column holding `hc1:` followed by the
  URL-safe base64 Tink ciphertext. `None` and `""` are stored as they are.
  `max_length` and the other validators apply to the plaintext.
- **Reading.** Values decrypt lazily on first attribute access.
  `.values()` and `.values_list()` return the stored value wrapped in the
  `hyperscale.crypto.fields.Ciphertext` marker (a `str` subclass), not the
  plaintext. To copy a stored value unchanged, assign that `Ciphertext`
  instance; a plain `str` is always encrypted.
- **Lookups.** Randomised fields support `isnull` only. Deterministic fields
  support `exact`, `in` and `isnull`; any other lookup or transform raises
  `FieldError`. Ordering is unsupported: it would sort by ciphertext.
  `db_index` is rejected on every encrypted field, and `unique=True` on a
  randomised field (equal values encrypt differently, so the database cannot
  enforce it); use the deterministic variant. `auto_now` and `auto_now_add`
  are rejected on `EncryptedDateField`.
- **Bulk writes.** `QuerySet.update()`, `bulk_update()` and raw saves (such
  as `loaddata`) encrypt plaintext values too. A field with a `context`
  cannot be written that way (there is no instance to compute the context
  from) and raises `FieldError`; save the instances instead. Query
  expressions are refused (`FieldError`) because their result would be
  stored unencrypted or would corrupt the ciphertext; the exceptions are
  `F()` of the same field, `Value(x, output_field=<the field>)` and the
  `Case`/`When` tree `bulk_update()` builds from such values.
- **`context`** (randomised fields only) is a callable from the model instance
  to a string that is bound into the ciphertext. It must be a stable property
  of the row, such as an owner or tenant id. If a context input must change,
  read the field before changing it and save both together. Saving a row
  whose context changed while the field was never read raises
  `DecryptionError` (naming the model, field and pk) instead of storing a
  value that would no longer decrypt. Context fields cannot be read or
  written from a migration (historical models do not carry the callable);
  that raises `FieldError`. Use a management command with the live model.
- **Deterministic fields** require `unique=True` and reject `context`. Every
  deterministic write (`save()` of a new or loaded row, `QuerySet.update()`,
  `bulk_update()`) must run inside a transaction, on every backend; outside
  one it raises `TransactionManagementError`. On PostgreSQL the write also
  takes a share lock on the `KeysetState` row. Set `ATOMIC_REQUESTS = True`
  on the database and wrap background writers in `transaction.atomic()`.

## Rotation

Randomised (`aead`) keys rotate online:

```bash
python manage.py keyset_rotate --keyset aead      # new primary key
# commit aead.json and deploy everywhere
python manage.py keyset_reencrypt                 # --batch-size 500 (>0); resumable
python manage.py keyset_retire aead <old_key_id>  # refuses while rows use it
# after retention_days:
python manage.py keyset_destroy aead <old_key_id>
```

Deterministic (`daead`) keys switch in one step, because lookups must find
every row under a single key:

```bash
python manage.py keyset_add_key --keyset daead    # enabled, not active
# commit daead.json and deploy everywhere
python manage.py keyset_rotate --keyset daead --to <new_key_id> [--yes] [--lock-timeout 30]
python manage.py keyset_retire daead <old_key_id>
# after retention_days:
python manage.py keyset_destroy daead <old_key_id>
```

The daead switch prints the row counts, asks you to type `confirm` (unless
`--yes`), then re-encrypts every deterministic value in one transaction. It
locks the `KeysetState` row for that transaction (on PostgreSQL, the table in
`EXCLUSIVE` mode first, so the switch queues fairly behind in-flight writers
instead of starving), and every deterministic writer waits until it commits.
Reads are not blocked. `--lock-timeout <seconds>` (PostgreSQL) gives up
cleanly, changing nothing, if in-flight writers hold the lock longer than
that. On large tables, run it in a maintenance window.

`keyset_retire` refuses while any row is still encrypted under the key,
counting the startup canary row as one: after an aead rotation run
`keyset_reencrypt` (it moves the canary too) even if no model has data yet.

`keyset_destroy` refuses until `retention_days` have passed since the key was
retired. Set `retention_days` to cover your database backup window: a backup
that still holds values under a destroyed key cannot be decrypted.

Every command records a `KeyRotationEvent` row (`init`, `add_key`, `rotate`,
`reencrypt`, `retire`, `destroy`) with the key id, the actor, details such as
row counts, and `git_sha`. Pass `--actor <name>` to record who ran it; it
defaults to the OS user.

## Evidence

```bash
python manage.py keyset_status          # human-readable
python manage.py keyset_status --json   # machine-readable
```

The JSON has these top-level keys:

- `keysets`: per keyset, every key id with its `status` (`ENABLED`,
  `DISABLED`, `DESTROYED`) and whether it is `primary` and (daead) `active`.
- `rows_by_key`: per keyset, per field (`app.Model.field`), the number of
  rows under each key id, read from ciphertext prefixes without decrypting.
  The canary row is listed too, as
  `hyperscale_crypto.KeysetCanary.aead_value` / `.daead_value`.
- `last_events`: the time of the most recent event of each action, or `null`.
- `canary_ok`, and `canary_error` when the canary check failed.
- `state_matches_file`: whether the database's active daead key is an enabled
  key in the shipped `daead.json`.

## Startup

When the app is ready, it runs a canary check: it decrypts a known value
stored in `KeysetCanary` with both keysets (writing the value on first run)
and raises `ImproperlyConfigured`, naming the keyset, if either keyset
cannot be loaded (a missing file, or one that does not unwrap) or cannot
decrypt it. This
catches a wrong KMS key, a swapped keyset file or a database restored under
different keys before any request is served. The check is skipped for
`makemigrations`, `migrate`, `collectstatic`, `check`, `showmigrations` and
`sqlmigrate`, under pytest, and before the canary table has been migrated.

Startup neither arms the re-unwrap timer nor keeps its database
connection: under a pre-fork server (gunicorn, uWSGI) it runs in the parent,
and neither survives a fork. Instead, the first use of the keys in each
process starts a daemon timer that re-reads and re-unwraps the keysets every
`reunwrap_seconds`, so a revoked KMS grant takes effect without a restart. A
forked worker resets the inherited timer state and starts its own. If a
re-unwrap fails, the default is to keep the cached keys and log the error,
favouring availability during a KMS outage. With
`revoke_on_reunwrap_failure = True` the cached keys are dropped and every
later use must unwrap through KMS again, so revoking KMS access stops
decryption within one interval, at the cost of failing requests while KMS is
unreachable.

## Testing your app

Set `ENV=test` and point the settings at throwaway keysets from
`hyperscale.crypto.testing`:

```python
# conftest.py
import os
from collections.abc import Iterator
from pathlib import Path

import pytest

from hyperscale.crypto import keysets, testing

os.environ.setdefault("ENV", "test")


@pytest.fixture(autouse=True)
def crypto(tmp_path: Path, settings) -> Iterator[None]:
    testing.write_dev_keysets(tmp_path)  # cleartext dev/aead.json, dev/daead.json
    settings.HYPERSCALE_CRYPTO = {"mode": "cleartext", "keyset_dir": tmp_path}
    keysets.reset()
    yield
    keysets.reset()
```

To exercise the KMS code path without AWS, use `uri = testing.fake_kms_uri()`
and `testing.write_kms_keysets(tmp_path, uri)`, with
`{"mode": "kms", "kms_key_uri": uri, "keyset_dir": tmp_path}`.
`keysets.reset()` drops the cached keysets so each test sees its own.

## Security notes

- **Deterministic encryption leaks equality.** Equal plaintexts in the same
  column give equal ciphertexts, so anyone with database access can see which
  rows share a value and how often. On a unique column there is nothing to
  count, which is why deterministic fields require `unique=True`. Use them
  only where you must look rows up by value.
- **Associated data.** Every ciphertext is bound to
  `<app_label>.<model_name>.<field>` (plus `|<context>` when set). A value
  copied to another column, model or context fails to decrypt instead of
  being read in the wrong place.
- **Only `Ciphertext` values pass through.** A value is stored without
  encryption only when it is a `hyperscale.crypto.fields.Ciphertext`, the
  marker that `.values()`, `.values_list()` and unread loaded fields carry.
  Every plain `str`, including one that starts with `hc1:`, is plaintext and
  is encrypted. A ciphertext copied from another row and submitted as text is
  stored encrypted as that text; it never decrypts to the other row's value.
- **The canary** stops a process from serving with keys that cannot read the
  data (see [Startup](#startup)).

## Limitations

- **One database.** Keyset state, the canary and the rotation log live in the
  `default` database, and rotation reads and writes models there; a `using`
  argument or database router is not honoured.
- **PostgreSQL or sqlite.** Other backends are untested. The share and table
  locks that serialise deterministic writes against a daead switch exist
  only on PostgreSQL; on sqlite the database's own write lock serialises
  them.
- **`dumpdata` emits plaintext.** Serialisation reads fields through the
  model, so fixtures and dumps contain decrypted values. Treat them as
  sensitive. `loaddata` re-encrypts on the way in, but cannot load a field
  with a `context` (raw saves have no instance for it).
- **`keyset_status` / `rotation.status()` write the canary** on first run if
  it does not exist yet, like startup does.
- **Where each command runs.** Every command records its `KeyRotationEvent`
  in the database it is pointed at, so run them against the database whose
  evidence you want. `keyset_retire` and `keyset_destroy` read row counts and
  the retire date from the production database and edit the keyset files, so
  they need both the production database and a checkout of the repository
  (commit the changed file afterwards). `keyset_rotate` and `keyset_add_key`
  edit the keyset files too; `keyset_rotate --keyset daead` also rewrites
  production rows.

## Optional: background tasks

`hyperscale.crypto.tasks.reencrypt_aead_task` runs `keyset_reencrypt` as a
background task. It uses Django's built-in tasks framework (`django.tasks`,
Django 6.0 and later) and, on older Django, the
[django-tasks](https://github.com/RealOrangeOne/django-tasks) backport if it
is installed. With neither, the module defines nothing. Configure a task
backend in `TASKS` as usual.

```python
from hyperscale.crypto.tasks import reencrypt_aead_task

reencrypt_aead_task.enqueue(batch_size=500, actor="ops")
```

## Development

Requires [uv](https://docs.astral.sh/uv/) and Python 3.14+.

```bash
uv sync                             # create the venv and install dependencies
uv run pytest                       # run the tests
uv run pre-commit run --all-files   # lint, format and lockfile checks
```

See [CONTRIBUTING.md](CONTRIBUTING.md) for the full set of checks.

## Releasing

Bump `version` in `pyproject.toml` and `__version__` in
`src/hyperscale/crypto/__init__.py`, update `CHANGELOG.md`, then publish a
GitHub release tagged `v<version>`. The `publish` workflow checks the tag
matches the package version, builds, runs `twine check` and publishes to PyPI
with trusted publishing.

Before the first release, register the project on PyPI with this repository
and `publish.yml` as a trusted publisher, and create a `pypi` environment in
the repository settings.

## Design

Design spec: [`docs/superpowers/specs/2026-09-24-encryption-primitive-design.md`](docs/superpowers/specs/2026-09-24-encryption-primitive-design.md)

## License

MIT. See [LICENSE](LICENSE).
