Metadata-Version: 2.4
Name: reproq-django
Version: 0.1.533
Summary: Django 6.0 Tasks backend for reproq-worker (Postgres queue + deterministic runner)
Author-email: Alejandro Peña <adpena@gmail.com>
License:                                  Apache License
                                   Version 2.0, January 2004
                                http://www.apache.org/licenses/
        
           TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
        
           1. Definitions.
        
              "License" shall mean the terms and conditions for use, reproduction,
              and distribution as defined by Sections 1 through 9 of this document.
        
              "Licensor" shall mean the copyright owner or entity authorized by
              the copyright owner that is granting the License.
        
              "Legal Entity" shall mean the union of the acting entity and all
              other entities that control, are controlled by, or are under common
              control with that entity. For the purposes of this definition,
              "control" means (i) the power, direct or indirect, to cause the
              direction or management of such entity, whether by contract or
              otherwise, or (ii) ownership of fifty percent (50%) or more of the
              outstanding shares, or (iii) beneficial ownership of such entity.
        
              "You" (or "Your") shall mean an individual or Legal Entity
              exercising permissions granted by this License.
        
              "Source" form shall mean the preferred form for making modifications,
              including but not limited to software source code, documentation
              source, and configuration files.
        
              "Object" form shall mean any form resulting from mechanical
              transformation or translation of a Source form, including but
              not limited to compiled object code, generated documentation,
              and conversions to other media types.
        
              "Work" shall mean the work of authorship, whether in Source or
              Object form, made available under the License, as indicated by a
              copyright notice that is included in or attached to the work
              (an example is provided in the Appendix below).
        
              "Derivative Works" shall mean any work, whether in Source or Object
              form, that is based on (or derived from) the Work and for which the
              editorial revisions, annotations, elaborations, or other modifications
              represent, as a whole, an original work of authorship. For the purposes
              of this License, Derivative Works shall not include works that remain
              separable from, or merely link (or bind by name) to the interfaces of,
              the Work and Derivative Works thereof.
        
              "Contribution" shall mean any work of authorship, including
              the original version of the Work and any modifications or additions
              to that Work or Derivative Works thereof, that is intentionally
              submitted to Licensor for inclusion in the Work by the copyright owner
              or by an individual or Legal Entity authorized to submit on behalf of
              the copyright owner. For the purposes of this definition, "submitted"
              means any form of electronic, verbal, or written communication sent
              to the Licensor or its representatives, including but not limited to
              communication on electronic mailing lists, source code control systems,
              and issue tracking systems that are managed by, or on behalf of, the
              Licensor for the purpose of discussing and improving the Work, but
              excluding communication that is conspicuously marked or otherwise
              designated in writing by the copyright owner as "Not a Contribution."
        
              "Contributor" shall mean Licensor and any individual or Legal Entity
              on behalf of whom a Contribution has been received by Licensor and
              subsequently incorporated within the Work.
        
           2. Grant of Copyright License. Subject to the terms and conditions of
              this License, each Contributor hereby grants to You a perpetual,
              worldwide, non-exclusive, no-charge, royalty-free, irrevocable
              copyright license to reproduce, prepare Derivative Works of,
              publicly display, publicly perform, sublicense, and distribute the
              Work and such Derivative Works in Source or Object form.
        
           3. Grant of Patent License. Subject to the terms and conditions of
              this License, each Contributor hereby grants to You a perpetual,
              worldwide, non-exclusive, no-charge, royalty-free, irrevocable
              (except as stated in this section) patent license to make, have made,
              use, offer to sell, sell, import, and otherwise transfer the Work,
              where such license applies only to those patent claims licensable
              by such Contributor that are necessarily infringed by their
              Contribution(s) alone or by combination of their Contribution(s)
              with the Work to which such Contribution(s) was submitted. If You
              institute patent litigation against any entity (including a
              cross-claim or counterclaim in a lawsuit) alleging that the Work
              or a Contribution incorporated within the Work constitutes direct
              or contributory patent infringement, then any patent licenses
              granted to You under this License for that Work shall terminate
              as of the date such litigation is filed.
        
           4. Redistribution. You may reproduce and distribute copies of the
              Work or Derivative Works thereof in any medium, with or without
              modifications, and in Source or Object form, provided that You
              meet the following conditions:
        
              (a) You must give any other recipients of the Work or
                  Derivative Works a copy of this License; and
        
              (b) You must cause any modified files to carry prominent notices
                  stating that You changed the files; and
        
              (c) You must retain, in the Source form of any Derivative Works
                  that You distribute, all copyright, patent, trademark, and
                  attribution notices from the Source form of the Work,
                  excluding those notices that do not pertain to any part of
                  the Derivative Works; and
        
              (d) If the Work includes a "NOTICE" text file as part of its
                  distribution, then any Derivative Works that You distribute must
                  include a readable copy of the attribution notices contained
                  within such NOTICE file, excluding those notices that do not
                  pertain to any part of the Derivative Works, in at least one
                  of the following places: within a NOTICE text file distributed
                  as part of the Derivative Works; within the Source form or
                  documentation, if provided along with the Derivative Works; or,
                  within a display generated by the Derivative Works, if and
                  wherever such third-party notices normally appear. The contents
                  of the NOTICE file are for informational purposes only and
                  do not modify the License. You may add Your own attribution
                  notices within Derivative Works that You distribute, alongside
                  or as an addendum to the NOTICE text from the Work, provided
                  that such additional attribution notices cannot be construed
                  as modifying the License.
        
              You may add Your own copyright statement to Your modifications and
              may provide additional or different license terms and conditions
              for use, reproduction, or distribution of Your modifications, or
              for any such Derivative Works as a whole, provided Your use,
              reproduction, and distribution of the Work otherwise complies with
              the conditions stated in this License.
        
           5. Submission of Contributions. Unless You explicitly state otherwise,
              any Contribution intentionally submitted for inclusion in the Work
              by You to the Licensor shall be under the terms and conditions of
              this License, without any additional terms or conditions.
              Notwithstanding the above, nothing herein shall supersede or modify
              the terms of any separate license agreement you may have executed
              with Licensor regarding such Contributions.
        
           6. Trademarks. This License does not grant permission to use the trade
              names, trademarks, service marks, or product names of the Licensor,
              except as required for reasonable and customary use in describing the
              origin of the Work and reproducing the content of the NOTICE file.
        
           7. Disclaimer of Warranty. Unless required by applicable law or
              agreed to in writing, Licensor provides the Work (and each
              Contributor provides its Contributions) on an "AS IS" BASIS,
              WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
              implied, including, without limitation, any warranties or conditions
              of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
              PARTICULAR PURPOSE. You are solely responsible for determining the
              appropriateness of using or redistributing the Work and assume any
              risks associated with Your exercise of permissions under this License.
        
           8. Limitation of Liability. In no event and under no legal theory,
              whether in tort (including negligence), contract, or otherwise,
              unless required by applicable law (such as deliberate and grossly
              negligent acts) or agreed to in writing, shall any Contributor be
              liable to You for damages, including any direct, indirect, special,
              incidental, or consequential damages of any character arising as a
              result of this License or out of the use or inability to use the
              Work (including but not limited to damages for loss of goodwill,
              work stoppage, computer failure or malfunction, or any and all
              other commercial damages or losses), even if such Contributor
              has been advised of the possibility of such damages.
        
           9. Accepting Warranty or Additional Liability. While redistributing
              the Work or Derivative Works thereof, You may choose to offer,
              and charge a fee for, acceptance of support, warranty, indemnity,
              or other liability obligations and/or rights consistent with this
              License. However, in accepting such obligations, You may act only
              on Your own behalf and on Your sole responsibility, not on behalf
              of any other Contributor, and only if You agree to indemnify,
              defend, and hold each Contributor harmless for any liability
              incurred by, or claims asserted against, such Contributor by reason
              of your accepting any such warranty or additional liability.
        
           END OF TERMS AND CONDITIONS
        
           APPENDIX: How to apply the Apache License to your work.
        
              To apply the Apache License to your work, attach the following
              boilerplate notice, with the fields enclosed by brackets "[]"
              replaced with your own identifying information. (Don't include
              the brackets!)  The text should be enclosed in the appropriate
              comment syntax for the file format. We also recommend that a
              file or class name and description of purpose be included on the
              same "printed page" as the copyright notice for easier
              identification within third-party archives.
        
           Copyright [yyyy] [name of copyright owner]
        
           Licensed under the Apache License, Version 2.0 (the "License");
           you may not use this file except in compliance with the License.
           You may obtain a copy of the License at
        
               http://www.apache.org/licenses/LICENSE-2.0
        
           Unless required by applicable law or agreed to in writing, software
           distributed under the License is distributed on an "AS IS" BASIS,
           WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
           See the License for the specific language governing permissions and
           limitations under the License.
        
Classifier: Framework :: Django
Classifier: Framework :: Django :: 6.0
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: 3.15
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: Django!=6.1.0,!=6.1.1,!=6.1.2,<7.0,>=6.0.10; python_version < "3.15"
Requires-Dist: Django<7.0,>=6.1.3; python_version >= "3.15"
Requires-Dist: croniter>=2.0.5
Requires-Dist: psycopg[binary]>=3.1.0; python_version < "3.15"
Requires-Dist: psycopg[binary]>=3.3.6; python_version >= "3.15"
Requires-Dist: sqlparse>=0.6.0
Provides-Extra: dev
Requires-Dist: pytest>=9.0.3; extra == "dev"
Requires-Dist: pygments>=2.20.0; extra == "dev"
Requires-Dist: pytest-django>=4.9.0; extra == "dev"
Requires-Dist: ruff>=0.8.4; extra == "dev"
Dynamic: license-file

# Reproq Django 🚀

[![License: Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://www.apache.org/licenses/LICENSE-2.0)
[![Documentation](https://img.shields.io/badge/Docs-View%20Online-indigo)](https://adpena.github.io/reproq-django/)

**Deterministic Background Tasks for Django 6.0+ powered by Go.**

Reproq is a production-grade tasks backend that combines the ease of Django with the performance and reliability of the [Reproq Worker](https://github.com/adpena/reproq-worker), a high-performance execution engine written in Go.

---

## 🤝 Relationship with Reproq Worker

Reproq is split into two specialized components:
1. **Reproq Django (this repo)**: Task definition, enqueueing, results, and the Admin dashboard.
2. **[Reproq Worker](https://github.com/adpena/reproq-worker)**: A Go binary that claims and executes tasks from Postgres.

## Key Features

- **Postgres-Only**: Uses `SKIP LOCKED` for high-performance claiming.
- **Deterministic**: Each task has a `spec_hash` for deduplication.
- **Django Native**: Implements the Django 6.0 Tasks API.
- **Periodic Tasks**: Built-in scheduler with Django models.
- **Monitoring**: TaskRuns + Workers in Django Admin.

## ✅ Compatibility

| Component | Supported |
| :--- | :--- |
| Django | Python 3.12–3.14 supports Django 6.0.10 or later in the 6.0 series, or Django 6.1.3 or later in 6.x. Python 3.15 requires Django 6.1.3 or later in 6.x. |
| Python | 3.12–3.15, checked locally with isolated locked environments |
| Reproq Worker | Release tested with this SDK (install/upgrade via `uv run python manage.py reproq install` / `upgrade`) |

Use `uv run python manage.py reproq doctor` to verify schema, worker binary, and DSN alignment.
The Go process reads its queue connection from `REPROQ_DATABASE_URL` or explicit
configuration/flags. Generic `DATABASE_URL` remains Django's business connection;
the SDK derives queue connections from the selected Django alias and preserves
that business environment for task execution. Queue names must route to the
selected alias through `REPROQ_QUEUE_DATABASES`.

The [package requirements](pyproject.toml) exclude the affected Django
versions in the [November 3 security advisory](https://www.djangoproject.com/weblog/2026/nov/03/security-releases/).
The locked Python checks use Django 6.1.3, which added
[official Python 3.15 support](https://docs.djangoproject.com/en/6.1/releases/6.1.3/),
and `psycopg[binary]` 3.3.6.
Package builds require `setuptools>=84.1.0` and `wheel>=0.46.2`.

---

## ⚡ Quickstart

[SDK 0.1.533](https://pypi.org/project/reproq-django/0.1.533/) requires
[Reproq Worker v0.0.145](https://github.com/adpena/reproq-worker/releases/tag/v0.0.145)
with `worker_python_bin:1` and
the migration-owned schema-v2/executor-protocol2 contract. Worker v0.0.144 does
not satisfy this contract. Upgrade the SDK, worker and queue migrations as a
coordinated change; the historical-schema adoption and draining requirements
below still apply. Package publication does not qualify a production cutover.

### 1. Install
Install SDK 0.1.533 from PyPI:

```bash
uv add 'reproq-django==0.1.533'
```

Alternatively, use the version-pinned GitHub release tag:

```bash
uv add "reproq-django @ git+https://github.com/adpena/reproq-django.git@v0.1.533"
```

For reviewed paired development, add the local SDK instead:

```bash
uv add --editable /path/to/reviewed/reproq-django
```

### 2. Configure Settings
Add `reproq_django` to your `INSTALLED_APPS` and configure the `TASKS` backend:

```python
INSTALLED_APPS = [
    ...,
    "reproq_django",
]

TASKS = {
    "default": {
        "BACKEND": "reproq_django.backend.ReproqBackend",
        "OPTIONS": {
            "DEDUP_ACTIVE": True,
            "TIMEOUT_SECONDS": 900,
            "MAX_ATTEMPTS": 3,
        }
    }
}
```

### 3. Define a Task
Use the Django 6.0 Tasks API with a `@task` decorator:

```python
from django.tasks import task

@task(queue_name="default", priority=0)
def send_welcome_email(user_id: int) -> str:
    # business logic here
    return f"Email sent to {user_id}"
```

### 4. Bootstrap (Recommended)
Bootstrap writes a config file, installs the worker binary, and applies committed
Django migrations. This package's migrations own the queue schema.

```bash
uv run python manage.py reproq init
```

For reviewed paired development, select the local v0.0.145 worker and skip
the hosted download:

```bash
export REPROQ_WORKER_BIN=/absolute/path/to/reviewed/reproq
uv run python manage.py reproq init --skip-install
```
Use `--format toml` or `--skip-install` as needed. `--skip-migrate` skips the
project-wide invocation; `--skip-worker-migrate` skips the separate ReproQ-only
invocation. Both invocations use the same committed Django migrations. Use both
flags when migration execution must be deferred.

### 5. Download the Compatible Worker
SDK 0.1.533 selects the compatible v0.0.145 tag by default:

```bash
uv run python manage.py reproq install
```
The installer detects the OS/architecture and fetches that tag's pre-built
binary from GitHub. Reviewed paired development can use the local selector above.

Downloads must match the publisher's checksum sidecar or GitHub release-asset
SHA-256 digest before the binary is made executable or its version is queried.
For reproducible deployment, pin both a release tag and the digest of the
binary for the target OS and architecture. For v0.0.145, use the reviewed digest
of that release's binary for the selected platform:

~~~bash
uv run python manage.py reproq install --tag v0.0.145 --sha256 <platform-binary-sha256>
~~~

The same options work with init and upgrade. An explicit digest disables
download-to-source fallback. Missing or mismatched integrity metadata stops
installation and preserves the existing binary; it never triggers a source
build. Explicit --build still supports local development and can also verify
an expected digest. Successful installation replaces the binary atomically.

Run the complete locked local lint and test suite with make check. The optional
hosted CI workflow is manual and calls the same command.
Release builds are also manual: they require an existing reviewed tag matching
both package version declarations, run the complete locked checks, and create
a draft release. Existing releases and their assets are never overwritten by
that workflow. Local builds use uv build after make check; publishing and tag
retention remain separate reviewed actions.

### 6. Run Migrations
```bash
uv run python manage.py migrate
```

The historical `reproq migrate-worker --database <alias>` command now delegates to
the same Django migrations for ReproQ. It does not execute another SQL schema or
repair tables. The standalone Go worker requires the supported migration-owned
schema contract before claiming tasks or scheduling work.

Stop legacy workers and drain all `RUNNING` tasks before the forward contract
migration. Its validated lease constraint makes old claim/completion SQL fail
closed. The migration preserves historical task JSON and hashes, adds nullable
canonical-spec fields with `spec_version=0`, and leaves canonical schedule
reconciliation to an explicit later operation. Schema rollback requires a
verified database backup; this migration deliberately refuses a destructive
reverse operation.

Existing databases with legacy Go-created tables or a partially applied Django
squash require explicit adoption before ordinary migration. Inspect the selected
database first:

```bash
uv run python manage.py reproq adopt-schema --database <alias> --inspect
```

Adoption accepts only checked-in semantic catalog profiles and exact supported
historical ledger sets. It checks column types/nullability/defaults, constraints
and validation state, indexes, triggers/functions and sequence ownership. Unknown
or partial schemas fail without changing migration history. The exact historical
Django baseline plus the Go baseline's auxiliary tables and eight additional
indexes is also supported. If the baseline and its replacement records are
already coherent, use ordinary forward Django migration without re-adopting
history. The physical catalog and forward data checks still apply; a migration
record alone never certifies the schema. After reviewing the
inspection, supply its exact catalog digest and a new audit file:

```bash
uv run python manage.py reproq adopt-schema --database <alias> \
  --expected-catalog-sha256 <inspected-sha256> --audit-output <new-audit-file.json>
uv run python manage.py reproq migrate-worker --database <alias>
```

Adoption reconciles the replacement markers covered by the published squash;
it does not invent code for its never-committed third replacement member. The
receipt records before/after history and catalog identity. Adoption requires an
outermost autocommit boundary; it refuses nested or manually managed transactions.
An interrupted `prepared`, `unconfirmed`, or `commit_unknown` receipt requires
independent ledger verification. Losing a COMMIT acknowledgement does not prove
rollback. Receipt contents are fsynced and directory entries are flushed where
the platform supports it; database and filesystem commits are separate resources.
There is no
`--fake` or table-existence shortcut. The known raw Go profile's unvalidated
task-path check requires valid existing rows and is validated by the forward
Django migration.

The currently qualified PostgreSQL catalog profiles use the `public` schema.
Other queue namespaces require their own qualified catalog support; the guard
fails closed rather than treating a migration record as schema proof. The
forward migration also verifies all four core tables and any preexisting
auxiliary tables before establishing the new contract.
Adoption, migration and runtime checks bind all queue, marker and history
relations to ordinary permanent public tables. Temporary objects and shadow
schemas cannot substitute for the validated tables, even on a public-first path.

### 7. Start the Worker
```bash
uv run python manage.py reproq worker
```

### 8. (Optional) Schedule Periodic Tasks
You have two options (choose one):

**Option A: Run Beat (one process per database)**
```bash
uv run python manage.py reproq beat
```

**Option B: Use Postgres-native scheduling (pg_cron)**
```bash
uv run python manage.py reproq pg-cron --install
```
`pg_cron` requires the Postgres extension to be enabled (see `docs/deployment.md`).

---

## 🧰 Management Commands

Run `uv run python manage.py reproq <subcommand>` to manage the worker and day-to-day ops.
Full reference (examples + exit codes): `docs/cli.md`.

- **Bootstrap**: `init` writes `reproq.toml`, installs the worker, and runs migrations. Explicit YAML configuration requires the optional PyYAML reader in the Django environment.
- **Config**: `config --explain` prints the effective config and its precedence.
- **Doctor**: `doctor --strict` validates DSN, schema, worker binary, and allowlist.
- **Upgrade**: `upgrade` installs the release tested with this SDK and optionally applies the committed queue migrations.
- **Allowlist**: `allowlist --write --config reproq.yaml` populates `allowed_task_modules`.
- **Ops**: `status`/`stats`, `logs --id <result_id>`, `cancel --id <result_id>`.

---

## ✅ Why Reproq (vs Celery, RQ, Huey)

Reproq is built for teams who want deterministic background tasks without adding Redis or RabbitMQ.

- **No extra broker**: Uses Postgres only; no Redis/RabbitMQ to provision.
- **Deterministic deduping**: Identical tasks can be coalesced safely.
- **Django-native**: Implements the Django 6.0 Tasks API end-to-end.
- **Operationally lean**: One database + one Go worker binary.

If you need complex routing, multi-broker support, or huge existing Celery ecosystems, Celery may still fit better. Reproq prioritizes clarity, determinism, and low operational overhead.

---

## 📚 API Reference

Reproq fully implements the Django 6.0 Tasks API.

### Defining Tasks
Use the standard `@task` decorator. Reproq respects `queue_name` and `priority`.

```python
from django.tasks import task

@task(queue_name="high-priority", priority=100)
def send_welcome_email(user_id):
    # logic here
    return f"Email sent to {user_id}"
```

You can also annotate a task with a concurrency limit:

```python
from reproq_django.concurrency import limits_concurrency

@limits_concurrency(lambda user_id: f"user:{user_id}", to=1)
@task(queue_name="high-priority", priority=100)
def send_welcome_email(user_id):
    return f"Email sent to {user_id}"
```

### Enqueuing Tasks
Use Django's standard task API. Every enqueue keyword is a function argument;
ReproQ does not reserve names such as priority, run_after or concurrency_key.
Delivery options belong on the task, backend configuration or decorators:

```python
from datetime import timedelta
from django.utils import timezone

result = send_welcome_email.enqueue(123)
result = send_welcome_email.using(
    priority=42,
    run_after=timezone.now() + timedelta(minutes=10),
).enqueue(123)
```

Use `@limits_concurrency(lambda user_id: f"user:{user_id}", to=2)` for a shared
key, as shown above. Named task function attributes can supply custom lock/key
options; they never consume function kwargs. Direct, bulk and periodic enqueue
use the same typed arguments/spec builder. Strict integer budgets and optional
string keys are validated before database writes; bool/float coercion is rejected.
Concurrency capacity is the strongest positive candidate/active limit for a
shared key. A zero contributes no limit but does not bypass an active key's cap.

The former delivery-in-function-kwargs extension is retired. Migrate delivery
priority/run_after to Task.using and shared concurrency to the decorator; leave
same-named function parameters intact.

### Async Contexts (ASGI)
If you are in an async view or task producer, use `aenqueue()` to avoid blocking:

```python
result = await send_welcome_email.aenqueue(123)
```

In sync/Wsgi code, continue to use `enqueue()`.

### Task Context + Metadata
If your task needs context (result id, attempt, metadata), use `takes_context=True`.

```python
from django.tasks import task

@task(takes_context=True)
def process_upload(context, upload_id):
    context.metadata["stage"] = "starting"
    context.save_metadata()
    # work...
    context.metadata["stage"] = "done"
    context.save_metadata()
```

Metadata is stored in `task_runs.metadata_json` and surfaced in task results.

### Bulk Enqueuing
Use `bulk_enqueue` for batches of jobs. It groups database inserts and uses the
same task options, argument codec and canonical spec builder as `enqueue`.
Choose a batch size that fits the producer's memory budget.

```python
from django.tasks import task_backends
from datetime import timedelta
from django.utils import timezone

backend = task_backends["default"]
jobs = []
scheduled_at = timezone.now()

for i in range(1000):
    # (task_func, args, kwargs)
    jobs.append((
        send_welcome_email.using(run_after=scheduled_at + timedelta(seconds=i)),
        (i,),
        {},
    ))

backend.bulk_enqueue(jobs)
```

---

## ⏰ Periodic Tasks

Register source intent through the SDK, then run a scheduler. Migrations do not
seed or reset operator schedules. Direct ORM creation leaves a legacy/unassigned
row that requires explicit reconciliation; it cannot silently become runnable.

```python
from myapp.tasks import nightly_cleanup
from reproq_django.recurring import register_periodic

register_periodic(
    name="Nightly cleanup",
    schedule="CRON_TZ=America/Chicago 0 2 * * *",
    task=nightly_cleanup,
    queue_name="maintenance",
    scheduler_provider="go",  # or "pg_cron" on a configured database
)
```

The same canonical SDK serializer handles direct, bulk and periodic task specs,
including typed arguments, context, retry/timeout budget, routing and provenance.
Registration preserves an existing definition's enabled, next-run and last-run
state. Resetting schedule state or assigning another provider is explicit. The
admin permits pausing/resuming definitions; canonical spec fields are read-only.

Each definition belongs to one provider. Long-running Go beat and one-shot Go
scheduling use the same implementation:

The read-only stats API and `reproq status` distinguish disabled definitions,
registration problems, past-due Go cursors measured against the database clock,
and managed beat being disabled by configuration. Worker records are counted
separately as alive or stale against each selected database's statement clock;
their heartbeats do not prove scheduler execution. Admin status uses that same
classification. The API's existing `worker_health.dead` key counts stale records.
Explicit admin and CLI worker pruning also compare against the selected database
clock, preserving their existing retention thresholds and selections. Neither
status nor doctor runs pruning or changes task history.
`reproq doctor --strict` also fails on these scheduling warnings. A past-due
cursor does not identify the cause, and native pg_cron uses its own occurrence
clock. Prestart `reproq check` does not wait for due cursors to advance: beat
must first be allowed to start. `LOW_MEMORY_MODE=1` does not disable the current
supervisor's explicitly configured beat process.

Set secrets in the environment and reference them by variable name in a custom
`REPROQ_PRESTART_CMD`. The supervisor discards custom prestart stdout and stderr,
including Bash syntax diagnostics that can repeat the command. Supervisor status
and output from the built-in prestart check remain visible.

```bash
uv run python manage.py reproq beat
# Or invoke this from an external timer; do not also run the long-running beat:
uv run python manage.py reproq schedule
```

Go coalesces an overdue definition to one occurrence at its stored nominal
`next_run_at`, then advances beyond the database clock. It does not invent every
missed tick. The occurrence identity `(periodic_name, scheduled_for)` remains
unique after completion or failure, while separate occurrences may overlap
subject to the configured concurrency key/limit.

The optional native provider uses the same migration-owned PostgreSQL enqueue
primitive and the exact stored SDK bytes. It needs an administrator-installed
`pg_cron` extension and its named-job API; the SDK does not create extensions or
functions. Assign definitions to `pg_cron`, then reconcile only their owned jobs:

```bash
uv run python manage.py reproq pg-cron --install --dry-run
uv run python manage.py reproq pg-cron --install
uv run python manage.py reproq pg-cron --remove
```

`--if-supported` skips only an unavailable extension/database capability. It does
not hide invalid definitions or mismatched zones. The portable grammar uses ASCII numbers/names, lists, ascending ranges and positive wildcard/range steps; Weekdays are0–6 (Sunday0). L/#/wrapping ranges, scalar steps and macros are rejected. Bare cron expressions mean UTC;
`CRON_TZ`/`TZ` selects a named zone. Native `cron.timezone` must match that zone
(UTC/GMT aliases are accepted). Go supports per-definition zones; pg_cron uses
its database server's configured zone. Native occurrences are identified by the
UTC minute when their SQL starts, not an asserted nominal tick or a replay of
missed ticks. Native scheduler delays/catch-up follow pg_cron; the SDK does not
synthesize a misleading `next_run_at` for it. Inspect native job run history to
verify actual execution. See [deployment](docs/deployment.md) for prerequisites.

Before upgrading a database with the old native helper, explicitly remove its
known owned jobs with the reviewed SDK command while legacy schedulers are
stopped. Forward migration0003 owns retiring that exact legacy helper and
installing the shared primitive; unknown functions/dependencies/jobs fail closed.

Code-defined registrations can use `@recurring` with Django's standard `@task`:

```python
from django.tasks import task
from reproq_django.recurring import recurring

@recurring(schedule="CRON_TZ=America/Chicago 0 9 * * *", key="daily_report", args=(42,))
@task(queue_name="maintenance")
def send_report(account_id):
    ...
```

Run `uv run python manage.py reproq sync-recurring` explicitly after imports/deployment.
Named legacy rows can be canonicalized with `reconcile-periodic --name NAME`
(`--dry-run` previews with transaction rollback). Neither runs on `post_migrate`.
To run a task immediately, call its standard `enqueue()` method; this does not
rewrite its recurring schedule or pretend to be a scheduled occurrence.

---

## ⚙️ Configuration

Configure Reproq behavior via the `TASKS` setting in `settings.py`.

```python
TASKS = {
    "default": {
        "BACKEND": "reproq_django.backend.ReproqBackend",
        "OPTIONS": {
            # Deduplication (Default: True)
            # If True, enqueuing a task with the exact same arguments as a 
            # READY/RUNNING task will return the existing result_id.
            "DEDUP_ACTIVE": True,

            # Execution Timeout (Default: 900)
            # Max seconds a task can run before being killed by the worker.
            "TIMEOUT_SECONDS": 900,

            # Retry Limit (Default: 3)
            # Max number of attempts for a task.
            "MAX_ATTEMPTS": 3,

            # Expiry (Optional)
            # If set, tasks not picked up by this time will be marked expired.
            "EXPIRES_IN": timedelta(hours=24),
            
            # Provenance (Optional)
            # Metadata stored with the task for auditing.
            "CODE_REF": "git-sha-or-version",
            "PIP_LOCK_HASH": "hash-of-dependencies",
        }
    }
}
```

Worker config files (`reproq.yaml`/`reproq.toml`) are optional. Precedence is:
defaults < config file < env vars < CLI flags. Django management commands derive
an empty DSN from the selected queue database alias and validate explicit
file/environment targets against it. The standalone Go CLI also accepts `--dsn`.

<a id="executor-python-selection-sdk-01531--worker-v00145"></a>

### Executor Python selection (SDK 0.1.532 / Worker v0.0.145)

This interpreter binding requires the matching Go/SDK pair and queue migrations
described in Quickstart. Its local checks do not establish application workload,
native-platform capacity or production deployment qualification.

Run management commands through `uv run`. The Django worker defaults its
executor to that selected `sys.executable`, passed explicitly
through Go's `--python-bin`. An explicit `worker.python_bin` in a config file or
`--python-bin` command option takes precedence. Generated configuration omits
this optional value so it remains portable between environments.
`config --explain`, `check`, and `doctor` show the effective executable and its source.
The worker must advertise `worker_python_bin` operation version1; the SDK checks
this before installation or foreground startup. The standalone Go default
remains `python3`.

### Multi-Database Queues
Route specific queues to different Django database aliases.

```python
DATABASES = {
    "default": {...},
    "queues": {...},
}

REPROQ_QUEUE_DATABASES = {
    "high-priority": "queues",
    "bulk-*": "queues",  # glob patterns supported
}
```

Use `REPROQ_DEFAULT_DB_ALIAS` to change the fallback alias (defaults to
`"default"`). When multiple queue databases are configured, task result IDs are
prefixed with a canonically percent-encoded database alias (`queues:123`).
Multiple databases cannot disable this prefix. Unsupported encoded alias
lengths are rejected before producer writes, keeping native result IDs below
64 characters without truncation or guessed routing.
To route all Reproq tables to a non-default alias in ORM reads/writes (admin,
health checks, etc.), add the router:
```python
DATABASE_ROUTERS = ["reproq_django.db_router.ReproqRouter"]
```

Run one worker/beat per database:

```bash
uv run python manage.py reproq worker --database queues --queues high-priority,bulk-1
uv run python manage.py reproq beat --database queues
```

Use `REPROQ_STATS_DATABASES=["*"]` to aggregate stats across all queue databases.

---

## 🧪 Development

`.python-version` selects Python 3.14.10 for local development. The bootstrap
installs the locked development dependencies; the compatibility checks still
cover Python 3.12–3.15.

```bash
bash scripts/dev_bootstrap.sh
uv run pytest
```
Always run tests with `uv run pytest` so dependencies and settings stay consistent.

---

## ⛓ Workflows (Chains & Groups)

Reproq supports complex task dependencies.

### Chains (Sequential)
Execute tasks one after another. If a task fails, the chain stops.

```python
from reproq_django.workflows import chain

# task_a -> task_b -> task_c
c = chain(
    (task_a, (1,), {}),
    task_b, # no args
    (task_c, (), {"param": "val"})
)
results = c.enqueue()
# results[0] is READY, results[1..] are WAITING
```

### Groups (Parallel)
Execute tasks in parallel.

```python
from reproq_django.workflows import group

g = group(
    (resize_image, ("img1.jpg",), {}),
    (resize_image, ("img2.jpg",), {}),
)
results = g.enqueue()
```

### Chords (Group + Callback)
Run a callback once a group finishes.

```python
from reproq_django.workflows import chord

group_results, callback_result = chord(
    (resize_image, ("img1.jpg",), {}),
    (resize_image, ("img2.jpg",), {}),
    callback=notify_done,
).enqueue()
```
The callback runs only after all group tasks succeed. Failed tasks leave the callback waiting.
Failures mark the workflow as failed and the callback will not run.

Automatic retries keep dependents waiting until the task succeeds or exhausts
its budget. Terminal workflow members cannot be manually reset in place: that
would break dependency order or aggregate counters. Enqueue a fresh workflow to
recover the workflow, or explicitly replay one task as a detached new task.

---

## 🔁 Retries & Backoff

Retries are managed by the Go worker. When a task fails and attempts remain, it is re-queued with an **exponential backoff**:

- Base delay: 30s
- Backoff: `2^attempt` (attempt starts at 1)
- Cap: 1 hour

The worker updates `run_after` on the failed task, and the backend will only claim it after that timestamp.

---

## 🚦 Rate Limiting

Reproq Worker enforces token bucket limits stored in the `rate_limits` table. You can manage these from Django Admin or via the worker CLI.

Keys:
- `queue:<queue_name>` limits a specific queue.
- `task:<task_path>` limits a specific task (overrides queue/global).
- `global` is a fallback when no task/queue limit exists.

Example:
```bash
reproq limit set --key queue:default --rate 5 --burst 10
```
Defaults: global rate limiting is disabled until you set a positive rate.

---

## 🧰 Worker CLI Ops

The Go worker binary includes operational commands you can run directly:

```bash
# Request cancellation of a running task
reproq cancel --dsn "..." --id 12345

# Inspect failed tasks
reproq triage list --dsn "..." --limit 50
```

---

## 🖥 Management Commands

The `uv run python manage.py reproq` command is your Swiss Army knife.

| Subcommand | Description |
| :--- | :--- |
| `init` | Bootstraps Reproq in the current project. |
| `worker` | Starts the Go worker. Flags: `--config`, `--concurrency` (default 10), `--queues`, `--allowed-task-modules`, `--logs-dir`, `--payload-mode`, `--python-bin`, `--metrics-port`, `--metrics-addr`, `--metrics-auth-token`, `--metrics-allow-cidrs`, `--metrics-tls-cert`, `--metrics-tls-key`, `--metrics-tls-client-ca`, `--database`. Auto-configures allow-list when unset (unless config file is used). |
| `beat` | Starts the scheduler. Flags: `--config`, `--interval` (default 30s), `--once`, `--database`. |
| `schedule` | Enqueue due periodic tasks once and exit (alias for `beat --once`). |
| `pg-cron` | Syncs Postgres-native schedules. Flags: `--install` (default), `--remove`, `--prefix`, `--dry-run`, `--database`. |
| `install` | Downloads/builds the worker binary. |
| `upgrade` | Upgrades the worker binary and optionally runs `migrate-worker`. |
| `migrate-worker` | Applies committed ReproQ Django migrations on the selected database alias. |
| `adopt-schema` | Inspects or explicitly adopts a fully verified known legacy PostgreSQL schema. |
| `check` | Validates binary path, DB connection, and schema health. |
| `doctor` | Validates DSN, schema, worker binary, and allowlist; `--strict` fails on warnings. |
| `config` | Prints effective worker/beat config; use `--explain` for precedence. |
| `allowlist` | Prints `ALLOWED_TASK_MODULES` or writes them to a config file with `--write`. |
| `sync-recurring` | Sync code-defined recurring schedules into `periodic_tasks`. |
| `pause-queue` | Pause a queue (prevents new claims). |
| `resume-queue` | Resume a paused queue. |
| `logs` | Prints logs for a task run using `logs_uri` (supports aliased result IDs). |
| `cancel` | Requests cancellation of a task run by result ID (supports aliased result IDs). |
| `reclaim` | Requeue or fail tasks with expired leases (`--database` / `--all-databases`). |
| `prune-workers` | Delete workers not seen recently (`--database` / `--all-databases`). |
| `prune` | Delete terminal task history by status/age in bounded batches (`--database` / `--all-databases`). |
| `stats` / `status` | Shows task counts by status and active workers (`--database` / `--all-databases`). |
| `systemd` | Generates systemd service files for production. |

---

## 📊 Stats API (JSON)

When you include `reproq_django.urls` in your project, `GET /stats/` returns
JSON task counts, per-queue task counts, worker records, and periodic task
schedules. Access is granted to staff sessions, a signed TUI JWT, or
`METRICS_AUTH_TOKEN` as a bearer token.
Additional fields include `worker_health`, `queue_controls`, `scheduler`, and
per-database rollups when `REPROQ_STATS_DATABASES` is set.

## 🧭 TUI Integration

Set `METRICS_AUTH_TOKEN` to enable the TUI login flow and sign TUI JWTs. This
token is also accepted as a bearer token for `/reproq/stats/` and is forwarded
to the worker metrics proxy endpoints. If you do not want to expose SSE, set
`REPROQ_TUI_DISABLE_EVENTS=1` to omit the `/reproq/tui/events/` stream from the
TUI config payload.
If `METRICS_AUTH_TOKEN` is unset, the TUI auth endpoints return 404.

Set `LOW_MEMORY_MODE=1` to disable the worker proxy endpoints
(`/reproq/tui/metrics/`, `/reproq/tui/healthz/`, `/reproq/tui/events/`). The
endpoints return a 503 with a short hint while low-memory mode is enabled, and
events are omitted from the TUI config payload.

Set `REPROQ_MEMORY_LOG_INTERVAL=60s` to emit periodic memory logs from the
Django process (includes RSS on Linux).

Pass the TUI JWT in the `Authorization: Bearer ...` header when accessing
these endpoints programmatically.

---

## 🧾 Worker/Beat Config Files

The Go worker/beat support YAML/TOML config files. `manage.py reproq worker` and `manage.py reproq beat`
will load a config file when `--config` or `REPROQ_CONFIG` is set. Otherwise,
they look for `reproq.yaml`, `reproq.yml`, `reproq.toml`, `.reproq.yaml`, `.reproq.yml`, or `.reproq.toml`
in the current working directory, even when other flags are present. Actual CLI
options override environment and config values. Empty queue DSNs inherit from
the selected Django queue database; explicit targets must match that alias.

The canonical templates are [TOML](src/reproq_django/resources/reproq.example.toml)
and [YAML](src/reproq_django/resources/reproq.example.yaml). `reproq init` reads
these packaged resources; files in the current directory cannot override them.

Queue selection uses `--queues` (comma-separated). The worker's legacy `--queue`
flag has been retired; queue-control commands retain their positional queue name.

---

## ⚖️ Scaling: Workers vs Concurrency

- **Increase concurrency** (`--concurrency`): more goroutines in a single worker process; best for I/O-heavy tasks with minimal overhead.
- **Run multiple workers**: separate processes/hosts; best for CPU-heavy workloads and fault isolation.

Start with one task slot on a small shared web instance. Measure combined web,
Go, Python executor and database connection peaks under representative work;
reserve headroom before increasing concurrency. Each active Django task has its
own Python process, so a small Go footprint does not make a memory-heavy task
cheap. Move heavy ETL to a separate process/host when it exceeds the web budget.

---

## 🔍 Admin Dashboard

Reproq integrates deeply with the Django Admin.

- **Task Runs**: View all tasks. Filter by status, queue, or lease state.
- **Actions**:
    - **Replay**: Select tasks to re-enqueue them (creates a new copy).
    - **Retry Failed**: Reset failed tasks to READY.
    - **Cancel**: Request cancellation of running/ready tasks.
- **Workers**: Monitor active worker nodes, their concurrency, and last heartbeat.
- **Periodic Tasks**: Inspect registered definitions and enable/disable them. Register or change execution and schedule intent through the explicit SDK/management API; the admin does not create definitions or edit cron/next-run state.
- **Status Note**: `WAITING` projects to Django's `READY`; `CANCELLED` projects to `FAILED`. The original value remains available via `raw_status`.
- **Arguments**: Native `args`/`kwargs` contain JSON transport values. Explicit `decoded_args`/`decoded_kwargs` extensions restore timedeltas and saved models from their recorded source database on first access. Async callers use `await result.adecoded_args()` or `await result.adecoded_kwargs()` to keep model lookups off the event loop. A refresh invalidates the decoded cache without querying business models. If a referenced model was deleted, only decoded access raises; JSON arguments, status and successful return values remain inspectable.

Results and contexts inherit Django's frozen, slotted native classes.
`refresh()` and `arefresh()` update the result and return `None`. Native
`attempts` and `context.attempt` derive from the complete `worker_ids` history,
including the current worker as in Django's built-in backends. The separate
`budget_attempt` extension is ReproQ's resettable retry-budget counter. Context
metadata is an explicit mutable mapping; save it with the fenced
`save_metadata()` or `asave_metadata()` method rather than replacing the mapping.
Native `result.task.run_after` preserves original enqueue intent. The distinct
`eligible_after` extension reports mutable queue eligibility for periodic slots,
retry backoff or replay. Canonical text supplies v1 arguments; a JSONB projection
cannot preserve every source detail, including the sign of negative zero.

Task failures retain their qualified Python exception class and bounded stack
frames in Django's native `TaskError` records. Diagnostics omit source lines,
locals and chained details; an explicit marker explains when a message or stack
exceeds the budget. Worker or protocol failures without a recoverable Python
exception use `reproq_django.protocol.WorkerExecutionError`. `raw_errors` retains
the original worker records for inspection.

Use the unified `reproq check`, `reproq doctor` and `reproq worker` commands.
The standalone `reproq_check`, `reproq_health` and `reproq_init_executor`
commands, generated executor stubs and HTTP/CLI stress-test entry points have
been retired. The canonical executor module is `reproq_django.executor`, launched with the configured Python executable.

Timedeltas and saved model references use explicit typed envelopes. Other JSON
normalization follows DjangoJSONEncoder: Decimal, UUID and date/time values
arrive as strings; tuples arrive as lists. Do not depend on those Python types
surviving a queue round trip. Pass explicit task-domain representations when
their type or precision is part of the task contract.

---

## 🚀 Production Deployment

### Recommended Setup (Systemd)
Generate service files to run the worker and beat processes under systemd 251 or newer.

```bash
uv run python manage.py reproq systemd --user myuser --concurrency 20
```

This generates `reproq-worker-<project>.service` and `reproq-beat-<project>.service`.
Copy them to `/etc/systemd/system/` and enable them.
You can pass metrics flags (for example `--metrics-addr 127.0.0.1:9090`) or use
`--env-file` to load `METRICS_AUTH_TOKEN` and `METRICS_ALLOW_CIDRS` from one
required file. Relative paths are resolved against the project directory.

To use cron-style scheduling instead of a persistent beat process, generate a timer:

```bash
uv run python manage.py reproq systemd --schedule --schedule-on-calendar "*-*-* *:*:00"
```

Set `REPROQ_SCHEDULER_MODE=cron` in your EnvironmentFile and enable the timer to ensure
only one scheduler runs.

### Env Vars
The Go worker relies on standard environment variables:
- `REPROQ_DATABASE_URL`: `postgres://user:pass@host:5432/db` (optional if you provide `--dsn` or a config file)
- `WORKER_ID`: (Optional) Unique name for the node.
- `REPROQ_WORKER_BIN`: (Optional) Path to the binary if not using `manage.py reproq install`.
- `REPROQ_CONFIG`: (Optional) Path to a YAML/TOML worker/beat config file.
- `ALLOWED_TASK_MODULES`: (Optional) Comma-separated task module allow-list for the worker. If unset, `manage.py reproq worker` auto-configures it from discovered task modules.
- `REPROQ_LOGS_DIR`: (Optional) Directory to persist worker stdout/stderr logs (updates `task_runs.logs_uri`).
- `REPROQ_MEMORY_LOG_INTERVAL`: (Optional) Log memory stats at the given interval (e.g., `60s`).
- `METRICS_AUTH_TOKEN`: (Optional) Bearer token for `/metrics`, `/healthz`, and `/events`.
- `METRICS_ALLOW_CIDRS`: (Optional) Comma-separated IP/CIDR allow-list for metrics/health.
- `METRICS_TLS_CERT`: (Optional) TLS certificate path for health/metrics.
- `METRICS_TLS_KEY`: (Optional) TLS private key path for health/metrics.
- `METRICS_TLS_CLIENT_CA`: (Optional) Client CA bundle to require mTLS for health/metrics.

The SDK derives the queue DSN from the selected Django queue alias; namespaced environment/configuration must match it. Business DATABASE_URL is preserved for task execution.

Worker binary resolution order:
1. `REPROQ_WORKER_BIN` (setting or env)
2. `./.reproq/bin/reproq` (installed by `reproq install`)
3. `reproq_django/bin/reproq` (packaged fallback)
4. `PATH`

---

## 🤝 Contributing

Reproq is split into two repos:
- **Reproq Django**: This repo (Python/Django logic).
- **Reproq Worker**: The Go execution engine.

Issues and PRs are welcome in both!

## License

Apache License 2.0. See [LICENSE](LICENSE).
