Metadata-Version: 2.4
Name: xtr-orm-messenger
Version: 7.1.0
Summary: A database transport for xtr-messenger: messages kept in a table on an xtr-orm connection, claimed by workers, delayed, redelivered and woken by notifications.
Keywords: message-bus,message-queue,transport,database,sqlalchemy,asyncio
Author: Xterr
Author-email: Xterr <me@xterr.dev>
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Framework :: AsyncIO
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Database
Classifier: Typing :: Typed
Requires-Dist: msgspec>=0.20
Requires-Dist: sqlalchemy[asyncio]>=2.0.52
Requires-Dist: xtr-clock>=7.0,<8
Requires-Dist: xtr-messenger>=7.0,<8
Requires-Dist: xtr-orm>=7.0,<8
Requires-Python: >=3.14.4
Description-Content-Type: text/markdown

<div align="center">

# xtr-orm-messenger

**A database transport for xtr-messenger: messages kept in a table on an xtr-orm connection, claimed by workers, delayed, redelivered and woken by notifications.**

<img alt="python 3.14+" src="https://img.shields.io/badge/python-%E2%89%A5%203.14-3776AB?logo=python&logoColor=white">
<img alt="typed" src="https://img.shields.io/badge/typed-ty%20%2B%20basedpyright-1f6feb">
<img alt="license MIT" src="https://img.shields.io/badge/license-MIT-blue">

</div>

---

## Why?

A queue needs a broker, and a broker is one more server to run, back up and watch. An
application that already has a database can keep its messages there: in a table, on the
connection [xtr-orm](../xtr-orm) already configures, sent in the same transaction as the
changes they announce if it wants.

- 🗄️ **No broker** — `orm://default` is the application's own `default` connection.
- 🔒 **Workers never collide** — each claims the oldest due message `FOR UPDATE SKIP LOCKED`.
- ⏰ **Delays** — a `DelayStamp` makes a message due later.
- ♻️ **Nothing is lost to a dead worker** — a message never settled is handed out again.
- 🔔 **No polling on PostgreSQL** — a waiting worker is woken the moment a message is sent.
- 🤝 **Sent with the handler's changes** — `transactional=true` writes the message in the unit
  of work's session, so it commits or rolls back with them.

## Install

```sh
uv add xtr-orm-messenger
uv add "xtr-messenger[orm]"            # the same, from the message bus's side
```

It needs [xtr-messenger](../xtr-messenger) and [xtr-orm](../xtr-orm), which come with it. It
brings no database driver: a transport writes through the connection its DSN names, with the
driver the application installed for that connection — xtr-orm's `postgres`, `mysql` or
`sqlite` extra. On PostgreSQL through asyncpg, a waiting worker is woken by notifications; with
any other driver it polls. It ships no bundle: with a
container, the [orm bundle](../xtr-orm#message-bus-transport) registers the transport once this
package is installed.

## Quick start

Without a container, hand the factory the connections it builds on and the envelope serializer
the bus and its workers share:

```python
from xtr_messenger import (
    JsonSerializer,
    MessageBusConfig,
    MessageBusFactory,
    TransportConfig,
    WorkerFactory,
)
from xtr_orm import ConnectionRegistry
from xtr_orm_messenger import OrmTransportFactory
from xtr_serializer import JsonEncoder, PydanticNormalizer, Serializer

registry = ConnectionRegistry()
registry.register("default", engine=engine, migrator=migrator, database=database)

config = MessageBusConfig(
    transports={"jobs": TransportConfig("orm://default", queue="jobs")},
    routing={IngestDocument: "jobs"},
)
wire = JsonSerializer(Serializer([PydanticNormalizer()], [JsonEncoder()]))
factory = OrmTransportFactory(registry, wire)

await MessageBusFactory(config, [factory]).bus().dispatch(IngestDocument(document_id=1))
await WorkerFactory(config, [factory]).worker(["jobs"]).run()
```

The bus writes the message into `messenger_messages`; the worker claims it, hands it to its
handlers and deletes it. With a [container](#use-in-an-application) all of that is wired for you.

## How it works

| Step | What happens |
| --- | --- |
| Send | One row: the encoded envelope, its queue, `created_at`, and `available_at` — later by the delay, rounded up to a whole second |
| Claim | In one transaction, the oldest row of the queue that is due and not handed out is selected `FOR UPDATE SKIP LOCKED` and stamped `delivered_at` |
| Handle | The worker dispatches it; the envelope carries `OrmReceivedStamp` and `TransportMessageIdStamp`, both the row's id |
| Keep alive | While the worker holds it, the row's `delivered_at` is renewed every half of `redeliver_timeout`, so a long handler keeps its message |
| Settle | Acknowledged, the row is deleted. Rejected, it is retried or moved to the failure queue (see below) |
| Redeliver | A row not renewed for `redeliver_timeout` seconds is claimable again: its worker is taken for dead. A handler must be idempotent: a worker frozen that long has its message handled twice |

Due times and the redelivery window are measured on each worker's clock, not the database's:
keep the workers' clocks synchronized. A clock running ahead claims delayed messages early and
takes a live worker's message for abandoned.

A rejected message gets `max_attempts` attempts in all, the first included. While it has
attempts left its row is put back in its queue, due `base_delay_seconds` later, the delay
doubling with each attempt; the envelope carries a `RedeliveryStamp` with the attempt count and
an `ErrorDetailsStamp` with why the last one failed. Once its attempts run out the row moves to
the `failure_queue` of the same table, `failed` unless named otherwise, where `count()`, `all()`
and `find()` on a transport reading that queue show it, and a worker consuming it tries it
again, with a fresh count. A message failing for good in the failure queue itself is deleted.
The worker asks the transport before it announces a failure, so `WorkerMessageFailedEvent`
carries `will_retry=True` for an attempt that will be made again.

A worker that finds nothing due waits `poll_interval` seconds before asking again. On
PostgreSQL through asyncpg it listens instead: every send announces itself on the table's
channel, naming its queue, and the worker asks again at once — or after
`check_delayed_interval`, for a delayed message falling due. Listening holds one connection of
the pool for as long as the worker runs.
If the database drops that connection, the wait under way ends and the next one listens on a
fresh connection; while listening again fails, the worker polls every `poll_interval` and keeps
trying. A claim whose worker is cancelled before it hears of it is handed back at once, not left
until `redeliver_timeout`.

SQLite has no row locks, so the lock clause is left out there, and SQLite is for development and
tests. With the driver's default transaction mode two workers can claim one row; run one worker
on SQLite, or pass `connect_args={"autocommit": False}`, under which the second
gets "database is locked", which counts as a conflict and is tried again.

Conflicts between workers — a deadlock, a lock wait that timed out — say nothing about the
message. A poll sits through 3 in a row before it raises; an acknowledgement or a rejection
tries again with a pause doubling from 100 ms. One whose database went away waits up to 5
seconds for it and is replayed once: losing an acknowledgement to a restart would hand the
message out again.

A message that does not decode can never be handled, so it is rejected at once, its reason
logged on the `messenger` logger — never its body.

## The DSN

```sh
# .env
MESSENGER_DSN=orm://default
```

The format is `orm://<connection_name>`: the name of a connection the application configures
in xtr-orm, for when it has several and the messages go on one other than `default`; `orm://`
alone is the default connection. The URL, the driver and the pool are the connection's. The
transport creates a table named `messenger_messages` the first time it runs. To name the table
otherwise, give `table_name` in the DSN:

```sh
# .env
MESSENGER_DSN=orm://default?table_name=your_custom_table_name
```

The queue a transport reads and writes is `queue` — `?queue=jobs`, or
`TransportConfig(dsn, queue="jobs")` — the setting every xtr-messenger transport takes for it.

## Options

Every option may go in the DSN's query string or in `TransportConfig.options`; `options` wins.
An unknown one is refused with `UnknownTransportOptionError`, a bad value with
`InvalidTransportOptionError`.

| Option | Default | What it sets |
| --- | --- | --- |
| `queue` | `default` | the queue this transport reads and writes; also `TransportConfig(queue=...)` |
| `table_name` | `messenger_messages` | the table, a plain identifier |
| `redeliver_timeout` | `3600` | seconds after which a message handed out and never settled is handed out again |
| `auto_setup` | `true` | create the table the first time an operation finds it missing |
| `poll_interval` | `1.0` | seconds a worker waits before asking again when nothing was due |
| `transactional` | `false` | write the message in the session of the unit of work under way |
| `use_notify` | `true` | on PostgreSQL through asyncpg, wait for notifications instead of polling |
| `check_delayed_interval` | `60000` | milliseconds a listening worker waits at most before asking anyway |
| `get_notify_timeout` | `60000` | milliseconds one wait for a notification lasts at most |
| `max_attempts` | `3` | attempts a message gets, the first included, before it moves to the failure queue |
| `base_delay_seconds` | `1.0` | seconds before the second attempt; each later one waits twice as long |
| `failure_queue` | `failed` | the queue of the same table a message is moved to once its attempts run out |

`orm://` with no name uses the default connection. A name the application did not configure is
refused with `InvalidOrmDsnError` when the transport is built.

### Sending with the handler's changes

With `transactional=true`, a message dispatched while a message is handled — or a request, or a
command: any unit of work — is written in that unit's session and sent only when the unit
commits. Run the `orm_transaction` middleware, or commit yourself: a session never commits on
its own. Outside a unit of work the message is written in a transaction of its own, as it is
without the option.
A follow-up held back with `DispatchAfterCurrentBusStamp` in a worker goes out after the
handled message's unit is over; with `transactional=true` it is written in the worker's session
only when `orm_transaction` runs for it too, so keep that middleware on the bus the worker uses.

## Looking into a queue

`OrmTransport` is a `TransportInterface` with more to ask:

```python
transport = factory.transport("jobs", config.transports["jobs"])

await transport.count()  # what a worker could claim now
async for envelope in transport.all(limit=10):
    ...  # the same, oldest first
await transport.find(message_id)  # one message, handed out or not
await transport.keepalive(envelope, seconds=600)  # a long handler keeps its message
await transport.setup()  # create the table ahead of time
```

`keepalive` restarts a message's redelivery clock; `seconds` is how long until the next call,
refused when longer than `redeliver_timeout`. A worker already renews what it holds on its own;
call it for a message held some other way.

These are the messenger's `ListableReceiverInterface`, `MessageCountAwareInterface` and
`SetupableTransportInterface`, so the console commands read a queue through them:
`messenger:setup-transports`, `messenger:stats`, `messenger:show`, and the
`messenger:failed:*` commands on the transport reading `failure_queue` — name it the
configuration's `failure_transport`. A message moved to `failure_queue` carries a
`SentToFailureTransportStamp` naming the transport it failed on and when, which
`messenger:failed:show` shows and `--failed-after` / `--failed-before` select by.

## Use in an application

Everything adding this package to an application takes — and, read backwards, what removing it
undoes. It ships no bundle: the [orm bundle](../xtr-orm#message-bus-transport) registers the
transport, built on its connections, when the messenger bundle is active too.

- **Install** — `uv add xtr-orm-messenger`, or `uv add "xtr-messenger[orm]"`, beside
  [xtr-messenger](../xtr-messenger/README.md#use-in-an-application) and
  [xtr-orm](../xtr-orm/README.md#use-in-an-application) set up as their *Use in an application*
  says. The database driver is the connection's own.
- **Recipe** — `uv run xtr-recipes recipes:sync` writes nothing: there is no bundle to list, no
  configuration file, environment variable or ignore line. It prints the step naming a
  transport, which a recipe cannot make for you.
- **Configure** — name a transport `orm://default` in `<app>/config/packages/messenger.py` and
  route messages to it, as below.
- **Run** — `<script> orm:migrations:diff` writes the messages table of every `orm://`
  transport into its connection's next revision; until it is applied, the transport creates the
  table the first time it runs, unless `auto_setup=false`. A worker is
  `<script> messenger:consume <transport>`.
- **Remove** — drop the `orm://` transports and the routing to them, and a migration dropping
  the table. Then `uv remove xtr-orm-messenger`.
- **Check** — `debug:config messenger` lists the `orm://` transports.

Everything else — the bundles, the connections, the bus — is xtr-orm's and xtr-messenger's.

```python
# app/config/packages/messenger.py
from xtr_dependency_injection import configure
from xtr_messenger.bundle import MessageBusConfig, TransportConfig


@configure
def messenger() -> MessageBusConfig:
    return MessageBusConfig(
        transports={"jobs": TransportConfig("orm://default?auto_setup=false", queue="jobs")},
        routing={IngestDocument: "jobs"},
    )
```

The orm bundle registers `OrmTransportFactory` with the messenger, built on its
`ConnectionRegistry` and the messenger's envelope serializer, and consulted ahead of the
factories found by DSN scheme. A bus and every worker of the container share it, and so share
each transport. When the kernel shuts down, the connections workers listened on are given back.

Without a container, `messages_tables(bus_config, connection, default_connection)` names the
tables the transports of a bus configuration keep on a connection, and `MessagesTableDriver`
adds one to the `MetaData` a connection's migrations compare with.

## Errors

Everything this library raises derives from `OrmMessengerError`, itself a
`MessageBusError` of [xtr-messenger](../xtr-messenger):

| Error | Raised when |
| --- | --- |
| `TransportError` | the database refuses what a transport asked — sending, claiming, settling, listing — or an envelope settled here was not received here; the driver's error is its cause |
| `InvalidOrmDsnError` | an `orm://` DSN names a connection the application does not configure; also a `LookupError` |

## Upgrading

What changes for an application moving to 7.0, and how to keep the old behaviour where that is
still possible: [Upgrading to 7.0](../../UPGRADING.md#xtr-orm-messenger).

## Development

Developed in the [python-xtr](https://github.com/xterr/python-xtr) monorepo, under
`packages/xtr-orm-messenger`; run the commands below from there. The `python-xtr-orm-messenger`
repository is a read-only copy, so send issues and pull requests to the monorepo.

```sh
uv sync
uv run ruff check && uv run ruff format --check && uv run basedpyright && uv run ty check && uv run pytest
```

Every test runs on SQLite files. The PostgreSQL suite under `tests/postgresql/` runs when
`XTR_TEST_POSTGRES_URL` names a database, as an asyncpg URL:

```sh
XTR_TEST_POSTGRES_URL=postgresql+asyncpg://user:password@127.0.0.1/test uv run pytest tests/postgresql
```

## License

MIT — see [LICENSE](LICENSE).
