Metadata-Version: 2.1
Name: django-jaiminho
Version: 2.0.2
Summary: A broker agnostic implementation of outbox and other message resilience patterns for Django apps
Home-page: https://github.com/loadsmart/django-jaiminho
Author: Loadsmart
Author-email: jaiminho@loadsmart.com
Project-URL: Documentation, https://github.com/loadsmart/django-jaiminho/blob/master/README.md
Project-URL: Source, https://github.com/loadsmart/django-jaiminho
Project-URL: Changelog, https://github.com/loadsmart/django-jaiminho/blob/master/CHANGELOG.md
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: Other/Proprietary License
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.8, <4
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: Django
Requires-Dist: sentry-sdk
Requires-Dist: dill==0.4.0

# jaiminho

[![CircleCI](https://dl.circleci.com/status-badge/img/gh/loadsmart/django-jaiminho/tree/master.svg?style=svg)](https://dl.circleci.com/status-badge/redirect/gh/loadsmart/django-jaiminho/tree/master)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/python/black)

A broker agnostic implementation of the outbox and other message resilience patterns for Django apps. 

![Jaiminho](https://github.com/loadsmart/django-jaiminho/blob/master/assets/jaiminho.jpg?raw=true)

## Migrating from 1.x.x

`SIGN_EVENTS` and `VERIFY_EVENTS_SIGNATURE` are enabled by default once you install the latest version. However, if you're
migrating from a previous version to the latest one, you'll have to do it in two steps:

- Install the latest version of `django-jaiminho` and set `VERIFY_EVENTS_SIGNATURE` to `False`;
- Ensure there are no unsent events without a signature and set `VERIFY_EVENTS_SIGNATURE` to `True`.

If there are unset events without a signature and `VERIFY_EVENTS_SIGNATURE` is enabled, your events will be considered tampered and won't be relayed.

## Getting Started

To use jaiminho with your project, you just need to do 6 steps:

### 1 - Install it

```sh
python -m pip install jaiminho
```

### 2 - Add jaiminho to the INSTALLED_APPS 

```python
INSTALLED_APPS = [
    ...
    "jaiminho"
]
```

### 3 - Run migrations

```sh
python manage.py migrate
```

### 4 - Configure jaiminho options in Django settings.py:
```python
JAIMINHO_CONFIG = {
    "PERSIST_ALL_EVENTS": False,
    "DELETE_AFTER_SEND": True,
    "DEFAULT_ENCODER": DjangoJSONEncoder,
    "PUBLISH_STRATEGY": "publish-on-commit",
}

```

### 5 - Decorate your functions with @save_to_outbox
```python
from jaiminho.send import save_to_outbox

@save_to_outbox
def any_external_call(**kwargs):
    # do something
    return
```

### 6 - Run the relay events command

```
python manage.py events_relay --run-in-loop --loop-interval 1

```

If you don't use `--run-in-loop` option, the relay command will run only 1 time. This is useful in case you want to configure it as a cronjob.


## Details

Jaiminho `@save_to_outbox` decorator will **intercept** decorated function and **persist** it in a **database table** in the same **transaction** that is active in the decorated function context. The event relay **command**, is a **separated process** that fetches the rows from this table and execute the functions. When an outage happens, the event relay command will **keep retrying until it succeeds**. This way, **eventual consistency is ensured** by design.

### Configuration options

- `PUBLISH_STRATEGY` - Strategy used to publish events (publish-on-commit, keep-order)
- `PERSIST_ALL_EVENTS` - Saves all events and not only the ones that fail, default is `False`. Only applicable for `{ "PUBLISH_STRATEGY": "publish-on-commit" }` since all events needs to be stored on keep-order strategy. 
- `DELETE_AFTER_SEND` - Delete the event from the outbox table immediately, after a successful send
- `DEFAULT_ENCODER` - Default Encoder for the payload (overwritable in the function call)
- `SIGN_EVENTS` - Signs events to support verification later
- `VERIFY_EVENTS_SIGNATURE` - Verifies previously generated signatures

### Strategies

#### Keep Order
This strategy is similar to transactional outbox [described by Chris Richardson](https://microservices.io/patterns/data/transactional-outbox.html). The decorated function intercepts the function call and saves it on the local DB to be executed later. A separate command relayer will keep polling local DB and executing those functions in the same order it was stored. 
Be carefully with this approach, **if any execution fails, the relayer will get stuck** as it would not be possible to guarantee delivery order.  

#### Publish on commit

This strategy will always execute the decorated function after current transaction commit. With this approach, we don't depend on a relayer (separate process / cronjob) to execute the decorated function and deliver the message. Failed items will only be retried
through relayer. Although this solution has a better performance as only failed items is delivered by the relay command, **we cannot guarantee delivery order**.


### Relay Command
We already provide a command to relay items from DB, [EventRelayCommand](https://github.com/loadsmart/django-jaiminho/blob/master/jaiminho/management/commands/events_relay.py). The way you should configure depends on the strategy you choose. 
For example, on **Publish on Commit Strategy** you can configure a cronjob to run every a couple of minutes since only failed items are published by the command relay. If you are using **Keep Order Strategy**, you should run relay command in loop mode as all items will be published by the command, e.g `call_command(events_relay.Command(), run_in_loop=True, loop_interval=0.1)`.  


### How to clean older events

You can use Jaiminho's [EventCleanerCommand](https://github.com/loadsmart/django-jaiminho/blob/master/jaiminho/management/commands/event_cleaner.py) in order to do that. It will query for all events that were sent before a given time interval (e.g. last 7 days) and will delete them from the outbox table.

The default time interval is `7 days`. You can use the `TIME_TO_DELETE` setting to change it. It should be added to `JAIMINHO_CONFIG` and must be a valid [timedelta](https://docs.python.org/3/library/datetime.html#timedelta-objects).

### Running as cron jobs

You can run those commands in a cron job. Here are some config examples:

```yaml
  - name: relay-failed-outbox-events
    schedule: "*/15 * * * *"
    suspend: false
    args:
      - ddtrace-run
      - python
      - manage.py
      - events_relay
    resources:
      requests:
        cpu: 1
      limits:
        memory: 384Mi

  - name: delete-old-outbox-events
    schedule: "0 5 * * *"
    suspend: false
    args:
      - ddtrace-run
      - python
      - manage.py
      - event_cleaner
    resources:
      requests:
        cpu: 1
      limits:
        memory: 384Mi
```

### Relay per stream and Overwrite publish strategy

Different streams can have different requirements. You can save separate events per streams by using the `@save_to_outbox_stream` decorator:

````python
@save_to_outbox_stream("my-stream")
def any_external_call(payload, **kwargs):
    # do something
    pass
````

you can also overwrite publish strategy configure on settings:

````python
@save_to_outbox_stream("my-stream", PublishStrategyType.KEEP_ORDER)
def any_external_call(payload, **kwargs):
    # do something
    pass
````

And then, run relay command with stream filter option
````shell
python manage.py relay_event True 0.1 my-stream
````

In the example above, `True` is the option for run_in_loop; `0.1` for loop_interval; and `my_stream` is the name of the stream.

### Signals

Jaiminho triggers the following Django signals:

| Signal                  | Description                                                                     |
|-------------------------|---------------------------------------------------------------------------------|
| event_published         | Triggered when an event is sent successfully                                    |
| event_failed_to_publish | Triggered when an event is not sent, being added to the Outbox table queue      |


### How to collect metrics from Jaiminho?

You could use the Django signals triggered by Jaiminho to collect metrics. 
Consider the following code as example:

````python
from django.dispatch import receiver

@receiver(event_published)
def on_event_sent(sender, event_payload, **kwargs):
    metrics.count(f"event_sent_successfully: {event_payload}")

@receiver(event_failed_to_publish)
def on_event_send_error(sender, event_payload, **kwargs):
    metrics.count(f"event_failed: {event_payload}")

````

### Jaiminho with Celery

Jaiminho can be very useful for adding reliability to Celery workflows. Writing to the database and enqueuing Celery tasks in the same workflow is very common in many applications, and this pattern can benefit greatly from the outbox pattern to ensure message delivery reliability.

Instead of configuring the `@save_to_outbox` decorator for every individual Celery task, you can integrate it at the Celery class level by overriding the `send_task` method, which is used by Celery to enqueue new tasks. This way, all tasks automatically benefit from the outbox pattern without requiring individual configuration.

Here's how to implement this:

```python
from celery import Celery
from jaiminho import save_to_outbox


class CeleryWithJaiminho(Celery):
    """
    Custom Celery class that inherits from Celery base class
    and adds Jaiminho functionality
    """
    
    @save_to_outbox
    def send_task(self, *args, **kwargs):
        """Send task with outbox pattern for reliability"""
        return super().send_task(*args, **kwargs)


app = CeleryWithJaiminho("tasks")
```

With this approach, all tasks sent through your Celery app will automatically use the outbox pattern, ensuring that task enqueuing is resilient to transient failures and network issues.

## Development

Create a virtualenv

```bash
virtualenv venv
pip install -r requirements-dev.txt
tox -e py39
```
## Collaboration

If you want to improve or suggest improvements, check our [CONTRIBUTING.md](https://github.com/loadsmart/django-jaiminho/blob/master/CONTRIBUTING.md) file.


## License

This project is licensed under MIT License.

## Security

If you have any security concern or report feel free to reach out to security@loadsmart.com;
