Metadata-Version: 2.4
Name: imap-dispatcher
Version: 2.0.0
Summary: Rule-based IMAP mail dispatcher with AI classification
Author-email: Marek Stiefenhofer <sti@posteo.de>
License-Expression: MIT
Project-URL: Homepage, https://github.com/stiefenm/imap-dispatcher
Project-URL: Repository, https://github.com/stiefenm/imap-dispatcher
Project-URL: Issues, https://github.com/stiefenm/imap-dispatcher/issues
Project-URL: Changelog, https://github.com/stiefenm/imap-dispatcher/blob/main/CHANGELOG.md
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Console
Classifier: Intended Audience :: System Administrators
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Communications :: Email :: Filters
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: imapclient>=3.1
Requires-Dist: pyyaml>=6.0
Requires-Dist: mistralai<3,>=2.5
Requires-Dist: httpx>=0.28
Requires-Dist: html2text>=2024.2.26
Provides-Extra: dev
Requires-Dist: pytest>=9; extra == "dev"
Requires-Dist: pytest-mock>=3.15; extra == "dev"
Dynamic: license-file

# imap-dispatcher

[![PyPI](https://img.shields.io/pypi/v/imap-dispatcher)](https://pypi.org/project/imap-dispatcher/)
[![CI](https://github.com/stiefenm/imap-dispatcher/actions/workflows/ci.yml/badge.svg)](https://github.com/stiefenm/imap-dispatcher/actions/workflows/ci.yml)
[![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue)](https://www.python.org/)
[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)

Rule-based IMAP mail dispatcher with optional AI classification.

## Features

- **Three-tier matching** — substring rules first, then skip rules, then optional AI classification
- **AI classification via Mistral** — free-text rule descriptions, with graceful degradation if the API is unreachable
- **Auto-learning** — successful AI classifications are remembered so future mail from the same sender is matched instantly, without another AI call
- **Feedback loop** — adapts learned rules automatically when you manually move a mail the dispatcher sorted
- **Dry-run mode** — preview every move (and every learn) without changing anything
- **Environment-variable secrets** — override host/username/password/API key via env vars, 12-factor style
- **Single YAML config** — one file, no database, no external state beyond a small `state.json`
- **Minimal dependencies** — `imapclient`, `pyyaml`, `mistralai`, `httpx`, `html2text`
- **Python 3.10+** — runs anywhere, no Raspberry Pi requirement
- **UID-based state tracking** — safe alongside other IMAP clients (e.g. Thunderbird) working the same mailbox in parallel

## Quick Start

```bash
python3 -m venv .venv
source .venv/bin/activate
pip install imap-dispatcher

cp config.example.yaml config.yaml
chmod 600 config.yaml
# edit config.yaml: host, username, password, rules ...

imap-dispatcher --dry-run
```

### Install from source

```bash
git clone <repo-url>
cd imap-dispatcher

python3 -m venv .venv
source .venv/bin/activate
pip install .

cp config.example.yaml config.yaml
chmod 600 config.yaml
# edit config.yaml: host, username, password, rules ...

imap-dispatcher --dry-run
```

### Alternative install (venv copy)

If you prefer not to install the package (e.g. for a minimal deployment), copy the module directly and run it with `python -m`:

```bash
cp -r imap_dispatcher/ $INSTALL_DIR/
cp requirements.txt $INSTALL_DIR/
cd $INSTALL_DIR

python3 -m venv .venv
.venv/bin/pip install -r requirements.txt

cp config.example.yaml config.yaml
chmod 600 config.yaml

.venv/bin/python -m imap_dispatcher --config config.yaml --dry-run
```

Both invocation styles — `imap-dispatcher ...` (console script) and `python -m imap_dispatcher ...` (module) — work identically and are used interchangeably throughout this README.

## Configuration

All options live in `config.yaml` (template: `config.example.yaml`, fully commented).

| Key | Description | Default |
|-----|-------------|---------|
| `host` | IMAP server, e.g. `imap.example.com` | — (required) |
| `username` | Mail account username/address | — (required) |
| `password` | Password in plain text (see Security below) | — (required) |
| `inbox` | Source folder | `INBOX` |
| `state_file` | Path to the UID-tracking state file | `state.json` |
| `log_file` | Optional path to a log file (rotating, 1 MB, 3 backups); empty = stdout only | `""` |
| `rules` | List of sorting rules (see Rule fields below) | — |
| `skip_rules` | List of skip rules that exclude senders from AI classification | — |
| `ai` | AI classification block (see below) | — |
| `ai_language` | Language of the AI classification prompt (`en` \| `de`) | `en` |
| `private_domains` | Domains never auto-classified by AI (see below) | common freemail providers |

### AI classification (`ai:` block)

```yaml
ai:
  provider: mistral                 # currently supported: mistral
  api_key: ""                       # or set MISTRAL_API_KEY in the environment (env wins)
  model: "mistral-small-latest"     # model name, default is Mistral Small
```

- `provider` — only `mistral` is currently supported.
- `api_key` — your Mistral API key (create one at [console.mistral.ai](https://console.mistral.ai)); can also be provided via the `MISTRAL_API_KEY` environment variable.
- `model` — the Mistral model name; defaults to `mistral-small-latest`.

`ai_language` controls the language of the AI classification prompt: `en` (default) or `de`.

### Rule fields

Each rule under `rules:` supports:

| Field | Description |
|-------|-------------|
| `name` | Label used in logs |
| `target_folder` | Destination folder for matches |
| `from_contains` | List of substrings to match in the From address |
| `from_name_contains` | List of substrings to match in the From display name |
| `to_contains` | List of substrings to match in the To header |
| `subject_contains` | List of substrings to match in the Subject header |
| `mark_seen` | Optional: `true` to set the `\Seen` flag on move |
| `ai_description` | Free-text description for AI classification (optional; only applies if no substring matcher already matched) |

Every rule needs at least one matcher field. **The first matching rule wins.** Fields within a single rule are AND-ed (all must match); entries within a field are OR-ed (any one is enough). `skip` is only valid inside `skip_rules:` — it is rejected inside `rules:`.

### `private_domains`

Senders from these domains are treated as personal mail and are never auto-classified by AI. Defaults to common freemail providers (Gmail, Outlook, Yahoo, iCloud, Proton, and others); override the list in `config.yaml` if needed.

### Environment variables

12-factor style overrides (e.g. via a systemd `EnvironmentFile=` or a shell export): `IMAP_HOST`, `IMAP_USERNAME`, `IMAP_PASSWORD`, and `MISTRAL_API_KEY` override the corresponding values in `config.yaml` when set. Environment values always take precedence over YAML (env wins); an empty environment variable falls through to the YAML value.

### Security

This config file contains your mail password in plain text.

- Restrict permissions: `chmod 600 config.yaml`
- Even better: use a dedicated app password (if your provider offers one) instead of your main password
- Never commit `config.yaml` to a Git repo

## CLI Reference

Both invocation styles are shown below and behave identically: `imap-dispatcher ...` (after `pip install .`) and `python -m imap_dispatcher ...` (venv-copy path).

```bash
imap-dispatcher --config config.yaml [OPTIONS]
python -m imap_dispatcher --config config.yaml [OPTIONS]
```

| Flag | Description |
|------|-------------|
| `--version` | Print the version and exit |
| `--config PATH` | Path to config.yaml (default: `config.yaml`) |
| `--dry-run` | Only log moves, do not execute them |
| `--sort-existing` | Check all existing INBOX mail against the rules and sort it |
| `--sort-existing-ai` | Like `--sort-existing`, but including AI classification for mail without a substring match (opt-in, slower due to ~2.6s per mail) |
| `--limit N` | Process only the last N mails (for `--sort-existing` / `--sort-existing-ai`) |
| `--review-learned` | Show learned sender rules (audit) |

`--review-learned` reads `learned.yaml` from the config file's parent directory and prints an audit report, then exits — it does not connect to the mailbox.

## Advanced Topics

### Three-Tier Matching

Every incoming mail runs through up to three tiers until a target folder is found:

1. **Tier 1 — substring rules** — first matching rule wins. Fast, deterministic, no network call.
2. **Tier 2 — skip rules** — prevent the LLM call for known senders (e.g. known advertising without a matching rule). The mail stays in the INBOX.
3. **Tier 3 — AI classification** — only if Mistral was reachable at startup. On outage the dispatcher keeps running with substring rules only (graceful degradation).

```
incoming mail
     │
     ▼
Tier 1: substring rules ──match──► move + learn sender
     │ no match
     ▼
Tier 2: skip rules ──match──► stays in INBOX (no AI call)
     │ no match
     ▼
Tier 3: AI classification ──match──► move + learn sender
     │ no match / AI unreachable
     ▼
stays in INBOX

(later) feedback scan detects manual corrections to AI-classified moves
```

### UID Tracking & State

- **First run** — the dispatcher reads the current highest UID in the mailbox and stores it in `state.json`. No existing mail is sorted; only a baseline is set. From the next run on, only mail with a higher UID (newly arrived) is processed.
- **UIDVALIDITY reset** — if the IMAP server reports a new UIDVALIDITY (e.g. after a mailbox migration), the dispatcher discards the old state and sets a new baseline. Existing mail is again not sorted — only new mail.
- **Sorting existing mail** — `--sort-existing` checks all INBOX mail against the rules and moves matches. Combinable with `--dry-run` for a preview. Does not modify `state.json`.
- **Resetting state** — deleting `state.json` has the same effect as a first run. `state_file` is a config key (default `state.json`, resolved relative to the working directory).

### Auto-Learning

Every successful AI classification automatically learns the sender's full email address as a `from_contains` entry in a separate file `learned.yaml` (next to `config.yaml`, which is itself never modified). On the next run, tier 1 (substring matching) matches this sender directly — no more LLM call needed.

`learned.yaml` can be edited by hand: deleting a line removes the learned association again ("un-learn"). Use `--review-learned` to review all learned entries at a glance; output format is one line per entry:

```
from_addr -> rule_name (target_folder) [date]
```

Skip entries are shown as `INBOX (Skip)`.

**Guards:** mails without a sender address are never learned (poisoning protection); already-learned addresses are never written twice; `--dry-run` suppresses every write.

#### Feedback Loop

When you manually move a mail that the dispatcher had sorted, it notices and adapts. Moves are tracked for 7 days; on each run the dispatcher checks whether tracked mails are still where it put them. If you moved one to a different rule's folder, the learned sender is re-learned for that rule; moving it back to INBOX means the sender is skipped from now on; moving it to any other (unmapped) folder un-learns the sender.

**Technical details**

- Each AI-classified move is recorded as a `TrackedMove` (message ID, rule name, sender address, target folder, moved-at timestamp) in `feedback_tracking.json`, stored next to `config.yaml` and written atomically with mode `0600`.
- Tracking entries expire after **7 days**; corrections made later than that are silently ignored.
- Three correction outcomes:
  - **Re-learn** — the mail is found in another rule-mapped folder: the learned entry's rule is rewritten to the new rule.
  - **Skip** — the mail is moved back to INBOX: the learned entry becomes a skip entry, and the sender stays in INBOX from now on.
  - **Un-learn** — the mail is found in an unmapped folder: the learned entry is removed entirely.
- Corrections are logged with the `[CORRECTION]` tag.
- **Guards:** mails without a Message-ID are never tracked; a tracked mail found nowhere is silently discarded without touching `learned.yaml`; `--dry-run` logs the would-be correction but never writes.

## Scheduling

Two ways to run imap-dispatcher on a schedule, Linux only.

### systemd timer (recommended)

```bash
# 1. Create a dedicated, non-root system user
sudo useradd --system --no-create-home --shell /usr/sbin/nologin imap-dispatcher
sudo chown -R imap-dispatcher:imap-dispatcher $INSTALL_DIR

# 2. Copy the unit files
sudo cp imap-dispatcher.service /etc/systemd/system/
sudo cp imap-dispatcher.timer /etc/systemd/system/
```

The shipped `imap-dispatcher.service` hardcodes an install prefix (`/opt/imap-dispatcher`) in three lines — `WorkingDirectory=`, `ExecStart=`, and `ReadWritePaths=`. Edit those three paths to your `$INSTALL_DIR` before enabling the unit, e.g.:

```bash
sudo sed -i 's|/opt/imap-dispatcher|'"$INSTALL_DIR"'|g' /etc/systemd/system/imap-dispatcher.service
```

The unit also has a commented `EnvironmentFile=` hook for providing secrets via environment variables instead of `config.yaml` (see [Environment variables](#environment-variables) above) — `chmod 600` the referenced env file if you use it.

```bash
sudo systemctl daemon-reload
sudo systemctl enable --now imap-dispatcher.timer

# Check status and logs
systemctl status imap-dispatcher.timer
journalctl -u imap-dispatcher.service
```

The timer settings, quoted verbatim from `imap-dispatcher.timer`:

```ini
OnBootSec=2min
OnUnitActiveSec=1min
Persistent=true
```

`OnBootSec=2min` gives the network time to come up after boot. Because the service is `Type=oneshot` triggered by a timer, overlapping runs are impossible — the timer will not start a new run while the previous one is still active.

The unit ships a hardening block (`NoNewPrivileges`, `ProtectSystem=strict`, `ReadWritePaths`, and more). Check the applied score with:

```bash
systemd-analyze security imap-dispatcher.service
```

`TimeoutStartSec=600` covers the worst-case retry runtime (IMAP retries at 5/15/45s, plus buffer).

### cron (alternative)

A shorter alternative if you don't want to manage systemd units. Cron does not prevent overlapping runs by itself, so wrap the command in `flock -n`:

```cron
* * * * * /usr/bin/flock -n /var/lock/imap-dispatcher.lock $INSTALL_DIR/.venv/bin/imap-dispatcher --config $INSTALL_DIR/config.yaml
```

- `flock -n` skips a run while the previous one still holds the lock. systemd's oneshot+timer combination gives this for free; cron does not — and since the worst-case retry runtime (~10 minutes) can exceed the 1-minute interval, the lock matters. The kernel releases the lock automatically on process exit; no stale-lockfile cleanup needed.
- Use absolute paths — cron's `PATH` is minimal.
- Set `log_file` in `config.yaml`, since cron output has no journald integration.
- Env-var secrets can go in the crontab itself or a sourced env file.

## Troubleshooting

- **Config validation failures** print to stderr with the prefix `Config error:` followed by indented `- <detail>` lines, and exit with code 1. Example details: `'host' missing (required)`, `'target_folder' missing (required)`, `'state_file' directory not writable`, `unsupported ai provider '...' (supported: mistral)`.
- **`no such mailbox`** — the IMAP folder-hierarchy separator depends on your server (Posteo uses `.`, e.g. `INBOX.Newsletter`). Verify the exact folder name with `--dry-run` or your mail client before trusting a rule's `target_folder`.
- **AI unreachable at startup** — logs a warning and the dispatcher continues running with substring rules only (graceful degradation); no crash.
- **Retry behavior** — IMAP run-level retries use delays of 5/15/45 seconds (4 attempts total); Mistral 429 rate limits are retried with 2/5/15 second backoff.
- **File locations** — `learned.yaml` and `feedback_tracking.json` are stored next to `config.yaml` (derived from `--config`'s parent directory); `state_file` is a config key (default `state.json`, resolved relative to the working directory).
- `learned.yaml` and `feedback_tracking.json` are written with mode `0600` because they contain sender addresses.
