Metadata-Version: 2.4
Name: mail-deduplicate
Version: 9.5.0
Summary: 📧 CLI to deduplicate mails from mail boxes
Keywords: Babyl,CLI,dedup,dedupe,deduplicate,deduplication,deduplicator,duplicate-detection,duplicates,email,email-management,email-tools,EML,mail,mailbox,maildir,mbox,MH,MMDF,python
Author: Kevin Deldycke
Author-email: Kevin Deldycke <kevin@deldycke.com>
License-Expression: GPL-2.0-or-later
License-File: license
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: Operating System :: MacOS
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: 3.15
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Topic :: Communications :: Email
Classifier: Topic :: Utilities
Classifier: Typing :: Typed
Requires-Dist: click-extra[toml]>=10
Requires-Dist: extra-platforms>=14
Requires-Python: >=3.11
Project-URL: Changelog, https://github.com/kdeldycke/mail-deduplicate/blob/main/changelog.md
Project-URL: Documentation, https://kdeldycke.github.io/mail-deduplicate
Project-URL: Download, https://github.com/kdeldycke/mail-deduplicate/releases/tag/v9.5.0
Project-URL: Funding, https://github.com/sponsors/kdeldycke
Project-URL: Homepage, https://github.com/kdeldycke/mail-deduplicate
Project-URL: Issues, https://github.com/kdeldycke/mail-deduplicate/issues
Project-URL: Repository, https://github.com/kdeldycke/mail-deduplicate
Description-Content-Type: text/markdown

<p align="center">
  <a href="https://github.com/kdeldycke/mail-deduplicate/">
    <img src="https://raw.githubusercontent.com/kdeldycke/mail-deduplicate/main/docs/assets/mail-deduplicate-logo-header.png" alt="Mail Deduplicate">
  </a>
</p>

[![Last release](https://img.shields.io/pypi/v/mail-deduplicate.svg)](https://pypi.org/project/mail-deduplicate) [![Python versions](https://img.shields.io/pypi/pyversions/mail-deduplicate.svg)](https://pypi.org/project/mail-deduplicate) [![Downloads](https://static.pepy.tech/badge/mail-deduplicate/month)](https://pepy.tech/projects/mail-deduplicate) [![Unittests status](https://img.shields.io/github/actions/workflow/status/kdeldycke/mail-deduplicate/tests.yaml?branch=main&label=%F0%9F%94%AC%20Tests)](https://github.com/kdeldycke/mail-deduplicate/actions/workflows/tests.yaml?query=branch%3Amain) [![Documentation status](https://img.shields.io/github/actions/workflow/status/kdeldycke/mail-deduplicate/docs.yaml?branch=main&label=%F0%9F%93%9A%20Docs)](https://github.com/kdeldycke/mail-deduplicate/actions/workflows/docs.yaml?query=branch%3Amain) [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.7364256.svg)](https://doi.org/10.5281/zenodo.7364256)

**What is Mail Deduplicate?**

Provides the `mdedup` CLI, an utility to deduplicate mails from a set of boxes.

<p align="center">
  <img src="https://raw.githubusercontent.com/kdeldycke/mail-deduplicate/main/docs/assets/mdedup-header-cli.svg" alt="The mdedup command line and the first steps of a run merging two archives">
</p>

## Features

- Duplicate detection based on cherry-picked and normalized mail headers.
- Fetch mails from multiple sources.
- Reads and writes to `mbox`, `maildir`, `babyl`, `mh`, `mmdf` and `eml` formats.
- Deduplication strategies based on size, timestamp, file path or random choice, chainable as fallbacks.
- Copy, move, delete or hardlink the resulting set of duplicates.
- Dry-run mode.
- Protection against false-positives with safety checks on size and content differences. Two mails sharing every hashed header but not their body are left alone, while their real duplicate is acted on, here by `mdedup --strategy discard-matching-path --regexp old --action delete-discarded --dry-run new.mbox old.mbox`: ![A dry run deleting one duplicate and skipping a set of two different mails](https://raw.githubusercontent.com/kdeldycke/mail-deduplicate/main/docs/assets/mdedup-safeguard-cli.svg)
- Optional cache reusing the hashes computed by previous runs.
- Parallel hashing and selection over several processes.
- Supports macOS, Linux and Windows.
- [Standalone executables](#executables) for Linux, macOS and Windows.
- Shell auto-completion for Bash, Zsh and Fish.

## Installation

All [installation methods](https://kdeldycke.github.io/mail-deduplicate/install.html) are available in the documentation. Below are the most popular ones:

### Try it now

[`uv`](https://docs.astral.sh/uv/getting-started/installation/) is the fastest way to run `mdedup` on any platform, thanks to its [`uvx` command](https://docs.astral.sh/uv/guides/tools/#running-tools):

```shell-session
$ uvx --from mail-deduplicate -- mdedup
```

### macOS

`mdedup` is part of the official [Homebrew](https://brew.sh) default tap, so you can install it with:

```shell-session
$ brew install mail-deduplicate
```

### Executables

Standalone binaries of `mdedup`'s latest version are available as direct downloads for several platforms and architectures:

| Platform    | `arm64`                                                                                                                                | `x86_64`                                                                                                                           |
| ----------- | -------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| **Linux**   | [Download `mdedup-linux-arm64.bin`](https://github.com/kdeldycke/mail-deduplicate/releases/latest/download/mdedup-linux-arm64.bin)     | [Download `mdedup-linux-x64.bin`](https://github.com/kdeldycke/mail-deduplicate/releases/latest/download/mdedup-linux-x64.bin)     |
| **macOS**   | [Download `mdedup-macos-arm64.bin`](https://github.com/kdeldycke/mail-deduplicate/releases/latest/download/mdedup-macos-arm64.bin)     | [Download `mdedup-macos-x64.bin`](https://github.com/kdeldycke/mail-deduplicate/releases/latest/download/mdedup-macos-x64.bin)     |
| **Windows** | [Download `mdedup-windows-arm64.exe`](https://github.com/kdeldycke/mail-deduplicate/releases/latest/download/mdedup-windows-arm64.exe) | [Download `mdedup-windows-x64.exe`](https://github.com/kdeldycke/mail-deduplicate/releases/latest/download/mdedup-windows-x64.exe) |

## Quickstart

Duplicate mails pile up whenever boxes get copied around: backups of the same account taken at different times, per-folder exports where a mail tagged with several labels lands in several boxes, archives consolidated from multiple clients, or IMAP synchronizations gone wrong.

`mdedup` cleans this up. Point it at your boxes: it groups copies of the same mail by hashing a curated set of headers, applies a `--strategy` to pick which copy to keep in each group, then performs an `--action` on the result. The default action is the safest one: sources are only read, and the deduplicated selection is written to a brand new box.

So to merge two overlapping archives, `2024.mbox` and `2025.mbox`, into a single clean box, keeping the copy each mail has in the older one:

```shell-session
$ mdedup --strategy select-matching-path --regexp 2024 --export merged.mbox 2024.mbox 2025.mbox
```

Mails found in both archives are copied once into `merged.mbox`, along with all the mails that were unique to each source. Backups usually leave copies byte-identical, which no date or size strategy splits apart: `select-one` then keeps an arbitrary copy of each mail.

<p align="center">
  <img src="https://raw.githubusercontent.com/kdeldycke/mail-deduplicate/main/docs/assets/mdedup-run-cli.svg" alt="A whole mdedup run, ending on its two report tables">
</p>

To remove duplicates in place instead, switch to a destructive action, and rehearse with `--dry-run`:

```shell-session
$ mdedup --strategy select-matching-path --regexp 2024 --action delete-discarded --dry-run 2024.mbox 2025.mbox
```

The trail names each mail the run would delete, and alters nothing.

<p align="center">
  <img src="https://raw.githubusercontent.com/kdeldycke/mail-deduplicate/main/docs/assets/mdedup-dry-run-cli.svg" alt="A mdedup dry run on the two archives">
</p>

The [hands-on tutorial](https://kdeldycke.github.io/mail-deduplicate/tutorial.html) builds a small playground of duplicated mails, then walks through strategies, actions and safeguards on it. The [design page](https://kdeldycke.github.io/mail-deduplicate/design.html) explains how duplicates are detected.

> [!NOTE]
> Memory does not grow with the size of your boxes: each mail is reduced to a lightweight stub as soon as it is hashed, so a 215 MB maildir of 1,500 mails peaks at 46 MB. It does grow with their number, at roughly 1 KB of resident memory each, so a large enough collection can still exhaust your machine.
>
> Hashing is where a run spends most of its time, and `--cache` reuses it between runs. The [performance page](https://kdeldycke.github.io/mail-deduplicate/performance.html) covers both, and how to manage that cache.

You can support this project with pull requests, [business support 🤝](https://github.com/sponsors/kdeldycke) and [sponsorship 🫶](https://github.com/sponsors/kdeldycke).
