Metadata-Version: 2.2
Name: unquotemail
Version: 1.0.0
Summary: Parse a given Html/Text email and return only the new text, without the quoted part.
Home-page: https://github.com/getfernand/unquotemail
Author: Cyril Nicodeme
Author-email: contact@cnicodeme.com
License: MIT
Project-URL: Source, https://github.com/getfernand/unquotemail
Keywords: mail email parse parser unquote reply eml
Platform: any
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Topic :: Communications :: Email
Classifier: Topic :: Internet :: WWW/HTTP :: Dynamic Content
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Utilities
Classifier: Topic :: Scientific/Engineering :: Interface Engine/Protocol Translator
Classifier: Topic :: Text Processing
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.3
Classifier: Programming Language :: Python :: 3.4
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: Implementation :: PyPy
Requires-Python: >=3.3, <4
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: beautifulsoup4>=4.13.3
Requires-Dist: Markdown>=3.7
Requires-Dist: html2text>=2024.2.26
Provides-Extra: dev
Requires-Dist: pytest>=8.3.4; extra == "dev"
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: keywords
Dynamic: license
Dynamic: platform
Dynamic: project-url
Dynamic: provides-extra
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# UnquoteMail

This library is intended to parse a given HTML and/or Text message and return only the new message without the previous conversation(s).

It is used in production at [Fernand](https://getfernand.com).

Parsing an email is quite difficult because of the amount of various mail providers and their specific approaches.
Unfortunately, there is no standard for separating the new content to the previous conversation so we need to rely on different tricks.

We took a progressive approach on parsing the document, in the following order:

1. We first try to identify and then remove all the known markup language, such as ".gmail_quote", ".protonmail_quote" and the likes
2. If we don't find it, we fallback on Regex to identify standard "On YYYY/MM/dd HH:mm:ss, bob <bob@example.com> wrote:" patterns

If we succeed at the point 1, we then re-generate the text data by converting the remaining HTML to markdown using the html2text library.

If we succeed at point 2, we parse the HTML (again) to locate the matched regex pattern from the text version. We then remove everything from that point (including the matched pattern) in the HTML structure.

If we fail to locate the pattern in the HTML structure, we re-create a new HTML by converting the text to HTML 

We allow ourselves to rewrite the text to HTML in that last resort because we consider that an email containing previous data is generally an human-written reply and not a marketing email with advanced structure, so the content should be basic to parse (link, bold/italic/underline, images and that's almost all; All of what a Markdown converter can do).

## Usage

The library is very straightforward:

```python
from unquotemail import Unquote

# You previously retrieved the text/html and text/plain version of an email

unquote = Unquote(email_html, email_text)
print(unquote.get_html())  # Will output the email without the included replies.
```

The `Unquote` class accepts 4 parameters:

 * html: A string containing the HTML data
 * text: A string containing the Text data
 * sender: (Optional) The message_id of the email, generally under the form <{hash}@{hostname}>
 * parse: (Optional) - Defaults to True. Will parse the message.

All the parameters are optional by default BUT you must past either a valid `html` or `text` value (otherwise it's kind of useless, isn't it?).

The `sender` parameter is not required and doesn't do anything for now, but it's possible in the future that we will rely on the sender to better parse an email. (A @yahoo.com email might help the parser better know what to do, and not lookup for a "gmail_quote" div for instance)

Finally, the `parse` boolean, if set to false, won't ... well, parse the email.

The reason for this is quite simple. Imagine the following:

```python
unquote = Unquote(email_html, email_text, parse=False)

if not is_new_email:
    # We don't unquote a new email as we want to keep the context
    # But for all following emails, we do want to remove that context since we already have it
    unquote.parse()

message = Message(unquote.get_html(), unquote.get_text())
message.save()  # in the database
```


## Special thanks

We used the regex from the following libraries to create our own.
Most of the regex patterns you see on UnquoteMail have been modified, but the root is from these two libraries:

 * [Talon (Mailgun)](https://github.com/mailgun/talon)
 * [Email Reply Parser (Crisp)](https://github.com/crisp-oss/email-reply-parser)

So, thank you to them!


## Testing

Our `test/` folder contains a suite of test.
Some of the files present in the test folder have been retrieved from Crisp and Talon (again, thanks) and we adjusted these to our test case.

To run the tests, do the following:

`pytest`

Or to run only one test:

`pytest -k "test_unquote[talon_9.html-talon_9.html]"`

**WARN**: For now, we only have 105 tests successfully passing for a total of 168 tests.
We need to continue a bit of work to improve the test suite
