Metadata-Version: 2.4
Name: docscrub
Version: 0.1.0
Summary: A lightweight Python library to clean raw text and documents before NLP and LLM workflows.
Author: Saim Chaudhary
License: MIT License
        
        Copyright (c) 2026 DocScrub
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        MIT License
        
        Copyright (c) 2026 Saim Ch
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Requires-Python: >=3.10
Requires-Dist: ftfy>=6.3.1
Description-Content-Type: text/markdown

# DocScrub

[![Python](https://img.shields.io/badge/Python-3.14+-3776AB?style=for-the-badge&logo=python&logoColor=white)](https://www.python.org/)
[![pytest](https://img.shields.io/badge/pytest-9.1.1-0A9EDC?style=for-the-badge&logo=pytest&logoColor=white)](https://docs.pytest.org/)
[![Ruff](https://img.shields.io/badge/Ruff-0.16.0-D7FF64?style=for-the-badge&logo=ruff&logoColor=111111)](https://docs.astral.sh/ruff/)
[![ftfy](https://img.shields.io/badge/ftfy-text%20fixing-444444?style=for-the-badge)](https://ftfy.readthedocs.io/)

DocScrub is a lightweight Python library for cleaning raw text before NLP, LLM, and RAG workflows.

## Features

- Unicode fixing
- URL removal
- Email removal
- Page number removal
- Duplicate line removal
- Whitespace cleanup

## Installation

```bash
pip install docscrub
```

## Quick Start

```python
from docscrub import Cleaner

cleaner = Cleaner()

raw_text = "FranÃ§ais https://example.com user@example.com\n\nHello\nHello"
clean_text = cleaner.clean(raw_text)

print(clean_text)
```

## Examples

Input:

```text
Visit https://example.com for details
```

Output:

```text
Visit for details
```

Input:

```text
Hello
Hello
World
```

Output:

```text
Hello
World
```

## Supported Cleaners

### Unicode Cleaner

Fixes broken text encoding and normalizes Unicode characters. DocScrub uses `ftfy` for this step.

### URL Cleaner

Removes common URLs such as `https://example.com`.

### Email Cleaner

Removes email addresses such as `user@example.com`.

### Page Number Cleaner

Removes page-number-only lines such as `Page 1 of 10`, `Page 5`, or `12`.

### Duplicate Line Cleaner

Removes repeated lines while keeping the first copy.

### Whitespace Cleaner

Removes extra spaces, tabs, blank lines, and line padding.

## Project Structure

- `docscrub/` contains the library code.
- `docscrub/cleaners/` contains one simple function per cleaning step.
- `docscrub/patterns/` stores regular expression patterns shared by cleaners.
- `docscrub/utils/` is available for small helper functions.
- `tests/` contains beginner-friendly pytest tests.
- `examples/` contains short example scripts.

## Development Setup

Install the project dependencies with uv, then run the test and lint commands:

```bash
uv sync
uv run pytest
uv run ruff check .
uv run ruff format .
```

## Contributing

Beginner-friendly contributions are welcome.

1. Clone the repository.
2. Install the dependencies with `uv sync`.
3. Run `uv run pytest` before and after your change.
4. Keep the code simple and easy to read.
5. Open a pull request with a short description of what changed.

## License

DocScrub is released under the MIT License. See the [LICENSE](LICENSE) file for details.