Metadata-Version: 2.4
Name: docculus
Version: 0.0.1a1
Dynamic: Maintainer
Dynamic: Maintainer-email
Summary: library for LangChain documents
Project-URL: Homepage, https://github.com/durandtibo/docculus
Project-URL: Repository, https://github.com/durandtibo/docculus
Project-URL: Documentation, https://durandtibo.github.io/docculus/
Project-URL: Changelog, https://github.com/durandtibo/docculus/releases
Project-URL: Issues, https://github.com/durandtibo/docculus/issues
Author-email: Thibaut Durand <durand.tibo+gh@gmail.com>
License-Expression: BSD-3-Clause
License-File: LICENSE
Keywords: document
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: BSD License
Classifier: Operating System :: MacOS
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries
Requires-Python: >=3.10
Requires-Dist: coola<2.0,>=1.1.10a1
Requires-Dist: langchain-core<2.0,>=1.4
Provides-Extra: all
Requires-Dist: duckdb<2.0,>=1.3; extra == 'all'
Requires-Dist: faker<41.0,>=40.0; extra == 'all'
Requires-Dist: persista<1.0,>=0.0.3a5; extra == 'all'
Requires-Dist: rich<16.0,>=14.0.0; extra == 'all'
Provides-Extra: duckdb
Requires-Dist: duckdb<2.0,>=1.3; extra == 'duckdb'
Provides-Extra: faker
Requires-Dist: faker<41.0,>=40.0; extra == 'faker'
Provides-Extra: persista
Requires-Dist: persista<1.0,>=0.0.3a5; extra == 'persista'
Provides-Extra: rich
Requires-Dist: rich<16.0,>=14.0.0; extra == 'rich'
Description-Content-Type: text/markdown

# docculus

<p align="center">
    <a href="https://github.com/durandtibo/docculus/actions/workflows/ci.yaml">
        <img alt="CI" src="https://github.com/durandtibo/docculus/actions/workflows/ci.yaml/badge.svg">
    </a>
    <a href="https://github.com/durandtibo/docculus/actions/workflows/nightly-tests.yaml">
        <img alt="Nightly Tests" src="https://github.com/durandtibo/docculus/actions/workflows/nightly-tests.yaml/badge.svg">
    </a>
    <a href="https://github.com/durandtibo/docculus/actions/workflows/nightly-package.yaml">
        <img alt="Nightly Package Tests" src="https://github.com/durandtibo/docculus/actions/workflows/nightly-package.yaml/badge.svg">
    </a>
    <a href="https://codecov.io/gh/durandtibo/docculus">
        <img alt="Codecov" src="https://codecov.io/gh/durandtibo/docculus/branch/main/graph/badge.svg">
    </a>
    <br/>
    <a href="https://durandtibo.github.io/docculus/">
        <img alt="Documentation" src="https://github.com/durandtibo/docculus/actions/workflows/docs.yaml/badge.svg">
    </a>
    <a href="https://durandtibo.github.io/docculus/dev/">
        <img alt="Documentation" src="https://github.com/durandtibo/docculus/actions/workflows/docs-dev.yaml/badge.svg">
    </a>
    <br/>
    <a href="https://github.com/psf/black">
        <img  alt="Code style: black" src="https://img.shields.io/badge/code%20style-black-000000.svg">
    </a>
    <a href="https://google.github.io/styleguide/pyguide.html#s3.8-comments-and-docstrings">
        <img  alt="Doc style: google" src="https://img.shields.io/badge/%20style-google-3666d6.svg">
    </a>
    <a href="https://github.com/astral-sh/ruff">
        <img src="https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json" alt="Ruff" style="max-width:100%;">
    </a>
    <a href="https://github.com/guilatrova/tryceratops">
        <img  alt="try/except style: tryceratops" src="https://img.shields.io/badge/try%2Fexcept%20style-tryceratops%20%F0%9F%A6%96%E2%9C%A8-black">
    </a>
    <br/>
    <a href="https://pypi.org/project/docculus/">
        <img alt="PYPI version" src="https://img.shields.io/pypi/v/docculus">
    </a>
    <a href="https://pypi.org/project/docculus/">
        <img alt="Python" src="https://img.shields.io/pypi/pyversions/docculus.svg">
    </a>
    <a href="https://opensource.org/licenses/BSD-3-Clause">
        <img alt="BSD-3-Clause" src="https://img.shields.io/pypi/l/docculus">
    </a>
    <br/>
    <a href="https://pepy.tech/project/docculus">
        <img  alt="Downloads" src="https://static.pepy.tech/badge/docculus">
    </a>
    <a href="https://pepy.tech/project/docculus">
        <img  alt="Monthly downloads" src="https://static.pepy.tech/badge/docculus/month">
    </a>
    <br/>
</p>

## Overview

`docculus` is a lightweight Python library that makes it easy to analyze, transform, hash, and
validate [
`langchain_core.documents.Document`](https://python.langchain.com/api_reference/core/documents/langchain_core.documents.base.Document.html)
objects.

**Quick Links:**

- [Documentation](https://durandtibo.github.io/docculus/)
- [Installation](#installation)
- [Features](#features)
- [Contributing](#contributing)
- [License](#license)

## Why docculus?

Working with collections of LangChain documents often means writing the same boilerplate over and
over: counting duplicates, filtering by metadata, formatting documents into an LLM-friendly
prompt, or hashing content to detect changes. `docculus` packages these operations into a small,
well-tested API:

**Deduplicate and format documents for a prompt:**

```pycon
>>> from langchain_core.documents import Document
>>> from docculus.transform import deduplicate_documents, format_documents
>>> docs = [
...     Document(id="1", page_content="The cat sat on the mat."),
...     Document(id="2", page_content="The cat sat on the mat."),
...     Document(id="3", page_content="The dog chased the ball."),
... ]
>>> unique_docs = deduplicate_documents(docs)
>>> print(format_documents(unique_docs, output_format="markdown"))

```

**Inspect a corpus at a glance:**

```pycon
>>> from docculus.analysis import compute_content_stats_exact
>>> stats = compute_content_stats_exact(docs)
>>> stats["count"], stats["duplicate_count"]
(3, 1)

```

**Check consistency across documents sharing an id:**

```pycon
>>> from docculus.validation import validate_document_consistency
>>> validate_document_consistency(docs)

```

## Features

`docculus` provides a comprehensive set of utilities for working with `Document` objects:

### 📊 **Analysis**

Corpus-wide, read-only inspection utilities:

- Content statistics (exact and approximate) with `compute_content_stats_exact()` and
  `compute_content_stats_approx()`
- Metadata statistics with `compute_metadata_stats()`
- Duplicate and empty document detection with `find_duplicate_document_ids()`,
  `find_empty_documents()`, and `find_empty_document_ids()`
- Human-readable report printing with `print_content_stats_report()` and
  `print_metadata_stats_report()`

### 🔄 **Transform**

Utilities that produce a new document list:

- Deduplicate documents with `deduplicate_documents()`
- Filter by metadata with `filter_by_metadata()`, `filter_by_metadata_range()`, and
  `filter_by_metadata_values()`
- Sort with `sort_by_metadata()` and truncate with `truncate_documents()`
- Assign or copy ids with `assign_ids()` and `copy_ids_to_metadata()`
- Format documents into LLM-friendly strings (XML, Markdown, JSON) with `format_documents()`

### 📄 **Document**

Per-document utilities:

- Id generation with `generate_id()`, `generate_random_id()`, and `generate_deterministic_id()`
- Emptiness checks with `is_empty()` and `is_whitespace_only()`
- Length helpers with `get_length()`, `get_lengths()`, `get_longest_document()`, and
  `get_shortest_document()`

### #️⃣ **Hashing**

Deterministic hashing of documents:

- Hash a document or a sequence of documents with `hash_document()` and `hash_documents()`
- Generate a stable UUID from a document with `hash_document_to_uuid()`
- Register custom hashing strategies with `register_document_hasher()`

### ✅ **Validation**

Consistency checks across documents sharing an id, via `validate_document_consistency()`.

### 🖨️ **Display**

Pretty-print documents and their metadata to the terminal with `print_document()`,
`print_documents()`, and `print_documents_metadata()`.

### 🗄️ **Store**

Persist and retrieve documents with a common `BaseDocumentStore` interface, with
`InMemoryDocumentStore`, `SQLiteDocumentStore`, and `DuckDBDocumentStore` implementations
(plus typed variants).

## Installation

We highly recommend installing
`docculus` in
a [virtual environment](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/)
to avoid dependency conflicts.

### Using uv (recommended)

[`uv`](https://docs.astral.sh/uv/) is a fast Python package installer and resolver:

```shell
uv pip install docculus
```

**Install with all optional dependencies:**

```shell
uv pip install docculus[all]
```

**Install with specific optional dependencies:**

```shell
uv pip install docculus[duckdb,rich]  # with DuckDB and rich
```

### Using pip

Alternatively, you can use `pip`:

```shell
pip install docculus
```

**Install with all optional dependencies:**

```shell
pip install docculus[all]
```

### Requirements

- **Python**: 3.10 or higher
- **Core dependencies**: [`coola`](https://github.com/durandtibo/coola),
  [`langchain-core`](https://python.langchain.com/api_reference/core/index.html)

**Optional dependencies** (install with `docculus[all]`):
[DuckDB](https://duckdb.org/) •
[Faker](https://faker.readthedocs.io/) •
[persista](https://github.com/durandtibo/persista) •
[rich](https://rich.readthedocs.io/)

### Compatibility Matrix

| `coola` | `coola`         | `langchain-core` | `duckdb`<sup>*</sup> | `faker`<sup>*</sup> | `persista`<sup>*</sup> | `rich`<sup>*</sup> | `python` |
|---------|-----------------|------------------|----------------------|---------------------|------------------------|--------------------|----------|
| `main`  | `>=1.1.10,<1.0` | `>=1.4,<2.0`     | `>=1.3,<2.0`         | `>=40.0,<41.0`      | `>=0.0.3,<1.0`         | `>=14.0.0,<16.0`   | `>=3.10` |

## Contributing

Contributions are welcome! We appreciate bug fixes, feature additions, documentation improvements,
and more. Please check the [contributing guidelines](CONTRIBUTING.md) for details on:

- Setting up the development environment
- Code style and testing requirements
- Submitting pull requests

Whether you're fixing a bug or proposing a new feature, please open an issue first to discuss
your changes.

## API Stability

:warning: **Important**: As `docculus` is under active development, its API is not yet stable and
may
change between releases. We recommend pinning a specific version in your project’s dependencies to
ensure consistent behavior.

## License

`docculus` is licensed under BSD 3-Clause "New" or "Revised" license available in [LICENSE](LICENSE)
file.
