Metadata-Version: 2.5
Name: complydoc
Version: 0.8.0
Summary: Compare loaders, trace every step and evaluate what reaches your model. Catch hidden text and personal data, and gate it in CI.
Project-URL: Homepage, https://github.com/complydoc/complydoc
Project-URL: Repository, https://github.com/complydoc/complydoc
Project-URL: Changelog, https://github.com/complydoc/complydoc/blob/main/src/complydoc/CHANGELOG.md
Project-URL: Issues, https://github.com/complydoc/complydoc/issues
Author-email: Duarte Cardoso <caldasdcardoso@gmail.com>
License: MIT
License-File: LICENSE
Keywords: document-loaders,embeddings,ingestion,langchain,llamaindex,llm,observability,ocr,pdf,pii,prompt-injection,rag
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: Legal Industry
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing
Classifier: Topic :: Text Processing :: Markup :: HTML
Classifier: Topic :: Utilities
Classifier: Typing :: Typed
Requires-Python: <3.14,>=3.11
Requires-Dist: lxml>=5.0
Requires-Dist: numpy>=1.26
Requires-Dist: openpyxl>=3.1
Requires-Dist: pdfplumber>=0.11
Requires-Dist: pillow>=10.3
Requires-Dist: py3langid>=0.4.0
Requires-Dist: pydantic>=2.7
Requires-Dist: pypdf>=4.2
Requires-Dist: pypdfium2>=4.30
Requires-Dist: python-docx>=1.1
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.7
Requires-Dist: tiktoken>=0.7
Requires-Dist: typer>=0.12
Provides-Extra: assistant
Requires-Dist: langchain-openai>=0.3; extra == 'assistant'
Requires-Dist: langchain>=1.0; extra == 'assistant'
Provides-Extra: multilingual-names
Requires-Dist: torch>=2.2; extra == 'multilingual-names'
Requires-Dist: transformers>=4.40; extra == 'multilingual-names'
Provides-Extra: ner
Requires-Dist: preshed!=3.0.14; extra == 'ner'
Requires-Dist: spacy>=3.7; extra == 'ner'
Provides-Extra: notebook
Requires-Dist: pandas>=2.1; extra == 'notebook'
Provides-Extra: ocr
Requires-Dist: rapidocr-onnxruntime>=1.3; extra == 'ocr'
Provides-Extra: s3
Requires-Dist: boto3>=1.28; extra == 's3'
Provides-Extra: typesafe
Requires-Dist: langchain-typesafe>=0.0.1a3; extra == 'typesafe'
Description-Content-Type: text/markdown

<div align="center">
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/complydoc/complydoc/main/.github/images/logo-dark.svg">
    <img alt="complydoc" src="https://raw.githubusercontent.com/complydoc/complydoc/main/.github/images/logo-light.svg" width="42%">
  </picture>

  <h3>Get document loading right.</h3>
  <p>Compare loaders, trace every step and evaluate what reaches your model.</p>

  <a href="https://pypi.org/project/complydoc/"><img src="https://img.shields.io/pypi/v/complydoc?color=56b4e0&label=pypi" alt="PyPI"></a>
  <a href="https://github.com/complydoc/complydoc/actions/workflows/checks.yml"><img src="https://github.com/complydoc/complydoc/actions/workflows/checks.yml/badge.svg?branch=main" alt="Tests"></a>
  <a href="https://complydoc.github.io/complydoc/docs/"><img src="https://img.shields.io/badge/docs-complydoc-56b4e0" alt="Documentation"></a>
  <a href="https://opensource.org/licenses/MIT"><img src="https://img.shields.io/badge/license-MIT-56b4e0" alt="License"></a>
</div>

<br>

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/complydoc/complydoc/main/.github/images/viewer-trace-dark.webp">
  <img alt="The complydoc viewer: a LangChain ingestion pipeline traced step by step, each loader, splitter, vector store and embedding call on a time axis with its time, cost and the identifiers it passed on" src="https://raw.githubusercontent.com/complydoc/complydoc/main/.github/images/viewer-trace-light.webp" width="100%">
</picture>

A PDF can say one thing to a person and another to your model. White text, tiny type, an
IBAN behind a black bar: it's all still in the text layer, and most loaders pass it straight
through. complydoc shows what your loaders actually extracted, flags hidden text and
personal and financial identifiers before they're embedded, and traces every ingestion step
with its time and cost. Nothing leaves your machine.

Works with LangChain, LlamaIndex, Unstructured, Haystack, Docling, LlamaParse, Azure
Document Intelligence, Marker, Chonkie and the common vector stores and embedding APIs.

## Try it

```bash
uv tool install complydoc
complydoc ui ./documents
```

`complydoc ui` audits the folder and opens the viewer on it, served from your machine. To
see a hidden clause caught, try the one-page contract in
[`examples/hidden_clause.pdf`](examples/hidden_clause.pdf).

## Observe a pipeline

```python
import complydoc as cd

with cd.observe("contracts-ingest"):
    documents = PyMuPDF4LLMLoader("contract.pdf").load()
    chunks = splitter.split_documents(documents)
    vector_store.add_documents(chunks)
```

Every loader, splitter, embedding call and vector store becomes a step, with its settings,
time, tokens, cost and the identifiers it passed on. Open it with `complydoc ui`.

## What it shows

- **Traces** of each ingestion step, and how runs compare over time.
- **Loader diffs**: each loader's reading of a page side by side, with the page itself.
- **Identifiers** from Europe, the Americas, India and Australia, checksum-validated and
  masked until you choose to see them.
- **Hidden content**: text a reader does not see and a model does, and instructions
  written to a model.
- **Evals**: the facts a document must keep, checked on every reading.
- **Chunks and cost**: splitters side by side, and tokens priced across models.

## In CI

```yaml
- uses: complydoc/complydoc@v0.8.0
  with:
    path: documents
    policy: policy.yaml
```

`complydoc check` fails when a folder breaks your rules; `finance`, `legal` and `hr` ship as
ready policies. See [GitHub Action](https://complydoc.github.io/complydoc/docs/guides/github-action/).

## More

- [Documentation](https://complydoc.github.io/complydoc/docs/), and the
  [playground](https://github.com/complydoc/playground) with everything ready to run
- [Run it for your team](https://complydoc.github.io/complydoc/docs/guides/team/), in a
  [container](https://complydoc.github.io/complydoc/docs/guides/container/) or from a bucket
- [Network isolation](https://complydoc.github.io/complydoc/docs/explanation/offline/): every
  run blocks outbound connections and records that it did
- Optional extras for scans and names: `complydoc doctor` says what is installed and how to
  add the rest
- [Changelog](https://github.com/complydoc/complydoc/blob/main/src/complydoc/CHANGELOG.md) ·
  [Contributing](https://github.com/complydoc/complydoc/blob/main/CONTRIBUTING.md) · MIT

If complydoc is useful to you, a star helps other engineers find it.
