Metadata-Version: 2.4
Name: docling-eval
Version: 0.6.0
Summary: Evaluation of Docling
Author-email: Peter Staar <taa@zurich.ibm.com>, Panos Vagenas <pva@zurich.ibm.com>, Ahmed Nassar <ahn@zurich.ibm.com>, Nikos Livathinos <nli@zurich.ibm.com>, Maxim Lysak <mly@zurich.ibm.com>, Michele Dolfi <dol@zurich.ibm.com>, Christoph Auer <cau@zurich.ibm.com>
License-Expression: MIT
Project-URL: homepage, https://github.com/docling-project/docling-eval
Project-URL: repository, https://github.com/docling-project/docling-eval
Project-URL: issues, https://github.com/docling-project/docling-eval/issues
Project-URL: changelog, https://github.com/docling-project/docling-eval/blob/main/CHANGELOG.md
Keywords: docling,evaluation,convert,document,pdf,docx,html,markdown,layout model,segmentation,table structure,table former
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: POSIX :: Linux
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Programming Language :: Python :: 3
Requires-Python: <4.0,>=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: docling[vlm]<3.0.0,>=2.37.0
Requires-Dist: docling-core>=2.39.0
Requires-Dist: pydantic<3.0.0,>=2.0.0
Requires-Dist: lxml<6.0.0,>=5.3.0
Requires-Dist: datasets<4.0.0,>=3.2.0
Requires-Dist: apted<2.0.0,>=1.0.3
Requires-Dist: Distance<0.2.0,>=0.1.3
Requires-Dist: matplotlib<4.0.0,>=3.10.0
Requires-Dist: torch<3.0.0,>=2.5.1
Requires-Dist: torchmetrics<2.0.0,>=1.6.0
Requires-Dist: pycocotools<3.0.0,>=2.0.8
Requires-Dist: tabulate<0.10.0,>=0.9.0
Requires-Dist: tqdm<5.0.0,>=4.67.1
Requires-Dist: pillow<12.0.0,>=10.3.0
Requires-Dist: evaluate<0.5.0,>=0.4.3
Requires-Dist: nltk<4.0.0,>=3.9.1
Requires-Dist: beautifulsoup4<5.0.0,>=4.12.3
Requires-Dist: jiwer<4.0.0,>=3.1.0
Requires-Dist: urllib3<2.0.0,>=1.24.2
Requires-Dist: pandas<3.0.0,>=2.2.3
Requires-Dist: openpyxl<4.0.0,>=3.1.5
Requires-Dist: pypdf2<4.0.0,>=3.0.1
Requires-Dist: xmltodict<0.15.0,>=0.14.2
Requires-Dist: edit-distance<2.0.0,>=1.0.6
Requires-Dist: mlx-vlm<0.2; sys_platform == "darwin" and platform_machine == "arm64"
Requires-Dist: pyarrow>=20.0.0
Requires-Dist: typer>=0.16.0
Provides-Extra: hyperscalers
Requires-Dist: azure-ai-documentintelligence<2.0.0,>=1.0.2; extra == "hyperscalers"
Requires-Dist: azure-common<2.0.0,>=1.1.28; extra == "hyperscalers"
Requires-Dist: azure-core<2.0.0,>=1.33.0; extra == "hyperscalers"
Requires-Dist: boto3<2.0.0,>=1.37.8; extra == "hyperscalers"
Requires-Dist: google-cloud-documentai<4.0.0,>=3.2.0; extra == "hyperscalers"
Requires-Dist: ibm-cos-sdk<3.0.0,>=2.1.40; extra == "hyperscalers"
Provides-Extra: cvat-tools
Requires-Dist: cvat-sdk>=2.40.0; extra == "cvat-tools"
Requires-Dist: ocrmac>=1.0.0; (sys_platform == "darwin" and platform_machine == "arm64") and extra == "cvat-tools"
Dynamic: license-file

<p align="center">
  <a href="https://github.com/docling-project/docling-eval">
    <img loading="lazy" alt="Docling" src="docs/assets/docling-eval-pic.png" width="40%"/>
  </a>
</p>

# Docling-eval


[![arXiv](https://img.shields.io/badge/arXiv-2408.09869-b31b1b.svg)](https://arxiv.org/abs/2408.09869)
[![PyPI version](https://img.shields.io/pypi/v/docling-eval)](https://pypi.org/project/docling-eval/)
[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/docling-eval)](https://pypi.org/project/docling-eval/)
[![uv](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/uv/main/assets/badge/v0.json)](https://github.com/astral-sh/uv)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![Imports: isort](https://img.shields.io/badge/%20imports-isort-%231674b1?style=flat&labelColor=ef8336)](https://pycqa.github.io/isort/)
[![Pydantic v2](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/pydantic/pydantic/main/docs/badge/v2.json)](https://pydantic.dev)
[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white)](https://github.com/pre-commit/pre-commit)
[![License MIT](https://img.shields.io/github/license/docling-project/docling-eval)](https://opensource.org/licenses/MIT)

Evaluate [Docling](https://github.com/docling-project/docling) on various datasets.

## Features

Evaluate docling on various datasets. You can use the cli

```shell
terminal %> docling-eval --help
                                                                                                                                                                                                                                                
 Usage: docling_eval [OPTIONS] COMMAND [ARGS]...                                                                                                                                                                                                
                                                                                                                                                                                                                                                
 Docling Evaluation CLI for benchmarking document processing tasks.                                                                                                                                                                             
                                                                                                                                                                                                                                                
╭─ Options ────────────────────────────────────────────────────────────────────────────╮
│ --help          Show this message and exit.                                          │
╰──────────────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ───────────────────────────────────────────────────────────────────────────╮
│ create        Create both ground truth and evaluation datasets in one step.          │
│ create-eval   Create evaluation dataset from existing ground truth.                  │
│ create-gt     Create ground truth dataset only.                                      │
│ evaluate      Evaluate predictions against ground truth.                             │
│ visualize     Visualize evaluation results.                                          │
╰──────────────────────────────────────────────────────────────────────────────────────╯


```

## Benchmarks

- General
    - [DP-Bench benchmarks](docs/DP-Bench_benchmarks.md): Text, layout, reading order and table structure evaluation on the DP-Bench dataset.
    - [OmniDocBench benchmarks](docs/OmniDocBench_benchmarks.md): Text, layout, reading order and table structure evaluation on the OmniDocBench dataset.
- Layout
    - [DocLayNetV1 Benchmarks](docs/DocLayNetv1_benchmarks.md): Text and layout evaluation on the DocLayNet v1.2 dataset.
- Table-Structure
    - [FinTabnet Benchmarks](docs/FinTabNet_benchmarks.md): Table structure evaluation on the FinTabNet dataset.
    - [PubTabNet benchmarks](docs/PubTabNet_benchmarks.md): Table structure evaluation on the PubTabNet dataset.
    - [Pub1M benchmarks](docs/P1M_benchmarks.md): Table structure evaluation on the Pub1M dataset.

On our list for next benchmarks:

- [OmniOCR](getomni-ai/ocr-benchmark)
- Hyperscalers
- [CoMix](https://github.com/emanuelevivoli/CoMix/tree/main/docs/datasets)
- [DocVQA](https://huggingface.co/datasets/lmms-lab/DocVQA)
- [rd-tablebench](https://huggingface.co/datasets/reducto/rd-tablebench)
- [BigDocs-Bench](https://huggingface.co/datasets/ServiceNow/BigDocs-Bench)
  
## Contributing

Please read [Contributing to Docling](https://github.com/docling-project/docling/blob/main/CONTRIBUTING.md) for details.


## License

The Docling codebase is under MIT license.
For individual model usage, please refer to the model licenses found in the original packages.


## IBM ❤️ Open Source AI

Docling-eval has been brought to you by IBM.
