Metadata-Version: 2.4
Name: destiny-deet
Version: 0.5.0
Summary: Data Extraction Evaluation Toolkit (DEET): A suite of tools, data models, etc. for extracting data from documents (e.g. papers) and evaluating the performance of such extraction tasks
Author-email: Harry Moss <h.moss@ucl.ac.uk>, Max Callaghan <max.callaghan@pik-potsdam.de>, Nik Loynes <n.loynes@ucl.ac.uk>, Sagar Uprety <s.uprety@ucl.ac.uk>
License-Expression: Apache-2.0
Project-URL: Changelog, https://github.com/destiny-evidence/data-extraction-evaluation-toolkit/blob/main/CHANGELOG.md
Project-URL: Documentation, https://destiny-evidence.github.io/data-extraction-evaluation-toolkit/
Project-URL: Issues, https://github.com/destiny-evidence/data-extraction-evaluation-toolkit/issues
Project-URL: Repository, https://github.com/destiny-evidence/data-extraction-evaluation-toolkit
Keywords: data extraction,evaluation,evidence synthesis,llm
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: <3.14,>=3.12.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: clevercsv>=0.8.5
Requires-Dist: destiny-sdk[labs]>=0.9.1
Requires-Dist: diskcache>=5.6.3
Requires-Dist: inquirerpy>=0.3.4
Requires-Dist: jinja2>=3.1.6
Requires-Dist: litellm>=1.96.2
Requires-Dist: loguru>=0.7.3
Requires-Dist: openai>=1.0.0
Requires-Dist: pdfminer.six>=20221105
Requires-Dist: pillow>=10.4.0
Requires-Dist: pydantic-extra-types>=2.10.5
Requires-Dist: pydantic-settings>=2.10.1
Requires-Dist: pydantic>=2.11.7
Requires-Dist: python-dotenv>=1.1.1
Requires-Dist: pyyaml>=6.0.2
Requires-Dist: rapidfuzz>=3.9.0
Requires-Dist: scikit-learn>=1.7.1
Requires-Dist: tabulate>=0.9.0
Requires-Dist: typer>=0.21.2
Provides-Extra: lang-quality
Requires-Dist: nltk>=3.9.1; extra == "lang-quality"
Provides-Extra: parsers
Requires-Dist: marker-pdf>=1.9.1; extra == "parsers"
Requires-Dist: pypandoc>=1.15; extra == "parsers"
Provides-Extra: semantic
Requires-Dist: sentence-transformers>=5.5.1; extra == "semantic"
Dynamic: license-file

# data-extraction-evaluation-toolkit

The Data Extraction Evaluation Toolkit (DEET) is an open-source toolkit designed to support the development and evaluation of AI-assisted data extraction for evidence synthesis.

Extracting structured information from research studies is one of the most time-consuming stages of the evidence synthesis process. While large language models (LLMs) and other AI methods have the potential to reduce this workload, inaccurate extraction can introduce errors into reviews and ultimately affect evidence used for research and policy decisions. Robust evaluation is therefore essential to ensure that AI tools are reliable, transparent, and fit for purpose.

DEET provides an end-to-end, modular workflow for configuring, running and evaluating automated data extraction methods. It enables users to compare AI-generated outputs against human-created reference (gold standard) datasets, helping to assess the accuracy, consistency and performance of different approaches. Designed around open and FAIR (Findable, Accessible, Interoperable and Reusable) principles, DEET supports reproducible evaluation through reusable code, APIs and standardised data models.

DEET is intended for researchers, evidence synthesis teams, developers and organisations interested in evaluating or developing AI-enhanced evidence synthesis workflows. Users can expect structured extraction outputs, performance metrics and transparent evaluation results that support informed decisions about the use of AI in evidence synthesis.

Within DESTINY, DEET will be used to evaluate an AI-powered data extraction workflow for intervention studies. Researchers will compare data extracted automatically by a large language model with manually extracted reference data, identify where the AI performs well or makes errors, and use these results to improve extraction methods before they are incorporated into evidence synthesis workflows.

More detailed documentation can be found here [Docs](https://destiny-evidence.github.io/data-extraction-evaluation-toolkit/)

## Quickstart

`deet` is published on PyPI as [`destiny-deet`](https://pypi.org/project/destiny-deet/).

### To use the `deet` CLI

```sh
uv tool install destiny-deet
deet --help
```

### To use `deet` as a package

```sh
uv add destiny-deet  # or: pip install destiny-deet
```

Optional extras are available for document parsing (`destiny-deet[parsers]`) and text quality checks (`destiny-deet[lang-quality]`).

To install the latest unreleased code instead, replace `destiny-deet` with `git+https://github.com/destiny-evidence/data-extraction-evaluation-toolkit.git`.

## Using `deet`

The `data-extraction-evaluation-toolkit` (`deet`) contains mutliple modules which can be leveraged alone, or orchestrated together to form a `Pipeline`. The goal of `DEET` is to be modular and extensible, allowing users to customise a specific pipeline or workflow to their needs.

Typical pipelines can be run using the CLI app `deet --help`

## Contributing

If you want to contribute to this project -- awesome, everyone's welcome.
Please see the [contributing guidelines](https://github.com/destiny-evidence/data-extraction-evaluation-toolkit/blob/main/CONTRIBUTING.md) for details on how best to contribute.

A few important steps when contributing:

```sh
pre-commit install
uv run pre-commit install --hook-type commit-msg
```

This will force you to use [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/).

## Commit message format

All commits must use [**conventional commits**](https://www.conventionalcommits.org/en/v1.0.0/). The pre-commit hook will reject any commit that doesn't.

| prefix | example | version effect |
|---|---|---|
| `fix:` | `fix: handle null input` | patch: `0.1.0 -> 0.1.1` |
| `feat:` | `feat: add login page` | minor: `0.1.0 -> 0.2.0` |
| `feat!:` or `BREAKING CHANGE:` footer | `feat!: remove legacy api` | major: `0.1.0 -> 1.0.0` |
| `chore:`, `docs:`, `ci:`, `test:` | `chore: update deps` | no bump |

## Tests

Tests are written using `pytest`. You can run the tests locally using

```shell
pytest
```

Unit tests are automatically run in [Continuous Integration](https://en.m.wikipedia.org/wiki/Continuous_integration) (CI) using github actions (see `.github/workflows/tests.yml`) on Pull Requests or merges into `main` or `development`. Integration tests are also run for pushes/PRs into `main` (Note: these will take approx 1-2h to complete, so consider a cup of coffee while you wait).

## Adding documentation

Please add to the docs whenever you feel it would be useful. The docs are built using [mkdocs](https://www.mkdocs.org/) and mix automatically-generated API documentation with more general documentation. An automatically generated html static site is built from the `docs/` directory, and the API documentation is generated from docstrings in the code.

To add your own documentation, add markdown files to the `docs/` directory _and_ add these to the `nav` block in `mkdocs.yml`. To add API documentation, add docstrings to the code and ensure that the relevant modules are included in the `nav` block in `mkdocs.yml`.

To build the docs locally, make sure you have the docs dependencies installed by running

```sh
uv sync --all-extras --all-groups
```

which will install the documentation dependencies alongside all other dependencies, including developer dependencies. Alternatively,

```sh
uv sync --group docs
```

will install _only_ the documentation dependencies, but may uninstall other optional dependencies you have installed.

Then, from the root of the repository, run `mkdocs serve --strict` from the root of the repository and open the link that is printed to the terminal. The documentation website is currently automatically built and deployed to GitHub Pages on pushes to the `main` branch, and uses the `gh-pages` branch to serve the docs.

The documentation website is available at [https://destiny-evidence.github.io/deet](https://destiny-evidence.github.io/deet).

## Citation

If you use `deet` in your work, please cite it as:

> Finnerty Mutlu, A. N., Loynes, N., Uprety, S., Callaghan, M., Danilenko, D., Gopal, I., Hair, K., Molyneux, S., Moss, H., Repke, T., Schmidt, L., Soliman, N., Tezok, K. O. T. (Miyaka)., Vairon, J., & Thomas, J. (2026). Data Extraction Evaluation Toolkit (version 0.4.0-dev.2.) [Computer software]. Zenodo. <https://doi.org/10.5281/zenodo.22897649>

## Acknowledgements

We acknowledge with thanks funding from the following funders and projects:

- Digital Evidence Synthesis Tool INnovation for Yielding Improvements in Climate & Health (DESTINY) [313586/Z/24/Z]
- Mobilising Evidence Through Artificial Intelligence and User-Informed Synthesis (METIUS) [UKRI3167]
- Global Alliance for Living Evidence on aNxiety, depressiOn and pSychosis (GALENOS)
- Living Evidence in Alzheimer's and Dementia (LEAD) [227443/Z/23/Z]
- Education Endowment Foundation (EEF)
