Metadata-Version: 2.4
Name: iomeval
Version: 0.0.2
Summary: IOM evaluation assistant
Home-page: https://github.com/franckalbinet/iomeval
Author: Solveit
Author-email: nobody@fast.ai
License: Apache Software License 2.0
Keywords: nbdev jupyter notebook python
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Natural Language :: English
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: License :: OSI Approved :: Apache Software License
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: fastcore
Requires-Dist: pandas
Requires-Dist: lisette
Requires-Dist: pydantic
Requires-Dist: tiktoken
Requires-Dist: mistocr
Provides-Extra: dev
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: keywords
Dynamic: license
Dynamic: license-file
Dynamic: provides-extra
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# IOMEval


<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->

[![CI](https://github.com/franckalbinet/iomeval/actions/workflows/test.yaml/badge.svg)](https://github.com/franckalbinet/iomeval/actions/workflows/test.yaml)
[![PyPI](https://img.shields.io/pypi/v/iomeval.png)](https://pypi.org/project/iomeval/)

`iomeval` streamlines the mapping of [IOM](https://www.iom.int)
evaluation reports against strategic frameworks like the [Strategic
Results Framework (SRF)](https://srf.iom.int/) and [Global Compact for
Migration
(GCM)](https://www.un.org/en/development/desa/population/migration/generalassembly/docs/globalcompact/A_RES_73_195.pdf).
It uses LLMs to process PDF reports, extract key sections, and tag/map
them to framework components, turning dispersed, untagged evaluation
documents into structured, searchable knowledge maps.

## Why This Matters

UN agencies produce extensive evaluation reports and other public
documents. For IOM, this body of knowledge is extensive and variegated,
but putting it to practical use becomes more challenging as volume
increases, particularly when documentation is stored across different
repositories with no single index available.

**The Challenge for IOM**

IOM’s evaluation production is highly decentralized, with reports stored
across multiple repositories (the [IOM Evaluation
Repository](https://evaluation.iom.int/evaluation-search-pdf), IOM
Library, IOM Protection Platform). Quality varies greatly—quality
control processes are not applied uniformly, and variation also reflects
the inherent subjectivity in evaluation approaches and interpretations.
Reports also vary significantly in structure: some follow common formats
with executive summaries, findings, and recommendations, while others
have different structures entirely. This inconsistency makes systematic
mapping challenging.

Critically, existing metadata doesn’t indicate which elements of IOM’s
strategic frameworks—the [Strategic Results Framework
(SRF)](https://srf.iom.int/) or the [Global Compact for Migration
(GCM)](https://www.un.org/en/development/desa/population/migration/generalassembly/docs/globalcompact/A_RES_73_195.pdf)—each
report addresses. This is a major gap that limits the ability to connect
evaluation evidence with key strategic frameworks.

**Evidence Maps as a Solution**

Evidence Maps display the extent and nature of research and evaluation
available on a subject. Following the 2025 UNEG Eval Week, four primary
use cases emerged: guiding future evidence generation, informing policy
decisions, knowledge management, and enhancing collaboration. The maps
created by `iomeval` serve primarily as **knowledge management
tools**—structured repositories that make identifying relevant sources
easier by organizing them against strategic framework components.

**What This Enables**

By tagging reports against SRF outputs, enablers, cross-cutting
priorities, and GCM objectives, these maps help answer questions like:
Which framework elements are well-covered by existing evaluations? Where
are the knowledge gaps that should prioritize future evaluation work?
Which themes have enough evidence for a dedicated synthesis report?

## Installation

``` sh
pip install iomeval
```

## Configuration

iomeval uses Claude for intelligent extraction and mapping. Set your
Anthropic API key:

``` sh
export ANTHROPIC_API_KEY='your-key-here'
```

add mistral also …

## Key Features

- **Automated PDF Processing**: Download and OCR evaluation reports with
  proper heading hierarchy
- **Intelligent Section Extraction**: LLM-powered extraction of
  executive summaries, findings, conclusions, and recommendations
- **Strategic Framework Mapping**: Map report content to IOM’s SRF
  Enablers, Cross-Cutting Priorities, GCM Objectives, and SRF Outputs
- **Checkpoint/Resume**: Built-in state persistence - interrupt and
  resume long-running pipelines
- **Granular Control**: Use the full pipeline or individual components
  as needed

## Quick Start

Process an evaluation report end-to-end:

``` python
from iomeval.pipeline import Report, run_pipeline

report = Report(id="IOM-2024-001", pdf_url="evaluation_report.pdf")
run_pipeline(report)
```

The rich display shows processing status across all pipeline stages -
from OCR through framework mapping.

## Detailed Workflow

For more control, use individual pipeline stages:

### 1. Load evaluation metadata

``` python
from iomeval.readers import read_iom

evals = read_iom()  # Returns DataFrame of all IOM evaluations
evals.head()
```

### 2. Download and OCR a report

``` python
from iomeval.downloaders import download_pdf
from iomeval.core import pdf_to_md

# Download PDF
pdf_path = download_pdf(report.pdf_url, dest_folder="pdfs")

# Convert to markdown with heading hierarchy
report.md = pdf_to_md(pdf_path)
```

### 3. Extract key sections

``` python
from iomeval.extract import extract_exec_summary, extract_findings, extract_conclusions, extract_recommendations

report.exec_summary = extract_exec_summary(report.md)
report.findings = extract_findings(report.md)
report.conclusions = extract_conclusions(report.md)
report.recommendations = extract_recommendations(report.md)
```

### 4. Map to strategic frameworks

``` python
from iomeval.mapper import map_srf_enablers, map_srf_crosscutting, map_gcm, map_srf_outputs

# Each mapper returns structured results with centrality scores
report.srf_enablers = map_srf_enablers(report.conclusions)
report.srf_crosscutting = map_srf_crosscutting(report.conclusions)
report.gcm = map_gcm(report.conclusions)
report.srf_outputs = map_srf_outputs(report.conclusions)
```

### 5. Save/restore checkpoints

``` python
# Save progress
report.save("checkpoint.json")

# Resume later
report = Report.load("checkpoint.json")
```

## Development

iomeval is built with [nbdev](https://nbdev.fast.ai/), which means the
entire library is developed in Jupyter notebooks. The notebooks serve as
both documentation and source code.

### Setup for development

``` sh
git clone https://github.com/franckalbinet/iomeval.git
cd iomeval
pip install -e '.[dev]'
```

### Key nbdev commands

``` sh
nbdev_test          # Run tests in notebooks
nbdev_export        # Export notebooks to Python modules
nbdev_preview       # Preview documentation site
nbdev_prepare       # Export, test, and clean notebooks (run before committing)
```

### Workflow

1.  Make changes in the `.ipynb` notebook files
2.  Run `nbdev_prepare` to export code and run tests
3.  Commit both notebooks and exported Python files
4.  Documentation is automatically generated from the notebooks

Learn more about nbdev’s literate programming approach in the [nbdev
documentation](https://nbdev.fast.ai/).

### Contributing

Contributions are welcome! Please: - Follow the existing notebook
structure - Add tests using nbdev’s `#| test` cells - Run
`nbdev_prepare` before submitting PRs
