Metadata-Version: 2.4
Name: seec
Version: 0.0.2
Summary: Schema-driven LLM extraction pipeline for metadata, classifications, entities, and relations
Project-URL: Repository, https://uva-hva.gitlab.host/bi-staff/gen_extraction
Author-email: Paulo Zirlis <paulozrdm@gmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: LLM,classifications,entities,extraction,metadata,relations,schema
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.12
Requires-Dist: instructor>=1.14
Requires-Dist: openai>=1.0
Requires-Dist: pandas>=2.0
Requires-Dist: pydantic>=2.0
Requires-Dist: requests>=2.28
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == 'dev'
Description-Content-Type: text/markdown

# SEEC

**S**tructured **E**ntity **E**xtraction and **C**lassification — a schema-driven LLM extraction pipeline. Define what to extract in a simple Markdown file, and the pipeline handles metadata, classifications, entities, and relations from any plain-text documents using a local Ollama model.

------------------------------------------------------------------------

## Quick Start

### 1. Install the package

``` bash
pip install seec
```

### 2. Install Ollama

Download from [ollama.ai](https://ollama.ai), then:

``` bash
ollama serve          # start the server (keep running)
ollama pull qwen3.5:9b   # pull the default model
```

### 3. Set up your project

``` python
from seec import init_project

init_project("./my_config")
# Creates: my_config/extraction_schema.md  (define what to extract)
#          my_config/prompts/              (LLM prompt templates)
```

Edit `my_config/extraction_schema.md` to define the fields, categories, entities, and relations relevant to your domain (see [Configuration](#configuration) below).

### 4. Run extraction

``` python
from seec import extract, load_extractions

results = extract(
    input_dir="data/documents",       # folder with .txt or .md files
    output_dir="outputs/extractions",
    model="qwen3.5:9b",
    config_dir="my_config",           # points to your schema + prompts
)

# Load results as DataFrames
docs_df, entities_df, relations_df = load_extractions("outputs/extractions")
```

Results are saved as one JSON file per document. Re-runs skip passes whose schema section hasn't changed.

------------------------------------------------------------------------

## Configuration {#configuration}

### Schema file

The extraction schema (`extraction_schema.md`) has 4 sections, each a Markdown table:

| Section | What it extracts | Example |
|-------------------|----------------------------------|-------------------|
| 1\. Metadata | Fields from the document header | title, author, year |
| 2\. Classifications | Document-level labels | study type, language |
| 3\. Entities | Named things in the text | tools, organizations, methods |
| 4\. Relations | Directed links between entities | tool *used_for* task |

Each section maps to one extraction pass. **Leave a section empty (keep the header, delete data rows) to skip that pass entirely.**

Copy-paste examples for each section:

``` markdown
# Metadata
| title | str | Full document title |

# Classifications — plain enum
| language | enum | English; Dutch; French | Language of the document |

# Classifications — enum with descriptions
| study_type | enum | qualitative: Based on interviews; quantitative: Based on numerical data | Study methodology |

# Entities
| organization | Named company or institution | "Acme Corp" |

# Relations
| employs | organization | person | Organization employs this person |
```

### Prompt customization

The `prompts/` folder contains system and user prompt files for each pass. Edit the **system prompts** to adjust the persona for your domain (e.g., change "document" to "legal contract"). Edit the **user prompts** to change framing, but keep the template variables (`{document_text}`, `{summary_text}`, etc.) as-is.

Note: prompt edits do **not** invalidate the cache. Delete a document's JSON file to force re-extraction.

### Model selection

| Hardware                | Recommended model      |
|-------------------------|------------------------|
| CPU only / 6-8 GB VRAM  | `qwen3.5:4b`           |
| 8-12 GB VRAM            | `qwen3.5:9b` (default) |
| 16+ GB VRAM             | `qwen3.5:14b`          |
| Apple Silicon, 8-16 GB  | `qwen3.5:9b`           |
| Apple Silicon, 24-36 GB | `qwen3.5:14b`          |
| Apple Silicon, 48+ GB   | `qwen3:30b`            |

> Apple Silicon uses unified memory — all system RAM is available for inference.

> Not sure? Run `ollama run qwen3.5:9b "Hello"`. If it loads, you're good. If it runs out of memory, drop to `qwen3.5:4b`.

------------------------------------------------------------------------

## API Reference

### `extract()`

Main entry point. Runs the 3-pass pipeline on all `.txt`/`.md` files in a directory.

``` python
extract(
    input_dir,                          # folder with input documents
    output_dir,                         # where JSON results are saved
    model,                              # Ollama model name (e.g. "qwen3.5:9b")
    config_dir=None,                    # folder from init_project() with schema + prompts
    url="http://localhost:11434/v1",    # Ollama endpoint
    schema_path=None,                   # override: direct path to extraction_schema.md
    prompts_dir=None,                   # override: direct path to prompts/ folder
    summarize=False,                    # summarize long docs before extraction
    summarize_model="qwen3.5:4b",      # model for summarization
    temperature=0.2,                    # LLM temperature (0.0-1.0)
    seed=42,                            # random seed for reproducibility
    timeout=300.0,                      # seconds before an LLM call times out
    num_ctx=32768,                      # context window size in tokens
    think=True,                         # enable model reasoning mode
    sample_size=None,                   # process only first N files (for testing)
    verbose=True,                       # print debug info
)
```

Returns a `dict` mapping `doc_id` to the extraction result.

### `load_extractions(output_dir)`

Loads all JSON results into three pandas DataFrames:

| DataFrame | Columns |
|---------------------------------------|---------------------------------|
| `docs_df` | `doc_id` + all metadata and classification fields |
| `entities_df` | `doc_id`, `entity_id`, `entity_type`, `value`, `evidence`, `section` |
| `relations_df` | `doc_id`, `relation_type`, `subject_entity_id`, `object_entity_id` |

### `init_project(target_dir)`

Copies the template schema and prompt files into the given directory. Only creates files that don't already exist (safe to re-run).

------------------------------------------------------------------------

## Technical Details

### Pipeline architecture

The pipeline runs three sequential passes per document:

1.  **Pass 1 — Metadata**: extracts structured fields from the first 1000 words of the document.
2.  **Pass 2 — Classifications**: assigns document-level labels using the full text (or summary).
3.  **Pass 3 — Entities & Relations**: extracts named entities with evidence quotes, plus directed relations between them.

Each pass uses [Instructor](https://github.com/instructor-ai/instructor) to enforce a dynamically generated Pydantic model as the LLM output schema, ensuring type-safe structured extraction.

### Caching

Each pass stores a SHA-256 fingerprint of its schema section in the output JSON. On re-run: - **Schema unchanged** — pass is skipped, cached result is reused. - **Schema changed** — pass re-runs and overwrites the cached value.

To clear the cache for a document, delete its JSON file. To re-run everything:

``` bash
rm outputs/extractions/*.json
```

### Summarization

For long documents (\>8000 words), enable `summarize=True` to pre-summarize before passes 2 and 3. This keeps the input within the model's context window and can improve extraction quality. Pass 1 always runs on the raw text (first 1000 words only).

### Reproducibility

Pass `seed=42` (default) for deterministic output. The same seed + model + schema + document produces the same extraction. Ollama must also be configured with the same `num_ctx` for full reproducibility.