Metadata-Version: 2.4
Name: creel
Version: 0.1.1
Summary: Extract linked information from a mess or sources
Project-URL: Homepage, https://github.com/thorwhalen/creel
Project-URL: Repository, https://github.com/thorwhalen/creel
Project-URL: Documentation, https://thorwhalen.github.io/creel
Author: Thor Whalen
License: mit
License-File: LICENSE
Requires-Python: >=3.10
Requires-Dist: jsonschema>=4
Requires-Dist: networkx>=3
Requires-Dist: pydantic>=2
Provides-Extra: aix
Requires-Dist: aix>=0.1; extra == 'aix'
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.40; extra == 'anthropic'
Provides-Extra: dev
Requires-Dist: pytest-cov>=4.0; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: ruff>=0.1.0; extra == 'dev'
Provides-Extra: docs
Requires-Dist: sphinx-rtd-theme>=1.0; extra == 'docs'
Requires-Dist: sphinx>=6.0; extra == 'docs'
Provides-Extra: er
Requires-Dist: splink>=4; extra == 'er'
Provides-Extra: eval
Requires-Dist: deepeval>=0.21; extra == 'eval'
Provides-Extra: graphdb
Requires-Dist: neo4j>=5; extra == 'graphdb'
Requires-Dist: pyoxigraph>=0.3; extra == 'graphdb'
Requires-Dist: rustworkx>=0.14; extra == 'graphdb'
Provides-Extra: ingest
Requires-Dist: docling>=2; extra == 'ingest'
Requires-Dist: openpyxl>=3.1; extra == 'ingest'
Requires-Dist: python-docx>=1.1; extra == 'ingest'
Requires-Dist: trafilatura>=1.6; extra == 'ingest'
Provides-Extra: llm
Requires-Dist: instructor>=1; extra == 'llm'
Provides-Extra: ocr
Requires-Dist: pytesseract>=0.3; extra == 'ocr'
Provides-Extra: query
Requires-Dist: duckdb>=0.10; extra == 'query'
Requires-Dist: jmespath>=1; extra == 'query'
Provides-Extra: semantic
Requires-Dist: linkml>=1.7; extra == 'semantic'
Requires-Dist: pyshacl>=0.25; extra == 'semantic'
Requires-Dist: pyyaml>=6; extra == 'semantic'
Requires-Dist: rdflib>=7; extra == 'semantic'
Description-Content-Type: text/markdown

# creel

**Extract a typed graph from a mess of sources.**

creel is a general, AI-powered **source-to-graph extraction engine**. You give it
(a) **sources** — freeform prose, tables, JSON, PDFs; (b) a **grammar** of the graph
you want — its node-types and edge-types and the typed values they carry; and (c)
**extractors** — pluggable strategies that know how to find each element. creel returns
a clean, auditable, typed **property graph** as a single source of truth, canonically a
JSON graph specification:

```python
extract(sources, graph_spec, extractors) -> graph
```

Everything downstream — persistence, query, graph-RAG, annotation, rendering to
slides/reports — is a *projection* of that one graph.

## Install

```bash
pip install creel                 # core: pydantic, jsonschema, networkx
pip install "creel[query]"        # SQL/JSON query extractors (duckdb, jmespath)
pip install "creel[ingest]"       # document loaders (docling, trafilatura, openpyxl, python-docx)
pip install "creel[aix]"          # real LLM extraction/judging/embedding via aix
```

## A first taste

Declare a grammar, extract a graph from prose, validate it, emit canonical JSON:

```python
from creel import (
    GraphSpec, NodeType, EdgeType, AttrSchema, EnumDef,
    extract, validate_graph, to_canonical_json,
)

spec = GraphSpec(
    enums=(EnumDef("Currency", ("USD", "EUR")),),
    node_types=(
        NodeType("donor", attributes=(AttrSchema("name", required=True),)),
        NodeType("project", attributes=(AttrSchema("title", required=True),)),
    ),
    edge_types=(
        EdgeType("funds", subject_type="donor", object_type="project",
                 attributes=(AttrSchema("amount", range="integer", required=True, minimum=0),
                             AttrSchema("currency", range="Currency", required=True))),
    ),
)

# Deterministic pattern extractors (no LLM): regex over prose.
bindings = {
    "donor": ("regex_node", {"pattern": r"Donor:\s*(?P<name>.+)", "id_attribute": "name"}),
    "project": ("regex_node", {"pattern": r"Project:\s*(?P<title>.+)", "id_attribute": "title"}),
    "funds": ("regex_edge", {
        "pattern": r"(?P<donor>[\w ]+?) funds (?P<project>[\w ]+?) with (?P<currency>[A-Z]{3}) (?P<amount>\d+)",
        "source_id_template": "donor:{donor}", "target_id_template": "project:{project}",
        "casts": {"amount": "int"}, "exclude_groups": ("donor", "project")}),
}
src = "Donor: Gov X\nProject: WASH\nGov X funds WASH with USD 1000000"

g = extract(src, spec, bindings, on_missing_binding="skip")
assert validate_graph(g, spec) == []
print(to_canonical_json(g))                 # deterministic, git-diffable JSON
print(g.evidence)                            # every element traced back to its source span
```

### With a real LLM (schema-as-extractor)

The attribute `description`s become the extraction instruction; the LLM client is
injected (no provider SDK in the core):

```python
from creel.extract.llm import aix_client
g = extract(prose, spec, {"donor": ("llm", {})},
            services={"llm": aix_client()}, on_missing_binding="skip")
```

## What you get

- **Labeled Property Graph** — attributes (funding amounts, indicator values) live *on
  edges*, which have their own identity; deterministic, git-diffable canonical JSON.
- **Three extractor families** behind one `Extractor` protocol: deterministic
  **pattern/function**, **query** (DuckDB SQL / JMESPath over structured sources), and
  **LLM** (schema-as-extractor, validate-retry, faithfulness gate) — plus **cluster-pass**
  (extract several coupled types in one LLM call).
- **Ingestion** (`ingest()`): route-by-format file loaders (md/csv/json/txt built-in;
  PDF/DOCX/XLSX/HTML via extras).
- **Auditability**: every node/edge/value carries a separable **evidence** record
  (provenance + grounding selector back to the exact source span + confidence). A
  **reverse-trace index** answers "which elements did this passage produce?", and a
  re-anchoring resolver keeps highlights valid across re-ingestion.
- **Evaluation by pluggable verifiers, not `==`**: `numeric_tolerance`, `set_match`,
  `graph_match` (partial credit), `llm_rubric` (NL-defined, G-Eval), … with a corpus runner.
- **Entity resolution** cascade (normalize → registry → LLM), the **reify** edge↔node
  toggle, **views** (DOT/Mermaid/Cytoscape/tables), and **export** adapters (JGF, GraphML,
  parameterized Cypher, RDF-star).

## Design & docs

The full reasoning lives in the research + design docs: start with the
[synthesis](misc/docs/research/00-synthesis-and-design-implications.md) (decisions
D1–D15), then the [roadmap](misc/docs/design/ROADMAP.md), the
[decision log](misc/docs/design/DECISIONS.md), and the
[progress log](misc/docs/design/PROGRESS.md). A worked example is in
[`examples/`](examples/).

## License

MIT.
