Metadata-Version: 2.4
Name: creel
Version: 0.1.3
Summary: Extract linked information from a mess or sources
Project-URL: Homepage, https://github.com/thorwhalen/creel
Project-URL: Repository, https://github.com/thorwhalen/creel
Project-URL: Documentation, https://thorwhalen.github.io/creel
Author: Thor Whalen
License: mit
License-File: LICENSE
Requires-Python: >=3.10
Requires-Dist: jsonschema>=4
Requires-Dist: networkx>=3
Requires-Dist: pydantic>=2
Provides-Extra: aix
Requires-Dist: aix>=0.1; extra == 'aix'
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.40; extra == 'anthropic'
Provides-Extra: dev
Requires-Dist: pytest-cov>=4.0; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: ruff>=0.1.0; extra == 'dev'
Provides-Extra: docs
Requires-Dist: sphinx-rtd-theme>=1.0; extra == 'docs'
Requires-Dist: sphinx>=6.0; extra == 'docs'
Provides-Extra: er
Requires-Dist: splink>=4; extra == 'er'
Provides-Extra: eval
Requires-Dist: deepeval>=0.21; extra == 'eval'
Provides-Extra: graphdb
Requires-Dist: neo4j>=5; extra == 'graphdb'
Requires-Dist: pyoxigraph>=0.3; extra == 'graphdb'
Requires-Dist: rustworkx>=0.14; extra == 'graphdb'
Provides-Extra: ingest
Requires-Dist: docling>=2; extra == 'ingest'
Requires-Dist: openpyxl>=3.1; extra == 'ingest'
Requires-Dist: python-docx>=1.1; extra == 'ingest'
Requires-Dist: trafilatura>=1.6; extra == 'ingest'
Provides-Extra: llm
Requires-Dist: aix>=0.1; extra == 'llm'
Provides-Extra: ocr
Requires-Dist: pytesseract>=0.3; extra == 'ocr'
Provides-Extra: query
Requires-Dist: duckdb>=0.10; extra == 'query'
Requires-Dist: jmespath>=1; extra == 'query'
Provides-Extra: semantic
Requires-Dist: linkml>=1.7; extra == 'semantic'
Requires-Dist: pyshacl>=0.25; extra == 'semantic'
Requires-Dist: pyyaml>=6; extra == 'semantic'
Requires-Dist: rdflib>=7; extra == 'semantic'
Description-Content-Type: text/markdown

# creel

**Extract a typed graph from a mess of sources.**

creel is a general, AI-powered **source-to-graph extraction engine**. You give it
(a) **sources** — freeform prose, tables, JSON, PDFs; (b) a **grammar** of the graph
you want — its node-types and edge-types and the typed values they carry; and (c)
**extractors** — pluggable strategies that know how to find each element. creel returns
a clean, auditable, typed **property graph** as a single source of truth, canonically a
JSON graph specification:

```python
extract(sources, graph_spec, extractors) -> graph
```

Everything downstream — persistence, query, graph-RAG, annotation, rendering to
slides/reports — is a *projection* of that one graph.

## Install

```bash
pip install creel                 # core: pydantic, jsonschema, networkx
pip install "creel[query]"        # SQL/JSON query extractors (duckdb, jmespath)
pip install "creel[ingest]"       # document loaders (docling, trafilatura, openpyxl, python-docx)
pip install "creel[aix]"          # real LLM extraction/judging/embedding via aix (default)
pip install "creel[anthropic]"    # real LLM extraction via the Anthropic SDK directly
pip install "creel[semantic]"     # LinkML authoring + RDF-star/Turtle export (linkml, rdflib)
```

Other extras: `graphdb` (Neo4j/Oxigraph), `er` (Splink entity resolution), `eval`
(DeepEval), `ocr`. The headline ones above cover most use; `pip install "creel[llm]"`
is an alias for the default provider (`aix`).

## A first taste

Declare a grammar, extract a graph from prose, validate it, emit canonical JSON:

```python
from creel import (
    GraphSpec, NodeType, EdgeType, AttrSchema, EnumDef,
    extract, validate_graph, to_canonical_json,
)

spec = GraphSpec(
    enums=(EnumDef("Currency", ("USD", "EUR")),),
    node_types=(
        NodeType("donor", attributes=(AttrSchema("name", required=True),)),
        NodeType("project", attributes=(AttrSchema("title", required=True),)),
    ),
    edge_types=(
        EdgeType("funds", subject_type="donor", object_type="project",
                 attributes=(AttrSchema("amount", range="integer", required=True, minimum=0),
                             AttrSchema("currency", range="Currency", required=True))),
    ),
)

# Deterministic pattern extractors (no LLM): regex over prose.
bindings = {
    "donor": ("regex_node", {"pattern": r"Donor:\s*(?P<name>.+)", "id_attribute": "name"}),
    "project": ("regex_node", {"pattern": r"Project:\s*(?P<title>.+)", "id_attribute": "title"}),
    "funds": ("regex_edge", {
        "pattern": r"(?P<donor>[\w ]+?) funds (?P<project>[\w ]+?) with (?P<currency>[A-Z]{3}) (?P<amount>\d+)",
        "source_id_template": "donor:{donor}", "target_id_template": "project:{project}",
        "casts": {"amount": "int"}, "exclude_groups": ("donor", "project")}),
}
src = "Donor: Gov X\nProject: Water\nGov X funds Water with USD 1000000"

g = extract(src, spec, bindings, on_missing_binding="skip")
assert validate_graph(g, spec) == []
print(to_canonical_json(g))                 # deterministic, git-diffable JSON
print(g.evidence)                            # every element traced back to its source span
```

### With a real LLM (schema-as-extractor)

The attribute `description`s become the extraction instruction; the LLM client is
injected (no provider SDK in the core). This block is self-contained:

```python
from creel import GraphSpec, NodeType, AttrSchema, extract
from creel.extract.llm import aix_client

spec = GraphSpec(node_types=(
    NodeType("donor", description="An entity that provides funding.",
             attributes=(AttrSchema("name", required=True,
                                    description="The donor's official name."),)),
))
prose = "Donor: Foundation Alpha (ref 301). Donor: Agency Beta (ref 918)."
g = extract(prose, spec, {"donor": ("llm", {})},
            services={"llm": aix_client()}, on_missing_binding="skip")
```

## Skills (the AI-native interface)

creel is **AI-first**, and that includes how you *learn and drive it*. It ships
**agent skills** — `SKILL.md` cheat-sheets that make Claude (and any `gh skill`-aware
agent: Copilot, Cursor, Codex, Gemini) actually good at using creel, instead of
guessing from docstrings. They install with `pip` (bundled, offline) **or** `gh skill`:

```bash
# All consumer skills ride along with the package:
pip install creel            # creel/data/skills/* is bundled in the wheel

# …or install individually into your agent (cross-agent, pinnable):
gh skill install thorwhalen/creel creel-extract       # the end-to-end workflow
gh skill install thorwhalen/creel creel-grammar       # author a typed GraphSpec
gh skill install thorwhalen/creel creel-bindings      # choose extractor strategies
gh skill install thorwhalen/creel creel-evaluation    # pluggable verifiers (≠ ==)
gh skill install thorwhalen/creel creel-ai            # real LLM extraction
gh skill install thorwhalen/creel creel-projections   # resolve / view / export / trace
# add --agent <copilot|cursor|codex|gemini|claude> to target a host; @vX.Y.Z to pin
```

Once installed, just ask in plain language ("extract a graph from these documents with
creel", "write a verifier for the amounts") — the matching skill triggers automatically.
The skills cross-reference each other, so an agent that starts at `creel-extract` is
handed off to `creel-grammar`/`creel-bindings`/`creel-ai`/`creel-evaluation` as needed.
Prefer hand-written API calls? Scroll on — everything below still applies.

## What you get

- **Labeled Property Graph** — attributes (funding amounts, indicator values) live *on
  edges*, which have their own identity; deterministic, git-diffable canonical JSON.
- **Three extractor families** behind one `Extractor` protocol: deterministic
  **pattern/function**, **query** (DuckDB SQL / JMESPath over structured sources), and
  **LLM** (schema-as-extractor, validate-retry, faithfulness gate) — plus **cluster-pass**
  (extract several coupled types in one LLM call).
- **Ingestion** (`ingest()`): route-by-format file loaders (md/csv/json/txt built-in;
  PDF/DOCX/XLSX/HTML via extras).
- **Auditability**: every node/edge/value carries a separable **evidence** record
  (provenance + grounding selector back to the exact source span + confidence). A
  **reverse-trace index** answers "which elements did this passage produce?", and a
  re-anchoring resolver keeps highlights valid across re-ingestion.
- **Evaluation by pluggable verifiers, not `==`**: `numeric_tolerance`, `set_match`,
  `graph_match` (partial credit), `llm_rubric` (NL-defined, G-Eval), … with a corpus runner.
- **Entity resolution** cascade (normalize → registry → LLM), the **reify** edge↔node
  toggle, **views** (DOT/Mermaid/Cytoscape/tables), and **export** adapters (JGF, GraphML,
  parameterized Cypher, RDF-star).

## Design & docs

The full reasoning lives in the research + design docs: start with the
[synthesis](misc/docs/research/00-synthesis-and-design-implications.md) (decisions
D1–D15), then the [roadmap](misc/docs/design/ROADMAP.md), the
[decision log](misc/docs/design/DECISIONS.md), and the
[progress log](misc/docs/design/PROGRESS.md). A worked example is in
[`examples/`](examples/).

## License

MIT.
