Metadata-Version: 2.4
Name: graphrag-stage1
Version: 0.2.0
Summary: Ontology-driven knowledge-extraction pipeline: paragraphs to a typed, provenance-anchored Knowledge Artifact Graph.
Author: Gedi
License: MIT
Project-URL: Homepage, https://github.com/gedi-onto/graphrag-stage1
Project-URL: Repository, https://github.com/gedi-onto/graphrag-stage1
Project-URL: Documentation, https://github.com/gedi-onto/graphrag-stage1/blob/main/DOCUMENTATION.md
Keywords: graphrag,knowledge-graph,ontology,nlp,rag,provenance
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests<3,>=2.31
Provides-Extra: ontology
Requires-Dist: rdflib<8,>=7.0; extra == "ontology"
Requires-Dist: pyshacl<1,>=0.28; extra == "ontology"
Provides-Extra: validation
Requires-Dist: jsonschema>=4.18; extra == "validation"
Provides-Extra: grounding
Requires-Dist: oaklib>=0.5; extra == "grounding"
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.40; extra == "anthropic"
Provides-Extra: openai
Requires-Dist: openai>=1.40; extra == "openai"
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: jsonschema>=4.18; extra == "dev"
Dynamic: license-file

# graphrag-stage1

**Turn unstructured scientific text into a typed, provenance-anchored knowledge graph.**

`graphrag-stage1` is the ingestion/enrichment engine for an ontology-driven GraphRAG
system. It is a **library**, not an app or a service: you import it, hand it your own
LLM client, and call functions. It ships no model of its own.

Standard RAG chops documents into chunks and stores a vector per chunk — throwing away
the reasoning (cause/effect, negation, who-claimed-it, exact numbers). This keeps all of
it, running a multi-pass pipeline modeled on a compiler.

```
text  ──▶  Stage 1  ──▶  Stage 2  ──▶  Stage 3  ──▶  RDF / JSON-LD / SHACL-validated N-Quads
          atomic facts   verified       ontology
          + facets       frames         mapping
```

- **Stage 1** breaks a paragraph into atomic, self-contained facts, each typed along
  seven orthogonal facets (measurement? cause? prediction? negated? who claimed it?) and
  anchored to its exact character span in the source.
- **Stage 2** turns each fact into a structured semantic frame — grounding-checked
  against the source text, with a computed confidence and a routing decision.
- **Stage 3** maps verified frames onto a formal OWL ontology (BFO/CCO/IAO + your domain)
  and emits RDF ready for a graph store such as Amazon Neptune.

The design is **neuro-symbolic** and that is the point: deterministic code (regex cue
extraction, a recall floor, schema and logic validators) provides guarantees; the LLM
provides semantic judgement; symbolic validators act as gates. The model is never the
only thing standing between the document and the graph. For example, if no number appears
in the text, `has_measurement` is forced false regardless of what the model claimed.

## Install

```bash
pip install "graphrag-stage1[ontology,grounding] @ git+https://github.com/gedi-onto/graphrag-stage1"
```

Extras are additive:

| Extra          | Adds                                                          |
|----------------|--------------------------------------------------------------|
| *(none)*       | Stage 1 + Stage 2. Only dependency is `requests`.            |
| `[ontology]`   | Stage 3 mapping + the Neptune/SHACL export (rdflib, pyshacl) |
| `[grounding]`  | entity grounding to real OBO term IDs (oaklib)              |
| `[validation]` | `validate()` against the shipped JSON Schemas               |
| `[anthropic]`  | built-in Claude client                                       |
| `[openai]`     | built-in OpenAI client                                       |

The **foundational ontologies** (IAO, CCO, RO, alignment + SHACL shapes, ~3.7 MB) ship
inside the package — nothing to download. You supply only your **domain** ontology.

Requires Python 3.10+.

## Quick start

```python
from graphrag_stage1 import analyze_paper, OllamaClient
from graphrag_stage1.grounding import OakGrounder

results = analyze_paper(
    open("paper.txt").read(),
    client=OllamaClient(model="qwen3:8b"),        # or AnthropicClient() / OpenAIClient()
    domain_ontology="ontologies/Domain/ino_merged.owl",
    grounder=OakGrounder(cache_path=".grounding_cache.json"),
)

for r in results:                                  # one entry per paragraph, in order
    if "error" in r:
        continue
    for fact in r["stage1"]["statements"]:
        f = fact["facets"]
        print(f["proposition_type"], "/", f["relation"], "|", fact["text"])
```

It works on **text**, not PDFs — extract text upstream (Textract, PyMuPDF, GROBID).

## Three things that will bite you

1. **Pass a `grounder`, or Stage 3 types almost nothing.** Without one, an entity that
   isn't already a class in your ontology can only generalise to a *true* ancestor —
   which in practice means `BFO:entity`, the ontology root. Measured on a real paper:
   0/10 entities domain-typed without a grounder, 8/22 with one.

2. **`num_ctx` × `OLLAMA_NUM_PARALLEL` (local Ollama).** Ollama allocates the context
   window *per parallel slot*. The defaults are tuned (`num_ctx=6144`,
   `STAGE2_BATCH_SIZE=5`); raising `num_ctx` to 8192 with `OLLAMA_NUM_PARALLEL=4` reserves
   32K of KV cache and thrashes VRAM on a 16 GB card. Measured 1026s vs 101s on identical
   work. If you raise it, lower `OLLAMA_NUM_PARALLEL`.

3. **`max_concurrency` is a budget, not a throttle.** Set it *to* your provider's
   allowance. Past it you get HTTP 429s, and each burns the full backoff before failing —
   slower *and* you lose paragraphs.

## Performance

Measured on `examples/pmc_dendrobine.txt` (qwen3:8b local, RTX 5080 laptop, defaults):
~25 s per real 70-word paragraph — an 80-paragraph paper in ~33 minutes. A bigger model
is **not** faster; it produces a better graph but costs more decode time per call. Speed
comes from parallel capacity (your quota or hardware), not model size.

## Status & limitations

Being straight about where this is:

- **122 tests pass.** Stages 1 and 2 are solid and well covered; Stage 3 is verified
  end-to-end from an installed wheel.
- **Evaluation is the weak point.** The reported metrics are 1.0, but against 8
  hand-written gold cases. The release gate (`release_gates.json`) requires 500
  adjudicated paragraphs across 50 documents and 8 domains; the corpus is downloaded
  (`osti_corpus.json`) but not yet adjudicated, so `readiness_check.py` correctly reports
  `BLOCKED`. Treat the numbers as directional, not proven.
- **Figure / table / equation extractors are routed but not implemented.** Stage 1 types
  such content by role and Stage 2 routes it to a specialist queue, but no handler
  consumes that queue yet. On a figure-heavy paper this is a meaningful share of content.
- **No layout parsing (L0).** The pipeline starts from pre-extracted text; provenance is
  character-offset based, not page/bbox.

## Documentation

- [`docs/graphrag_stage1_user_guide.pdf`](docs/graphrag_stage1_user_guide.pdf) — the
  field manual; start here.
- [`DOCUMENTATION.md`](DOCUMENTATION.md) — complete reference (API, config, performance).
- [`DESIGN.md`](DESIGN.md) — the architecture rationale.
- [`docs/CHOOSING_A_MODEL.md`](docs/CHOOSING_A_MODEL.md) — model/provider trade-offs, measured.

## License

MIT — see [LICENSE](LICENSE).
