Metadata-Version: 2.4
Name: kapipe
Version: 0.1.4
Summary: A modular pipeline for knowledge acquisition
Author-email: Noriki Nishida <norikinishida@gmail.com>
License: LICENSE
Project-URL: Homepage, https://github.com/norikinishida/kapipe
Keywords: NLP,knowledge acquisition,information extraction,knowledge graph,retrieval,question answering
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.10
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.22.2
Requires-Dist: scipy>=1.10.1
Requires-Dist: pandas>=1.5.3
Requires-Dist: spacy>=3.7.1
Requires-Dist: spacy-alignments>=0.9.1
Requires-Dist: scispacy
Requires-Dist: torch>=2.6.0
Requires-Dist: torch-tensorrt
Requires-Dist: torchdata
Requires-Dist: torchtext
Requires-Dist: torchvision
Requires-Dist: opt-einsum>=3.3.0
Requires-Dist: transformers>=4.46.0
Requires-Dist: accelerate>=1.0.1
Requires-Dist: bitsandbytes>=0.44.1
Requires-Dist: openai>=1.53.0
Requires-Dist: tenacity>=9.1.2
Requires-Dist: faiss-gpu-cu12>=1.12.0
Requires-Dist: Levenshtein>=0.25.0
Requires-Dist: networkx>=3.4.2
Requires-Dist: neo4j>=5.25.0
Requires-Dist: graspologic>=3.4.1
Requires-Dist: future>=1.0.0
Requires-Dist: pyhocon>=0.3.60
Requires-Dist: tqdm>=4.66.1
Requires-Dist: jsonlines>=4.0.0
Dynamic: license-file

<!-- ![KAPipe logo](./images/kapipe_logo_v01.png) -->

[!NOTE]
This repository is currently under active development.
Major updates are planned soon, including changes to APIs, component interfaces, model loading, and datasets.
The current implementation should be considered unstable until the update is complete.

# KAPipe

**KAPipe** is a modular framework for building ***Knowledge Acquisition Systems*** from unstructured data.

KAPipe decomposes knowledge acquisition into four main stages:

1. **Extraction**: extracting knowledge units from unstructured data.
2. **Organization**: organizing extracted knowledge units into structured representations such as knowledge graph.
3. **Retrieval**: retrieving relevant knowledge for a given query or task.
4. **Utilization**: using retrieved structured knowledge for downstream tasks such as question answering.

![An overview of knowledge acquisition system](./images/interviews_figure008.png)

KAPipe is used in the following papers:

- [Nishida et al., TACL 2026, **Dissecting GraphRAG: A Modular Analysis of Knowledge Structuring for Factoid Question Answering**.](https://aclanthology.org/2026.tacl-1.29/)
- [Oumaima and Nishida et al., BioNLP 2024, **Mention-Agnostic Information Extraction for Ontological Annotation of Biomedical Articles**.](https://aclanthology.org/2024.bionlp-1.37/)

![An example of graph-based RAG architecture](./images/nishida_et_al_tacl_2026.png)

## Installation

```bash
python -m pip install -U kapipe
```

For local development:

```bash
git clone https://github.com/norikinishida/kapipe.git
cd kapipe
python -m pip install -e .
```

Some pretrained models and configuration files are distributed separately.

```bash
mkdir -p ~/.kapipe
mv release.YYYYMMDD.tar.gz ~/.kapipe
cd ~/.kapipe
tar -zxvf release.YYYYMMDD.tar.gz
```

Release files are available here:
Please use the latest release file!

[KAPipe Release Files](https://drive.google.com/drive/folders/16ypMCoLYf5kDxglDD_NYoCNAfhTy4Qwp)

## Components

In KAPipe, a ***component*** is a modular processing unit that implements a specific approach within one of the four stages: extraction, organization, retrieval, or utilization.

The following table summarizes the components currently supported by KAPipe.

| Stage | Component | Module | Docs | Example |
|---|---|---|---|---|
| Extraction | Named Entity Recognition | `kapipe.ner` | [Docs](docs/components/ner.md) | [Example](experiments/ner) |
| Extraction | Entity Disambiguation (Retrieval) | `kapipe.ed_retrieval` | [Docs](docs/components/ed_retrieval.md) | [Example](experiments/ed_retrieval) |
| Extraction | Entity Disambiguation (Reranking) | `kapipe.ed_reranking` | [Docs](docs/components/ed_reranking.md) | [Example](experiments/ed_reranking) |
| Extraction | Document-level Relation Extraction | `kapipe.docre` | [Docs](docs/components/docre.md) | [Example](experiments/docre) |
| Organization | Entity Graph Construction | `kapipe.entity_graph_construction` | [Docs](docs/components/entity_graph_construction.md) | [Example](experiments/entity_graph_construction) |
| Organization | Community Clustering | `kapipe.community_clustering` | [Docs](docs/components/community_clustering.md) | [Example](experiments/community_clustering) |
| Organization | Report Generation | `kapipe.report_generation` | [Docs](docs/components/report_generation.md) | [Example](experiments/report_generation) |
| Organization | Chunking | `kapipe.chunking` | [Docs](docs/components/chunking.md) | [Example](experiments/chunking) |
| Retrieval | Passage Retrieval | `kapipe.passage_retrieval` | [Docs](docs/components/passage_retrieval.md) | [Example](experiments/passage_retrieval) |
| Utilization | Question Answering | `kapipe.qa` | [Docs](docs/components/qa.md) | [Example](experiments/qa) |

## Pipelines

Pipelines (`kapipe.pipelines`) are convenience classes for chaining components that are commonly used together.
Internally, a pipeline connects the outputs of one component to the inputs of the next component.

| Pipeline | Description | Docs | Example |
|---|---|---|---|
| `TripleExtractionPipeline` | Chains NER, Entity Disambiguation (Retrieval), Entity Disambiguation (Reranking), and Document-level Relation Extraction components | - | [Example](experiments/triple_extraction_pipeline) |
| `RAGPipeline` | Chains Passage Retrieval and Question Answering components | - | [Example](experiments/rag_pipeline) |
| `GraphRAGPipeline` | Chains triple extraction, Entity Graph Construction, Community Clustering, Report Generation, Passage Retrieval, and Question Answering components | - | [Example](experiments/graphrag_pipeline_tacl2026) |

## Quickstart

### Example 1

This example shows how to instantiate Passage Retrieval and QA components and run Retrieval-Augmented Generation (RAG).

```python
import os

from kapipe import utils
from kapipe.llms import OpenAILLM
from kapipe.passage_retrieval import Qwen3Embedding
from kapipe.qa import LLMQA


# Set input and output paths
data_dir = "experiments/passage_retrieval/data/examples"
index_dir = "./indexes"

# Load passages and questions
passages = utils.read_jsonl(os.path.join(data_dir, "passages.jsonl"))
questions = utils.read_json(os.path.join(data_dir, "questions.json"))

# Instantiate the Passage Retrieval component
passage_retrieval = Qwen3Embedding(
    model_name="Qwen/Qwen3-Embedding-0.6B",
    max_passage_length=8192,
    normalize=True,
    metric="inner-product",
    query_instruction="Given a question, retrieve relevant passages that answer the question.",
)

# Instantiate the QA component
llm = OpenAILLM(model_name="gpt-5.4-nano", max_new_tokens=8192)
qa = LLMQA(
    model=llm,
    prompt_template_name_or_path="qa_03_with_context",
)

# Build a retrieval index over passages
passage_retrieval.make_index(
    passages=passages,
    index_dir=index_dir,
    batch_size=64,
)

# Answer questions by chaining retrieval and QA
answers = []
for question in questions:

    # Retrieve relevant passages
    retrieved_passages = passage_retrieval.search(
        queries=[question["question"]],
        top_k=5,
    )[0]

    # Wrap retrieved passages in the QA input format
    contexts_for_question = {
        "question_key": question["question_key"],
        "contexts": retrieved_passages,
    }

    # Generate an answer
    answer = qa.answer(
        question=question,
        contexts_for_question=contexts_for_question,
    )

    # Preserve retrieved contexts
    answer["contexts"] = retrieved_passages

    answers.append(answer)

# Save the results
utils.write_json("./predictions.json", answers)
```

### Example 2

This example shows how to instantiate `GraphRAGPipeline`, structure knowledge, and run inference with GraphRAG.

The full executable version is available in [`experiments/graphrag_pipeline_tacl2026`](experiments/graphrag_pipeline_tacl2026).

```python
import os

from kapipe import utils
from kapipe.pipelines import GraphRAGPipeline
from kapipe.llms import OpenAILLM
from kapipe.ner import LLMNER
from kapipe.ed_retrieval import BlinkBiEncoder
from kapipe.ed_reranking import LLMED
from kapipe.docre import LLMDocRE
from kapipe.entity_graph_construction import EntityGraphConstructor
from kapipe.community_clustering import NeighborhoodAggregation
from kapipe.report_generation import TemplateBasedReportGenerator
from kapipe.chunking import Chunker
from kapipe.passage_retrieval import Qwen3Embedding
from kapipe.qa import LLMQA


# Set input and output paths
data_dir = "experiments/graphrag_pipeline_tacl2026/data/examples"
index_dir = "./indexes"

# Instantiate the components
llm = OpenAILLM(model_name="gpt-5.4-nano", max_new_tokens=8192)
ner = LLMNER.from_identifier(llm, "llm_ner_cdr")
ed_retrieval = BlinkBiEncoder.from_identifier("blink_bi_encoder_cdr")
ed_retrieval.make_index(use_precomputed_entity_vectors=True)
ed_reranking = LLMED.from_identifier(llm, "llm_ed_cdr")
docre = LLMDocRE.from_identifier(llm, "llm_docre_cdr")
entity_graph_construction = EntityGraphConstructor()
community_clustering = NeighborhoodAggregation(hop_size=1)
report_generation = TemplateBasedReportGenerator()
chunker = Chunker(model_name="en_core_sci_md")
passage_retrieval = Qwen3Embedding(
    model_name="Qwen/Qwen3-Embedding-0.6B",
    max_passage_length=8192,
    normalize=True,
    metric="inner-product",
    query_instruction="Given a question, retrieve relevant passages that answer the question."
)
qa = LLMQA(
    model=llm,
    prompt_template_name_or_path="qa_03_with_context",
)

# Instantiate the GraphRAG pipeline
graphrag = GraphRAGPipeline(
    ner=ner,
    ed_retrieval=ed_retrieval,
    ed_reranking=ed_reranking,
    docre=docre,
    entity_graph_construction=entity_graph_construction,
    community_clustering=community_clustering,
    report_generation=report_generation,
    chunker=chunker,
    passage_retrieval=passage_retrieval,
    qa=qa,
)

# Step 1. Extract triples from documents
documents = utils.read_json(os.path.join(data_dir, "documents.json"))
graphrag.extract_triples(
    documents=documents,
    retrieval_size=10,
    index_dir=index_dir,
)

# Step 2. Construct an entity graph from triples
graph = graphrag.construct_entity_graph(
    documents_path_list=[os.path.join(index_dir, "documents_with_triples.json")],
    entity_dict_path=os.path.join(data_dir, "entity_dict.json"),
    additional_triples_path=None,
    index_dir=index_dir,
)

# Step 3. Cluster the graph into communities (subgraphs)
communities = graphrag.cluster_communities(
    graph=graph,
    index_dir=index_dir,
)

# Step 4. Generate reports for each community
reports = graphrag.generate_community_reports(
    graph=graph,
    communities=communities,
    index_dir=index_dir,
)

# Step 5. Chunk community reports into chunks
chunked_reports = graphrag.chunk_reports(
    reports=reports,
    window_size=100,
    index_dir=index_dir,
)

# Step 6. Build a retrieval index over the chunked reports
graphrag.make_passage_retrieval_index(
    chunked_reports=chunked_reports,
    batch_size=64,
    index_dir=index_dir,
)

# Step 7. Load the retrieval index and answer questions
questions = utils.read_json(os.path.join(data_dir, "questions.json"))
answers = [
    graphrag.infer(question=question, top_k=5)
    for question in questions
]

# Save the results
utils.write_json("./predictions.json", answers)
```

The components can also be used independently.
Please see the corresponding documentation for each component for more details.

## Citation / Publication

If **KAPipe** is helpful for your work, please consider citing the following paper:

**Dissecting GraphRAG: A Modular Analysis of Knowledge Structuring for Factoid Question Answering**.
Noriki Nishida, Rumana Ferdous Munne, Shanshan Liu, Narumi Tokunaga, Yuki Yamagata, Fei Cheng, Kouji Kozaki, and Yuji Matsumoto.
Transactions of the Association for Computational Linguistics (TACL), vol. 14, pp. 627-655. 2026.
(Presented at ACL 2026)

```bibtex
@article{nishida-etal-2026-dissecting,
    title = "Dissecting {G}raph{RAG}: A Modular Analysis of Knowledge Structuring for Factoid Question Answering",
    author = "Nishida, Noriki  and
      Munne, Rumana Ferdous  and
      Liu, Shanshan  and
      Tokunaga, Narumi  and
      Yamagata, Yuki  and
      Cheng, Fei  and
      Kozaki, Kouji  and
      Matsumoto, Yuji",
    journal = "Transactions of the Association for Computational Linguistics",
    volume = "14",
    year = "2026",
    address = "Cambridge, MA",
    publisher = "MIT Press",
    url = "https://aclanthology.org/2026.tacl-1.29/",
    doi = "10.1162/tacl.a.615",
    pages = "627--655"
}
```

