Metadata-Version: 2.4
Name: coreason_refinery
Version: 0.2.2
Summary: industrial processing plant for data ingestion
License: # The Prosperity Public License 3.0.0
         
         Contributor: CoReason, Inc.
         
         Source Code: https://github.com/CoReason-AI/coreason_refinery
         
         ## Purpose
         
         This license allows you to use and share this software for noncommercial purposes for free and to try this software for commercial purposes for thirty days.
         
         ## Agreement
         
         In order to receive this license, you have to agree to its rules.  Those rules are both obligations under that agreement and conditions to your license.  Don't do anything with this software that triggers a rule you can't or won't follow.
         
         ## Notices
         
         Make sure everyone who gets a copy of any part of this software from you, with or without changes, also gets the text of this license and the contributor and source code lines above.
         
         ## Commercial Trial
         
         Limit your use of this software for commercial purposes to a thirty-day trial period.  If you use this software for work, your company gets one trial period for all personnel, not one trial per person.
         
         ## Contributions Back
         
         Developing feedback, changes, or additions that you contribute back to the contributor on the terms of a standardized public software license such as [the Blue Oak Model License 1.0.0](https://blueoakcouncil.org/license/1.0.0), [the Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0.html), [the MIT license](https://spdx.org/licenses/MIT.html), or [the two-clause BSD license](https://spdx.org/licenses/BSD-2-Clause.html) doesn't count as use for a commercial purpose.
         
         ## Personal Uses
         
         Personal use for research, experiment, and testing for the benefit of public knowledge, personal study, private entertainment, hobby projects, amateur pursuits, or religious observance, without any anticipated commercial application, doesn't count as use for a commercial purpose.
         
         ## Noncommercial Organizations
         
         Use by any charitable organization, educational institution, public research organization, public safety or health organization, environmental protection organization, or government institution doesn't count as use for a commercial purpose regardless of the source of funding or obligations resulting from the funding.
         
         ## Defense
         
         Don't make any legal claim against anyone accusing this software, with or without changes, alone or with other technology, of infringing any patent.
         
         ## Copyright
         
         The contributor licenses you to do everything with this software that would otherwise infringe their copyright in it.
         
         ## Patent
         
         The contributor licenses you to do everything with this software that would otherwise infringe any patents they can license or become able to license.
         
         ## Reliability
         
         The contributor can't revoke this license.
         
         ## Excuse
         
         You're excused for unknowingly breaking [Notices](#notices) if you take all practical steps to comply within thirty days of learning you broke the rule.
         
         ## No Liability
         
         ***As far as the law allows, this software comes as is, without any warranty or condition, and the contributor won't be liable to anyone for any damages related to this software or this license, under any kind of legal claim.***
License-File: LICENSE
License-File: NOTICE
Author: Gowtham A Rao
Author-email: gowtham.rao@coreason.ai
Requires-Python: >=3.11
Classifier: License :: Other/Proprietary License
Classifier: Programming Language :: Python :: 3.12
Classifier: Operating System :: OS Independent
Requires-Dist: anyio (>=4.12.1,<5.0.0)
Requires-Dist: fastapi (>=0.128.0,<0.129.0)
Requires-Dist: httpx (>=0.28.1,<0.29.0)
Requires-Dist: loguru (>=0.7.2,<0.8.0)
Requires-Dist: numba (>=0.61.0,<0.62.0)
Requires-Dist: openpyxl (>=3.1.5,<4.0.0)
Requires-Dist: pandas (>=3.0.0,<4.0.0)
Requires-Dist: pdf2image (>=1.17.0,<2.0.0)
Requires-Dist: pdfminer-six (>=20260107,<20260108)
Requires-Dist: pi-heif (>=1.2.0,<2.0.0)
Requires-Dist: pydantic (>=2.12.5,<3.0.0)
Requires-Dist: python-multipart (>=0.0.22,<0.0.23)
Requires-Dist: tabulate (>=0.9.0,<0.10.0)
Requires-Dist: unstructured (>=0.18.31,<0.19.0)
Requires-Dist: unstructured-inference (>=1.1.7,<2.0.0)
Requires-Dist: uvicorn (>=0.40.0,<0.41.0)
Project-URL: Documentation, https://github.com/CoReason-AI/coreason_refinery
Project-URL: Homepage, https://github.com/CoReason-AI/coreason_refinery
Project-URL: Repository, https://github.com/CoReason-AI/coreason_refinery
Description-Content-Type: text/markdown

# coreason-refinery

![License: Prosperity 3.0](https://img.shields.io/badge/license-Prosperity%203.0-blue)
![Build Status](https://github.com/CoReason-AI/coreason_refinery/actions/workflows/ci.yml/badge.svg)
![Code Style: Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)
![Documentation](https://img.shields.io/badge/docs-product_requirements.md-blue)

**coreason-refinery** is the industrial processing plant for data ingestion at CoReason-AI. It transforms raw documents (PDF, Excel, PPTX) into semantically structured, machine-readable Markdown, ready for RAG (Retrieval-Augmented Generation) pipelines.

Unlike traditional RAG tools that blindly chunk text, **coreason-refinery** is **Structure-Aware**. It preserves the logical hierarchy of documents (headers, sections) and ensures complex artifacts like tables and equations are kept intact.

## Features

*   **Multi-Modal Parsing**:
    *   **PDF**: Vision-first parsing to preserve table grids and layout.
    *   **Excel/CSV**: Treated as relational data, converted to Markdown tables with header preservation.
    *   **PowerPoint**: Flattens slides into linear narratives (planned support for speaker notes).
*   **Semantic Segmentation**:
    *   Splits documents by logical headers (#, ##) rather than character counts.
    *   **Table Rescue**: Never splits tables mid-row; merges tables spanning multiple pages.
*   **Context Injection ("Breadcrumbs")**:
    *   Enriches every chunk with its hierarchical path (e.g., `Context: Protocol > Section 4 > Toxicity`).
    *   Ensures LLMs understand the specific scope of any given text snippet.
*   **GxP Compliance Ready**:
    *   Designed for lineage tracking and metadata enrichment.

## Installation

```bash
pip install coreason-refinery
```

*Note: This package requires Python 3.12+.*

## Usage

Here is how to initialize and run a refinery job:

```python
import uuid
from coreason_refinery.models import IngestionJob, IngestionConfig
from coreason_refinery.pipeline import RefineryPipeline

# 1. Configure the Job
config = IngestionConfig(
    chunk_strategy="HEADER",
    segment_len=500
)

job = IngestionJob(
    id=uuid.uuid4(),
    source_file_path="path/to/document.pdf",
    file_type="auto",  # Infers PDF/Excel/CSV
    config=config,
    status="PROCESSING"
)

# 2. Run the Pipeline
pipeline = RefineryPipeline()
chunks = pipeline.process(job)

# 3. Inspect Results
for chunk in chunks:
    print(f"--- Chunk ID: {chunk.id} ---")
    print(chunk.text)
    print(f"Metadata: {chunk.metadata}")
```

## Architecture

The pipeline consists of three main stages:

1.  **The Cracker (Parsing)**: Routes files to specialized parsers (`UnstructuredPdfParser`, `ExcelParser`) to extract atomic elements.
2.  **The Cutter (Segmentation)**: Reassembles elements into `RefinedChunk`s based on document structure, applying "Rolling Context" to preserve hierarchy.
3.  **The Enricher (Metadata)**: (Planned) Adds lineage and semantic tags.

## License

This software is proprietary and dual-licensed under the **Prosperity Public License 3.0**.
Commercial use beyond a 30-day trial requires a separate license.
See `LICENSE` for details.

