Metadata-Version: 2.4
Name: xgen-doc2chunk
Version: 0.3.2
Summary: Convert raw documents into AI-understandable context with intelligent text extraction, table detection, and semantic chunking
Project-URL: Homepage, https://github.com/master0419/doc2chunk
Project-URL: Documentation, https://github.com/master0419/doc2chunk#readme
Project-URL: Repository, https://github.com/master0419/doc2chunk.git
Project-URL: Issues, https://github.com/master0419/doc2chunk/issues
Project-URL: Changelog, https://github.com/master0419/doc2chunk/releases
Author-email: master0419 <7slwm7@khu.ac.kr>
Maintainer-email: master0419 <7slwm7@khu.ac.kr>
License: Apache-2.0
License-File: LICENSE
Keywords: ai,chunking,document-processing,docx,hwp,langchain,llm,ocr,pdf,text-extraction,xlsx
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing
Requires-Python: >=3.12
Requires-Dist: beautifulsoup4>=4.12
Requires-Dist: cachetools>=6.0
Requires-Dist: chardet>=5.2
Requires-Dist: docx2pdf>=0.1.8
Requires-Dist: langchain-anthropic>=1.3.1
Requires-Dist: langchain-aws>=1.2.0
Requires-Dist: langchain-community>=0.4.1
Requires-Dist: langchain-core<2,>=1.3.3
Requires-Dist: langchain-google-genai>=4.1.3
Requires-Dist: langchain-openai>=1.1.14
Requires-Dist: langchain-text-splitters>=1.1.2
Requires-Dist: langchain<2,>=1.2.18
Requires-Dist: langgraph<1.2,>=1.1.10
Requires-Dist: langsmith>=0.8.0
Requires-Dist: olefile>=0.47
Requires-Dist: openpyxl>=3.1
Requires-Dist: orjson>=3.11.6
Requires-Dist: pandas>=2.2
Requires-Dist: pdf2image>=1.17
Requires-Dist: pdfminer-six>=20251230
Requires-Dist: pdfplumber>=0.11.9
Requires-Dist: pi-heif>=1.0
Requires-Dist: psutil>=6.0
Requires-Dist: pydantic-core>=2.27
Requires-Dist: pydantic-settings>=2.6
Requires-Dist: pydantic<3,>=2.10
Requires-Dist: pyhwp>=0.1b15
Requires-Dist: pymupdf>=1.26.7
Requires-Dist: pytesseract>=0.3.13
Requires-Dist: python-docx>=1.1
Requires-Dist: python-dotenv>=1.2.2
Requires-Dist: python-multipart>=0.0.27
Requires-Dist: python-pptx>=1.0
Requires-Dist: striprtf>=0.0.29
Requires-Dist: xlrd>=2.0
Description-Content-Type: text/markdown

# xgen-doc2chunk

**xgen-doc2chunk** is a document processing library that converts raw documents into AI-understandable context. It analyzes, restructures, and normalizes content so that language models can reason over documents with higher accuracy and consistency.

> **Current Version: 0.2.26** — See [CHANGELOG.md](CHANGELOG.md) for release history.

## Features

- **Multi-format Support**: Process a wide variety of document formats including:
  - PDF (adaptive complexity-based processing, multi-column layout, table detection)
  - Microsoft Office: DOCX, DOC, PPTX, PPT, XLSX, XLS
  - Korean documents: HWP, HWPX (Hangul Word Processor — full support)
  - Text formats: TXT, MD, RTF, CSV, TSV, HTML
  - Image files: JPG, PNG, GIF, BMP, WebP (via OCR)
  - Code files: Python, JavaScript, TypeScript, and 20+ languages
  - Config files: JSON, YAML, TOML, INI, ENV, and more

- **Intelligent Text Extraction**: 
  - Preserves document structure (headings, paragraphs, lists)
  - Extracts tables as HTML with proper `rowspan`/`colspan` handling
  - Handles merged cells and complex table layouts
  - Extracts and processes inline images
  - Header/footer extraction for DOC, DOCX, HWPX
  - Chart and diagram extraction from Office documents

- **OCR Integration**:
  - Pluggable OCR engine architecture
  - Supports **OpenAI**, **Anthropic**, **Google Gemini**, **AWS Bedrock**, and **vLLM** backends
  - Automatic OCR fallback for scanned documents or image-based PDFs
  - Standalone image file processing (JPG, PNG, etc.)
  - Custom image tag pattern support for OCR detection

- **Smart Chunking**:
  - Semantic text chunking with configurable size and overlap
  - Table-aware chunking that preserves table integrity (HTML & Markdown)
  - Page-based chunking with page number metadata
  - Protected regions for code blocks, tables, images, charts, and metadata
  - Small chunk merging to prevent table-title isolation
  - Nested table support in protected region detection
  - Position metadata (page number, line numbers, character offsets)

- **Metadata Extraction**:
  - Extracts document metadata (title, author, creation date, etc.)
  - Formats metadata in a structured, parseable format
  - Customizable metadata tag prefixes/suffixes

- **Storage Backends**:
  - Local file storage (default)
  - MinIO / S3 compatible cloud storage
  - Pluggable storage backend architecture

## Installation

```bash
pip install xgen-doc2chunk
```

Or using uv:

```bash
uv add xgen-doc2chunk
```

## Quick Start

### Basic Usage

```python
from xgen_doc2chunk import DocumentProcessor

# Create processor instance
processor = DocumentProcessor()

# Extract text from a document
text = processor.extract_text("document.pdf")
print(text)

# Extract text and chunk in one step
result = processor.extract_chunks(
    "document.pdf",
    chunk_size=1000,
    chunk_overlap=200
)

# Access chunks
for i, chunk in enumerate(result.chunks):
    print(f"Chunk {i + 1}: {chunk[:100]}...")

# Save chunks to markdown file
result.save_to_md("output/chunks.md")
```

### With OCR Processing

```python
from xgen_doc2chunk import DocumentProcessor
from xgen_doc2chunk.ocr.ocr_engine.openai_ocr import OpenAIOCR

# Initialize OCR engine
ocr_engine = OpenAIOCR(api_key="sk-...", model="gpt-4o")

# Create processor with OCR
processor = DocumentProcessor(ocr_engine=ocr_engine)

# Extract text with OCR processing enabled
text = processor.extract_text(
    "scanned_document.pdf",
    ocr_processing=True
)
```

### With Position Metadata

```python
from xgen_doc2chunk import DocumentProcessor

processor = DocumentProcessor()

result = processor.extract_chunks(
    "document.pdf",
    chunk_size=1000,
    include_position_metadata=True
)

# Access position metadata per chunk
if result.has_metadata:
    for chunk_data in result.chunks_with_metadata:
        print(f"Page {chunk_data['page_number']}, "
              f"Lines {chunk_data['line_start']}-{chunk_data['line_end']}: "
              f"{chunk_data['text'][:80]}...")
```

### Available OCR Engines

```python
from xgen_doc2chunk.ocr.ocr_engine.openai_ocr import OpenAIOCR
from xgen_doc2chunk.ocr.ocr_engine.anthropic_ocr import AnthropicOCR
from xgen_doc2chunk.ocr.ocr_engine.gemini_ocr import GeminiOCR
from xgen_doc2chunk.ocr.ocr_engine.bedrock_ocr import BedrockOCR
from xgen_doc2chunk.ocr.ocr_engine.vllm_ocr import VllmOCR

# OpenAI (recommended)
engine = OpenAIOCR(api_key="sk-...", model="gpt-4o")

# Anthropic Claude
engine = AnthropicOCR(api_key="sk-ant-...", model="claude-sonnet-4-20250514")

# Google Gemini
engine = GeminiOCR(api_key="...", model="gemini-2.0-flash")

# AWS Bedrock
engine = BedrockOCR(
    aws_access_key_id="AKIA...",
    aws_secret_access_key="...",
    aws_region="us-east-1",
    model="anthropic.claude-3-5-sonnet-20241022-v2:0"
)

# vLLM (self-hosted)
engine = VllmOCR(base_url="http://localhost:8000", model="Qwen/Qwen2-VL-7B-Instruct")
```

## Supported Formats

| Category | Extensions |
|----------|------------|
| Documents | `.pdf`, `.docx`, `.doc`, `.rtf`, `.pptx`, `.ppt`, `.hwp`, `.hwpx` |
| Spreadsheets | `.xlsx`, `.xls`, `.csv`, `.tsv` |
| Text | `.txt`, `.md`, `.markdown` |
| Web | `.html`, `.htm`, `.xhtml` |
| Images | `.jpg`, `.jpeg`, `.png`, `.gif`, `.bmp`, `.webp` |
| Code | `.py`, `.js`, `.ts`, `.jsx`, `.tsx`, `.java`, `.cpp`, `.c`, `.go`, `.rs`, `.cs`, `.swift`, `.kt`, `.rb`, `.php`, `.dart`, `.r`, `.scala`, `.sql`, `.vue`, `.svelte` |
| Config | `.json`, `.yaml`, `.yml`, `.xml`, `.toml`, `.ini`, `.cfg`, `.conf`, `.properties`, `.env` |
| Script | `.sh`, `.bat`, `.ps1`, `.zsh`, `.fish` |
| Log | `.log` |

## Architecture

```
xgen_doc2chunk/
├── core/
│   ├── document_processor.py       # Main entry point (DocumentProcessor, ChunkResult)
│   ├── processor/                  # Format-specific handlers
│   │   ├── base_handler.py         # Abstract base handler
│   │   ├── pdf_handler.py          # PDF processing (PyMuPDF + pdfplumber)
│   │   ├── docx_handler.py         # DOCX processing
│   │   ├── doc_handler.py          # DOC processing (auto-detects format)
│   │   ├── ppt_handler.py          # PowerPoint processing
│   │   ├── excel_handler.py        # Excel processing (XLSX/XLS)
│   │   ├── csv_handler.py          # CSV/TSV processing
│   │   ├── hwp_handler.py          # HWP (OLE) processing
│   │   ├── hwpx_handler.py         # HWPX (ZIP/XML) processing
│   │   ├── rtf_handler.py          # RTF processing
│   │   ├── text_handler.py         # Plain text / code processing
│   │   ├── html_reprocessor.py     # HTML document processing
│   │   ├── image_file_handler.py   # Standalone image processing (via OCR)
│   │   └── {format}_helper/        # Format-specific utilities
│   └── functions/
│       ├── img_processor.py        # Image handling & tag generation
│       ├── page_tag_processor.py   # Page/slide/sheet tag processing
│       ├── chart_extractor.py      # Chart data extraction
│       ├── chart_processor.py      # Chart formatting
│       ├── metadata_extractor.py   # Metadata extraction & formatting
│       ├── table_extractor.py      # Table data structures
│       ├── table_processor.py      # Table formatting (HTML/Markdown/Text)
│       ├── storage_backend.py      # Pluggable storage (Local, MinIO, S3)
│       ├── preprocessor.py         # File preprocessing
│       ├── file_converter.py       # File format conversion
│       └── utils.py                # General utilities
├── chunking/
│   ├── chunking.py                 # Main chunking API
│   ├── text_chunker.py             # Text-based chunking
│   ├── table_chunker.py            # Table-aware chunking (HTML & Markdown)
│   ├── page_chunker.py             # Page-based chunking
│   ├── sheet_processor.py          # Sheet/metadata processing
│   ├── protected_regions.py        # Protected region detection (nested tables)
│   └── constants.py                # Constants and patterns
└── ocr/
    ├── base.py                     # BaseOCR abstract class
    ├── ocr_processor.py            # OCR processing utilities
    └── ocr_engine/                 # OCR engine implementations
        ├── openai_ocr.py           # OpenAI GPT-4 Vision
        ├── anthropic_ocr.py        # Anthropic Claude Vision
        ├── gemini_ocr.py           # Google Gemini Vision
        ├── bedrock_ocr.py          # AWS Bedrock Vision
        └── vllm_ocr.py             # vLLM (self-hosted)
```

## Requirements

- Python 3.12+
- Required dependencies are automatically installed (see `pyproject.toml`)

### System Dependencies

For full functionality, you may need:

- **Tesseract OCR**: For local OCR fallback
- **LibreOffice**: For DOC/RTF conversion (optional)
- **Poppler**: For PDF image extraction

## Tag Customization

```python
processor = DocumentProcessor(
    # Image tag format (default: [Image:path])
    image_directory="output/images",
    image_tag_prefix="[Image:",
    image_tag_suffix="]",
    
    # Page tag format (default: [Page Number: N])
    page_tag_prefix="[Page Number: ",
    page_tag_suffix="]",
    
    # Slide tag format (default: [Slide Number: N])
    slide_tag_prefix="[Slide Number: ",
    slide_tag_suffix="]",
    
    # Chart tag format
    chart_tag_prefix="[chart]",
    chart_tag_suffix="[/chart]",
    
    # Metadata tag format
    metadata_tag_prefix="<Document-Metadata>",
    metadata_tag_suffix="</Document-Metadata>",
)
```

## Documentation

- [QUICKSTART.md](QUICKSTART.md) — Comprehensive guide with pipeline overview, OCR setup, and examples
- [CHANGELOG.md](CHANGELOG.md) — Release history
- [CONTRIBUTING.md](CONTRIBUTING.md) — Contribution guidelines

## License

Apache License 2.0 — see [LICENSE](LICENSE) for details.

## Contributing

Contributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
