Metadata-Version: 2.4
Name: super-document-processor
Version: 0.1.4
Summary: High-fidelity multi-format document parsing and chunking engine for RAG pipelines
Author: Cinderfox
License: MIT
Project-URL: Homepage, https://github.com/Cinderfox/Super-Document-Processor
Project-URL: Repository, https://github.com/Cinderfox/Super-Document-Processor
Keywords: document-processing,rag,pdf,docx,pptx,xlsx,multimodal,vlm,chunking,parsing
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: General
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Operating System :: OS Independent
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: PyMuPDF==1.28.2
Requires-Dist: python-docx==1.2.0
Requires-Dist: python-pptx==1.0.2
Requires-Dist: extract-msg==0.56.1
Requires-Dist: lxml==6.1.3
Requires-Dist: aspose-slides==26.7.0
Requires-Dist: python-dotenv==1.2.3
Requires-Dist: fastapi==0.141.1
Requires-Dist: uvicorn==0.52.4
Requires-Dist: python-multipart==0.0.32
Provides-Extra: dev
Requires-Dist: build>=1.0.0; extra == "dev"
Requires-Dist: twine>=5.0.0; extra == "dev"
Dynamic: license-file

# Super Document Processor

A high-fidelity, multi-format document parsing and chunking engine tailored for Retrieval-Augmented Generation (RAG) pipelines. It extracts structured text, preserves tables, optionally transcribes embedded charts and images with Vision LLMs, and outputs boundary-aligned chunks.

Available as a **Python library**, **CLI tool**, and **FastAPI microservice** (with Docker support).

---

## Supported Formats

| Category | Formats | Features Extracted |
|---|---|---|
| **Documents** | `.pdf`, `.docx` | Headings, paragraphs, markdown/HTML tables, hyperlinks, embedded images |
| **Presentations** | `.pptx` | Slide text, shapes, notes, multi-table separation, smart-art, charts |
| **Emails** | `.msg` | Headers (From, To, Subject, Date) and formatted email body |
| **Images** | `.jpg`, `.jpeg`, `.png`, `.jfif` | Full VLM vision analysis and table transcription |
| **Code & Text** | `.txt`, `.md`, `.json`, `.csv`, `.py`, `.js`, `.ts`, `.html`, etc. | Natural paragraph and token-bounded chunking |

---

## Quickstart

### 1. Install via PyPI
```bash
pip install super-document-processor
```

### 2. Or Local Clone
```bash
git clone https://github.com/Cinderfox/Super-Document-Processor.git
cd Super-Document-Processor

python -m venv .venv
source .venv/bin/activate  # On Windows: .\.venv\Scripts\activate
pip install -r requirements.txt
cp .env.example .env
```

### 3. Docker
```bash
docker run -d --name doc-processor -p 8000:8000 -v ${PWD}/data:/data cinderfox/super-document-processor:latest
```

---

## 1. Python Library Usage

```python
from super_document_processor import process_file

# 1. Standard text & table extraction (zero LLM calls, ultra-fast)
chunks = process_file("annual_report.pdf")

# 2. Extract with Vision LLM (describes embedded charts, diagrams, and photos)
chunks_with_vision = process_file(
    file_path="pitch_deck.pptx",
    process_images=True,
    model="google-gemini-3-flash"  # e.g. "gpt-4o", "google-gemini-3-flash", etc.
)

for chunk in chunks:
    print(f"--- Chunk {chunk['chunkIndex']} ({chunk['filename']}) ---")
    print(chunk["content"])
```

### Chunk Data Structure
```json
[
  {
    "content": "==================== Slide 1 ====================\nSlide Text:\n- Performance Scorecard...",
    "chunkIndex": 1,
    "filename": "pitch_deck.pptx"
  }
]
```

### Direct Chunker Access & Default Limits
Text-based document processors (`.docx`, `.msg`, `.txt` etc) use `ChunkCollector` for token-safe output partitioning:
- **Default Max Size**: `6,000` characters per chunk
- **Default Overlap**: `0` characters (zero duplicated text across chunks)

```python
from super_document_processor.utils.chunker import collector, chunk_text

# Using standalone utility with custom values:
chunks = chunk_text(text="Your long text...", chunk_size=6000, overlap=200)

# Or collecting and retrieving chunked texts:
collector.add("First section text...")
chunks = collector.get_chunked_texts(max_size=6000, overlap=200)
collector.clear()
```

### PPTX Slide Rendering & Image Conversion
Export PowerPoint slides directly to SVGs or PNGs:

```python
from super_document_processor import convert_pptx, create_pptx_render_zip

# 1. Convert all slides to SVGs (returns list of clean SVG strings)
res = convert_pptx("presentation.pptx", format="svg")
for s in res["slides"]:
    print(f"Slide {s['slide']}: {s['svg'][:80]}...")

# 2. Convert specific slide to high-res PNG bytes
res_png = convert_pptx("presentation.pptx", format="png", slide=1, scale=2.0)
with open("slide_1.png", "wb") as f:
    f.write(res_png["raw_bytes"])

# 3. Create a downloadable ZIP containing all rendered slides
zip_bytes = create_pptx_render_zip("presentation.pptx", format="png")
with open("presentation_pngs.zip", "wb") as f:
    f.write(zip_bytes)
```

---

## 2. CLI Usage

```bash
# Basic extraction to terminal (stdout)
python main.py document.pdf

# Save raw JSON chunks to a file
python main.py report.docx --json -o output.json

# Process embedded images using a Vision Model
python main.py slides.pptx --process-images --model google-gemini-3-flash --json -o output.json

# Convert all PPTX slides to clean SVG files in a directory
python main.py slides.pptx --convert-pptx --format svg -o ./output_svgs

# Convert specific slide to PNG
python main.py slides.pptx --convert-pptx --format png --slide 1 -o slide_1.png

# Convert PPTX and package all slides into a ZIP archive
python main.py slides.pptx --convert-pptx --format png --as-zip -o slides.zip

# Start the REST API server
python main.py --serve --port 8000
```

| Flag | Short | Default | Description |
|---|---|---|---|
| `file` | | *None* | Path to document or image file |
| `--json` | | `False` | Output as raw JSON chunks or JSON slide array |
| `--output` | `-o` | `stdout` | Destination file path or directory for output |
| `--convert-pptx` | `-c` | `False` | Convert PPTX slides to SVG or PNG images |
| `--format` | `-f` | `svg` | Target conversion format (`svg` or `png`) |
| `--slide` | | *All* | Specific 1-based slide index to convert |
| `--scale` | | `2.0` | Resolution scale factor for PPTX rendering |
| `--as-zip` | | `False` | Bundle all converted slides into a single `.zip` file |
| `--process-images` | | `False` | Enable AI vision processing for images/tables |
| `--model` | `-m` | *None* | Vision model name (required if `--process-images` is set) |
| `--serve` | `-s` | `False` | Start the FastAPI HTTP microservice |
| `--port` | `-p` | `8000` | Server listening port |

---

## 3. REST API Microservice

Start the server with `python main.py --serve --port 8000` or via Docker.
- **Swagger Docs**: `http://localhost:8000/docs`
- **Health Check**: `http://localhost:8000/health`

### 1. Document Parsing & Chunking: `POST /process`

Accepts **either** a file upload (`multipart/form-data`) **or** a local/volume file path (`application/json`). Providing both or neither returns an HTTP 400 error.

#### Option A: File Upload (Multipart Form)
```bash
# Basic upload
curl -X POST "http://localhost:8000/process" \
     -F "file=@/path/to/contract.pdf"

# With Vision LLM processing
curl -X POST "http://localhost:8000/process?process_images=true&model=google-gemini-3-flash" \
     -F "file=@/path/to/presentation.pptx"
```

#### Option B: Local File Path or Mounted Volume Path (JSON)
```bash
# Running locally:
curl -X POST "http://localhost:8000/process" \
     -H "Content-Type: application/json" \
     -d '{"file_path": "./test_docs/report.pdf"}'

# Running in Docker:
curl -X POST "http://localhost:8000/process" \
     -H "Content-Type: application/json" \
     -d '{"file_path": "/data/report.pdf", "process_images": true, "model": "gpt-4o"}'
```

---

### 2. PPTX to SVG / PNG Conversion: `POST /convert/pptx`

Renders PowerPoint slides into SVG or PNG images.

#### Stream Single Slide Directly as Image (`image/svg+xml` or `image/png`)
Ideal for embedding directly in `<img>` tags or displaying in browsers:
```bash
# Get Slide 1 as direct SVG
curl -X POST "http://localhost:8000/convert/pptx?slide=1&format=svg" \
     -F "file=@presentation.pptx" > slide_1.svg

# Get Slide 2 as direct high-res PNG
curl -X POST "http://localhost:8000/convert/pptx?slide=2&format=png&scale=2.0" \
     -F "file=@presentation.pptx" > slide_2.png
```

#### Download All Slides as a ZIP Archive
```bash
curl -X POST "http://localhost:8000/convert/pptx?format=png&as_zip=true" \
     -F "file=@presentation.pptx" -o presentation_slides.zip
```

#### Convert Mounted Volume Path via JSON Body
```bash
curl -X POST "http://localhost:8000/convert/pptx" \
     -H "Content-Type: application/json" \
     -d '{"file_path": "/data/presentation.pptx", "format": "svg"}'
```

#### Python Client Example
```python
import requests

url = "http://localhost:8000/convert/pptx"

# 1. Download Slide 1 directly as image
with open("pitch_deck.pptx", "rb") as f:
    resp = requests.post(url, files={"file": f}, params={"slide": 1, "format": "png"})
    with open("slide_1.png", "wb") as out:
        out.write(resp.content)

# 2. Get all slides as JSON payload (contains SVGs or base64 PNGs)
with open("pitch_deck.pptx", "rb") as f:
    resp = requests.post(url, files={"file": f}, params={"format": "svg", "response_type": "json"})
    data = resp.json()
    print(f"Rendered {data['total_slides']} slides.")
```

---

### 3. Async Processing: `POST /process-async` & `GET /jobs/{job_id}`

For large files or heavy multimodal extraction, submit jobs asynchronously to prevent client gateway timeouts. No Redis required (backed by persistent `/data/jobs` storage).

#### 1. Submit Job (`POST /process-async`)
```bash
curl -X POST "http://localhost:8000/process-async" \
     -F "file=@large_document.pdf"
```
```json
{
  "job_id": "e8d7a12b-34c5-4f67-8901-abcdef123456",
  "filename": "large_document.pdf",
  "status": "processing",
  "poll_url": "/jobs/e8d7a12b-34c5-4f67-8901-abcdef123456"
}
```

#### 2. Poll Status & Fetch Result (`GET /jobs/{job_id}`)
```bash
curl "http://localhost:8000/jobs/e8d7a12b-34c5-4f67-8901-abcdef123456"
```
- **In Progress**: `{"status": "processing", "progress": 0}`
- **Completed**: Returns the full JSON payload (`"status": "completed"`, `total_chunks`, `chunks`). The job file is automatically deleted from disk upon successful retrieval.

---

## 4. Docker Deployment

The container image includes all rendering dependencies (`libicu`, `ca-certificates`, etc.).

### Run as an API Server
```bash
# Port 8000 exposed, local ./data folder mounted to /data in container
docker run -d \
  --name doc-processor \
  -p 8000:8000 \
  --env-file .env \
  -v ${PWD}/data:/data \
  super-document-processor:latest
```

### Run as One-Off CLI Tool & Save Output to Host
Files created inside Docker default to the container's isolated layer. To save the output JSON directly on your host machine, use either **Volume Mounting** or **Shell Redirection**:

```bash
# Method 1: Mount folder and write with -o (appears in your host folder)
docker run --rm -v ${PWD}:/data super-document-processor:latest \
  python main.py /data/report.pdf --json -o /data/output.json

# Method 2: Stream stdout directly to host file (works with docker run or docker exec)
docker run --rm -v ${PWD}:/data super-document-processor:latest \
  python main.py /data/report.pdf --json > output.json

# Method 3: Run against an already active container
docker exec doc-processor python main.py /data/report.pdf --json > output.json
```

### Docker Compose
```yaml
version: "3.8"

services:
  doc-processor:
    image: super-document-processor:latest
    container_name: doc-processor
    restart: unless-stopped
    ports:
      - "8000:8000"
    env_file:
      - .env
    volumes:
      - ./data:/data
```

```bash
docker compose up -d
```

---

## Environment Variables

| Variable | Default | Description |
|---|---|---|
| `MAX_WORKERS` | `3` | Max concurrent worker threads for slide rendering, image extraction, and VLM calls |
| `PORT` | `8000` | HTTP port when running in server mode |
| `JOBS_DIR` | `/data/jobs` (or `./data/jobs`) | Storage directory for asynchronous background job results |
| `OPENAI_API_KEY` | *None* | API key for Vision LLM extraction |
| `OPENAI_BASE_URL` | *None* | Gateway URL (e.g. `https://api.openai.com/v1`, LiteLLM, Ollama, etc.) |

---

## License

[MIT License](LICENSE) © [Cinderfox](https://github.com/Cinderfox)
