Metadata-Version: 2.4
Name: pdf2qa
Version: 0.1.0
Summary: PDF → OCR → sliding-window chunking → multi-question generation via LLM → JSONL dataset for fine-tuning
Author-email: Muhammad Abul Bashar <abul.bashar@betopialimited.com>
License: MIT
Project-URL: Homepage, https://github.com/yourusername/pdf-qa-pipeline
Project-URL: Source, https://github.com/yourusername/pdf-qa-pipeline
Keywords: pdf,ocr,qa-generation,dataset,fine-tuning,bengali
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: PyMuPDF>=1.23
Requires-Dist: pytesseract>=0.3
Requires-Dist: Pillow>=10
Requires-Dist: requests>=2.31
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Requires-Dist: pytest-cov>=4; extra == "dev"
Dynamic: license-file

# pdf2qa

End-to-end pipeline: **PDF → OCR → sliding-window chunking → multi-question generation → JSONL dataset** for LLM fine-tuning.

```bash
pip install pdf2qa
pdf2qa book.pdf --api-key sk-...
```

## Features

- **OCR** — Scanned PDFs via PyMuPDF + Tesseract (optimised for Bengali/Arabic)
- **Sliding window** — Configurable chunk size/stride ensures no content is missed
- **Multi-question** — Generates 2–3 distinct questions per chunk, each saved as a separate JSONL object
- **Language-agnostic** — Default English prompt, override for any language with `--system-prompt`
- **Checkpoint/resume** — Survives interruptions and rate-limits
- **Retry logic** — Primary → fallback API with exponential backoff

## Quickstart

```bash
# With a PDF (auto-OCR)
pdf2qa book.pdf --api-key sk-... --output ./dataset

# With pre-extracted text
pdf2qa book.txt --skip-ocr --api-key sk-... --output ./dataset

# Custom chunking
pdf2qa book.pdf --window 4 --stride 2 --api-key sk-...
```

### Environment variable

Set `PDF_QA_API_KEY` instead of passing `--api-key` every time.

## Output format

JSONL (one object per line) in OpenAI fine-tuning format:

```jsonl
{"messages": [{"role": "user", "content": "প্রশ্ন?"}, {"role": "assistant", "content": "উত্তর..."}]}
{"messages": [{"role": "user", "content": "অন্য প্রশ্ন?"}, {"role": "assistant", "content": "উত্তর..."}]}
```

## Options

| Argument | Default | Description |
|----------|---------|-------------|
| `--lang` | `ben+ara` | Tesseract languages |
| `--dpi` | `400` | OCR render DPI |
| `--window` | `3` | Pages per chunk |
| `--stride` | `1` | Sliding stride |
| `--output` | `./output` | Output directory |
| `--skip-ocr` | – | Input is already .txt |
| `--model` | `qwen-code-35b` | LLM model ID |
| `--api-key` | – | API key (or `$PDF_QA_API_KEY`) |
| `--primary-url` | `http://...` | Primary API URL |
| `--fallback-url` | `https://...` | Fallback API URL |
| `--system-prompt` | *(built-in)* | Custom system prompt (`{num_questions}` placeholder) |
| `--user-prompt` | *(built-in)* | Custom user prompt (`{chunk}`, `{num_questions}`) |
| `--question-keywords` | `What Why How ...` | Keywords to identify a question line |

## Development

```bash
git clone https://github.com/yourusername/pdf2qa
cd pdf2qa
pip install -e ".[dev]"
pytest
```
