Metadata-Version: 2.5
Name: doccheck
Version: 0.1.0
Summary: Checks that source documents (PDF, Word) are safe, readable, complete and relevant before they are accepted
Author-email: Pranay Kumar <pranaykumar.sriramula@automatonsx.com>
License-Expression: MIT
License-File: LICENSE
Keywords: document validation,docx,ectd,life sciences,ocr,paddleocr,pdf,pharmaceutical,tesseract
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Healthcare Industry
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Office/Business
Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: lxml>=4.9
Requires-Dist: pillow>=10.0
Requires-Dist: pymupdf>=1.23.5
Requires-Dist: python-docx>=0.8.11
Provides-Extra: aws
Requires-Dist: boto3>=1.28; extra == 'aws'
Provides-Extra: azure
Requires-Dist: azure-ai-formrecognizer>=3.3; extra == 'azure'
Provides-Extra: cloud
Requires-Dist: azure-ai-formrecognizer>=3.3; extra == 'cloud'
Requires-Dist: boto3>=1.28; extra == 'cloud'
Requires-Dist: google-cloud-vision>=3.4; extra == 'cloud'
Provides-Extra: dev
Requires-Dist: azure-ai-formrecognizer>=3.3; extra == 'dev'
Requires-Dist: build>=1.2; extra == 'dev'
Requires-Dist: openpyxl>=3.1; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Requires-Dist: twine>=5; extra == 'dev'
Provides-Extra: doctr
Requires-Dist: python-doctr[torch]>=0.8; extra == 'doctr'
Provides-Extra: easyocr
Requires-Dist: easyocr>=1.7; extra == 'easyocr'
Provides-Extra: google
Requires-Dist: google-cloud-vision>=3.4; extra == 'google'
Provides-Extra: paddleocr
Requires-Dist: paddleocr>=2.7; extra == 'paddleocr'
Requires-Dist: paddlepaddle>=2.5; extra == 'paddleocr'
Provides-Extra: rapidocr
Requires-Dist: rapidocr-onnxruntime>=1.3; extra == 'rapidocr'
Provides-Extra: tesseract
Requires-Dist: pytesseract>=0.3.10; extra == 'tesseract'
Provides-Extra: xlsx
Requires-Dist: openpyxl>=3.1; extra == 'xlsx'
Description-Content-Type: text/markdown

# doccheck

**Check a document before you accept it.** doccheck takes a PDF or Word file and checks whether it is safe,
readable, complete and on-topic (life sciences by default). It answers with **accepted**, **review** or
**rejected**, plus the reasons.

```python
from doccheck import ValidationEngine

result = ValidationEngine().validate("specification.pdf")
print(result.decision, result.relevance_score)     # accepted 94
```

## Features

- **File checks:** type, renamed files, size, password protection, damage.
- **Safety:** PDF JavaScript and launch actions; Word macros, ActiveX, embedded programs, remote templates.
- **Page checks:** empty, low-content and scanned pages; too many empty pages means rejected.
- **Relevance:** BM25 keyword scoring, with a bundled list of 2,994 life-sciences keywords, or your own.
- **Any OCR for scanned pages:**
  - cloud: Azure, AWS, Google;
  - local: Tesseract, PaddleOCR, EasyOCR, docTR, RapidOCR;
  - any web service or program;
  - your own plugin.
- **Explainable:** every score traces back to its keywords and pages. Nothing is stored unless you ask.

## Installation

```bash
pip install doccheck                   # Python 3.10+; checks PDF and Word files
pip install "doccheck[tesseract]"      # + an OCR engine, only if you need to read scanned pages
```

Available extras:
- **OCR:** `azure`, `aws`, `google`, `cloud` (all three), `tesseract`, `paddleocr`, `easyocr`, `doctr`, `rapidocr`.
- **Keywords:** `xlsx` (Excel keyword files).
- **Notes:** Tesseract also needs the Tesseract program. The `http` and `command` engines need no extra.

## Usage

```python
from doccheck import ValidationConfig, ValidationEngine

engine = ValidationEngine(ValidationConfig(           # create once, reuse for every file
    ocr_provider="tesseract",
    max_file_size_mb=50,
    extra_keywords={"nitrosamine risk assessment": 0.9},
))

result = engine.validate("dossier.pdf")               # or validate_bytes(name, data); async versions too
print(result)
```
```
dossier.pdf: REVIEW (relevance 97%, 7 pages)
  To review (accept only after a person confirms):
    - EMPTY_PAGES: 1 of 7 pages (14%) is empty, with no text or images (page 3).
    - LOW_TEXT: 1 of 7 pages (14%) has less content (page 6).
```
The other cases print like this:
```
report.pdf: REJECTED (relevance 0%, 10 pages)
  Rejected because:
    - DOCUMENT_NOT_RELEVANT: The document does not match the expected life sciences content (match 0%). ...
qos.pdf: ACCEPTED (relevance 99%, 58 pages)
  No issues found.
```
**Using the result in code:**
- `result.decision` and `result.is_valid` give the outcome.
- `result.errors` and `result.warnings` give the reasons.
- `result.relevance_score` gives the score.

**Input:** a PDF or `.docx` file, as a path or as bytes. Other types are rejected with a reason.

**Output:** a `ValidationResult`. `result.to_dict()` gives it as JSON, and never includes the document text.
```json
{
  "filename": "batch-specification.pdf",
  "decision": "review",
  "relevance_score": 97,
  "errors": [],
  "warnings": [{"code": "EMPTY_PAGES", "message": "1 of 7 pages (14%) is empty, with no text or images (page 3).", "pages": [3]}],
  "matched_keywords": [{"keyword": "3 2 p 3", "weight": 1.0, "occurrences": 6, "score": 6.5153}],
  "page_results": [{"page": 1, "content": "text", "read_by": "text", "evidence": 0.8745, "relevant": true}],
  "keyword_fingerprint": "bdc18b90b40baee3",
  "extractor_version": "1"
}
```
**What the decision means:**
- `rejected`: at least one error, e.g. `UNSAFE_CONTENT`, `DOCUMENT_NOT_RELEVANT`, `TOO_MANY_EMPTY_PAGES`.
- `review`: warnings only, e.g. `EMPTY_PAGES`, `CONTAINS_IMAGES`, `LOW_RELEVANCE`.
- `accepted`: nothing found.

**Traceability:**
- `keyword_fingerprint` identifies the exact keyword list used.
- `extractor_version` identifies the version of the page-reading rules.

## Configuration

Every setting has a default, so you only set what you change. Settings can be given in code, as above, or in a
`.env` file, which keeps keys out of the code:

```env
DOCCHECK_OCR_PROVIDER=azure
DOCCHECK_OCR_ENDPOINT=https://<name>.cognitiveservices.azure.com/
DOCCHECK_OCR_KEY=<key>
DOCCHECK_OCR_MAX_PAGES=5
```
```python
engine = ValidationEngine.from_env()        # environment variables win over .env
```

| Main settings | Default | Meaning |
|---|---|---|
| `ocr_provider` | `none` | OCR engine for scanned pages |
| `ocr_endpoint`, `ocr_key`, `ocr_options` | – | Engine address, key and options, e.g. `{"lang": "eng"}` |
| `ocr_max_pages` | `10` | Most scanned pages sent to OCR per document (caps the cost) |
| `keyword_source` | `default` | `default`, `none`, a file (CSV/TSV/TXT/JSON/XLSX), or a dict |
| `extra_keywords`, `remove_keywords` | – | Add to or take from the keyword list |
| `max_file_size_mb` | `100` | Largest accepted file |
| `relevance_accept_threshold` / `relevance_review_threshold` | `70` / `40` | Score bands: relevant / review / rejected |
| `result_store` | `none` | `none`, `jsonl:<path>`, or your object with `save(result)` |

**See all 34 settings,** their defaults, your values and what each one does:
- `doccheck settings` in the terminal;
- `print(config.explain())` in code.

**Other commands:**
- `doccheck env-template > .env` writes a template with every setting.
- `doccheck check-ocr` tests the OCR connection with one page.

**Mistakes fail when the engine is created,** before any file is processed:
```
ConfigError: Unknown setting 'ocr_max_page' (did you mean 'ocr_max_pages'?).
ConfigError: ocr_provider='azure' needs ocr_endpoint and ocr_key ...
ConfigError: The OCR engine needs 'paddleocr', which is not installed: pip install doccheck[paddleocr]
```

**OCR engines:**
- **Cloud:** `azure`, `aws`, `google`.
- **Local:** `tesseract`, `paddleocr`, `easyocr`, `doctr`, `rapidocr`.
- **Universal:** `http` (any OCR web service), `command` (any OCR program, e.g. `{"command": "tesseract {image} - -l eng"}`).
- **Your own:** `register_ocr_provider("name", YourClass)`.

**Keywords:**
- **Format:** two columns, `keyword,weight`, with the weight between 0 and 1.
- **Weights:** 0.5 or more marks a strong, specialist term; lower weights help the score but can't make a page
  relevant alone.
- **Check a file first:** `check_keyword_file(path)` lists refused rows with reasons.

## How it works

```
 file ─► 1. File checks ─► 2. Safety ─► 3. Read pages ─► 4. Text enough? ─► 5. Score ─► decision
                                         (PyMuPDF /        no ─► OCR a few
                                          python-docx)          scanned pages
```

1. **File checks.** Type, real content (not just the extension), size, password and damage. A failure here rejects
   the file at once.
2. **Safety.** Scripts, macros, embedded programs and remote content are looked for **before any content is read**.
   Any found rejects the file.
3. **Read pages, locally and for free.**
   - **PDF:** PyMuPDF reads the text of every page, where the pictures are, and what sits in the header and footer bands.
   - **Word:** python-docx and lxml do the same.

   Each page is classed as text, low content, empty, header-only or scanned (pictures, no usable text). Too many pages
   without content rejects the file.
4. **OCR only when needed.**
   - **When it's skipped:** text pages are used as they are. If they are enough (at least 3, making up at least half
     the document), **no OCR is used at all**.
   - **When it's used:** a random sample of the scanned pages is sent to the OCR engine. That's 25% of them, at least
     3 and **at most 10**, as one batch.
   - **What counts:** a page counts only if OCR reads at least 150 characters from it.
   - **One retry:** if under half of those pages could be read, one more sample of other pages is tried, once.
   - **Word files:** their pictures are put into a small PDF for the same OCR.

   This keeps the OCR cost and time low, even for documents with hundreds of scanned pages.
5. **Score.** The text that was read is matched against the keyword list. Each page gets a BM25 score.
   - **Document score:** 50% from the typical page, 30% from the share of relevant pages, 20% from keyword variety.
     So one keyword-dense page can't carry a whole document.
   - **Decision:** ≥ 70 is relevant, 40–70 is review, < 40 is rejected.

**Design choices:**
- **Cheap work first.** File and safety checks and local text reading run before any OCR.
- **OCR is capped.** It only runs when needed, on a few pages, which bounds both cost and time.
- **Fails closed.** An unexpected error rejects the file instead of letting it through.
- **Nothing is written to disk,** and every step is recorded in the result (`page_results`, `extraction_log`), so
  each decision can be traced.

## Scope

**For:** document intake, where it stops wrong, unsafe or unusable files at upload:
- regulatory and eCTD systems;
- QMS;
- supplier portals;
- clinical and pharmacovigilance intake.

It also suits bulk screening of archives. It is life sciences by default, and works for any subject with your own
keyword list.

**Not for:**
- checking that content is *correct*;
- telling document types apart (a specification from a stability report);
- virus scanning;
- extracting text.

## Accuracy and limitations

**Repeatability:**
- **Text documents:** the same file and settings always give the same result. There's no AI model or randomness.
- **Scanned documents:** OCR reads a random sample of pages, so scores can vary by a few points between runs.

**Measured** with the bundled list on documents not used for tuning (text only, by relevance score):

| Documents | Relevant (≥70) | Review (40–70) | Not relevant (<40) |
|---|---|---|---|
| Regulatory dossier documents (128) | 81% | 13% | 5% |
| Other life-sciences documents (10) | 6 | 2 | 2 |
| Unrelated documents (14) | 0 | 0 | 100% |

This is a guide, not a guarantee: test on your own documents.

**Limitations:**
- Keywords match English (a–z) text only.
- Very short documents can score low.
- Logos and signatures trigger `CONTAINS_IMAGES`, which leads to `review`.
- Word page numbers are estimated when the file wasn't saved by Word.
- Without OCR, scanned pages aren't read.

## Data and privacy

Documents and page text stay in memory and are never saved. Results are saved only if you set `result_store`.

Only a **cloud or `http` OCR** sends anything off the machine: the sampled scanned pages, at most `ocr_max_pages`.

## Development

```bash
uv pip install -e ".[dev]"
python -m pytest && python -m ruff check src tests
```
Changes are listed in `CHANGELOG.md`, which ships with the source package.

## License

doccheck is released under the **MIT License** (see the `LICENSE` file).

**Note on PyMuPDF:** doccheck depends on PyMuPDF to read PDFs, and PyMuPDF has its own licence: **AGPL-3.0**, or a
commercial licence from Artifex. If you distribute a product built with doccheck, or offer it as a network service,
PyMuPDF's terms apply to that product.
