Metadata-Version: 2.4
Name: weji-gannet
Version: 0.2.0
Summary: Extract, index, and search any collection of PDF documents
Project-URL: Homepage, https://github.com/wejinortherntechnologiesinc/gannet
Project-URL: Issues, https://github.com/wejinortherntechnologiesinc/gannet/issues
License: MIT
License-File: LICENSE
Keywords: document-intelligence,fts5,full-text-search,ocr,pdf,pipeline,sqlite,tesseract
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Text Processing :: Indexing
Requires-Python: >=3.11
Requires-Dist: pdf2image>=1.17
Requires-Dist: pdfminer-six>=20221105
Requires-Dist: pytesseract>=0.3.10
Provides-Extra: dev
Requires-Dist: mypy; extra == 'dev'
Requires-Dist: numpy>=1.24; extra == 'dev'
Requires-Dist: pytest-cov; extra == 'dev'
Requires-Dist: pytest>=7; extra == 'dev'
Requires-Dist: ruff; extra == 'dev'
Requires-Dist: weji-goose>=0.3; extra == 'dev'
Provides-Extra: goose
Requires-Dist: numpy>=1.24; extra == 'goose'
Requires-Dist: weji-goose>=0.3; extra == 'goose'
Description-Content-Type: text/markdown

# weji-gannet

> Extract, index, and search any collection of PDF documents.

[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
[![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/downloads/)

**weji-gannet** turns a collection of PDFs into a properly structured, queryable
database — the same way a gannet dives into the ocean and surfaces with exactly
what it went in for.

Point it at any PDF collection — scanned documents, government disclosures, research
papers, contracts, archives — and get clean text extraction with full-text search and
optional columnar analytics via [weji-goose](https://github.com/wejinortherntechnologiesinc/goose).

Built by [WEJI Northern Technologies Inc.](https://wejinorth.ca) —
100% Indigenous-owned, Qalipu Mi'kmaq First Nation, Bay St. George, NL.

---

## How it works

```
Your PDFs  →  per-page extraction cascade  →  SQLite FTS5   (full-text search)
                                           →  weji-goose     (analytics, optional)
```

Each page is processed independently through three passes in order:

1. **pdftotext** — fast, lossless for digitally-generated PDFs
2. **tesseract 250 DPI / PSM 6** — OCR for scanned pages
3. **tesseract 300 DPI / PSM 3** — higher-res OCR for complex layouts

A hybrid PDF (typed cover page + scanned attachments) gets complete, correctly
ordered text from both sections with no duplication.

---

## Install

```bash
pip install weji-gannet

# System dependencies:
sudo apt install poppler-utils tesseract-ocr tesseract-ocr-eng   # Ubuntu/Debian
brew install poppler tesseract                                     # macOS
```

With [weji-goose](https://github.com/wejinortherntechnologiesinc/goose) columnar analytics (optional):
```bash
pip install "weji-gannet[goose]"
```

---

## Quick start — Python

```python
from weji_gannet import Pipeline
from weji_gannet.sources import LocalDirectorySource

p = Pipeline("./output")
p.ingest(LocalDirectorySource("./my_pdfs"))

# Full-text search
results = p.query("search terms here")
for r in results:
    print(r["doc_id"], r["excerpt"])

# Keyword intelligence — define your own categories
report = p.intelligence(targets={
    "contracts":     ["procurement", "tender", "awarded", "vendor"],
    "environmental": ["contamination", "remediation", "PFAS"],
    "complaints":    ["violation", "enforcement", "cease and desist"],
})

print(p.stats())
```

## Quick start — CLI

```bash
# Index a directory of PDFs
gannet --data-dir ./output ingest ./my_pdfs

# Search
gannet --data-dir ./output query "search terms"

# Intelligence report (define your targets in a JSON file)
echo '{"contracts": ["procurement", "tender"]}' > targets.json
gannet --data-dir ./output intel --targets targets.json

# Stats
gannet --data-dir ./output stats
```

---

## Extending to any PDF source

Implement `DocumentSource` to pull PDFs from anywhere — a web portal, an S3
bucket, a database of URLs, an API:

```python
from weji_gannet.sources import Document, DocumentSource
from pathlib import Path

class MySource(DocumentSource):
    def documents(self):
        for row in my_db.query("SELECT id, title, dept FROM docs"):
            yield Document(
                id=str(row.id),
                title=row.title,
                source=row.dept,
            )

    def download(self, doc, dest_dir):
        pdf_bytes = my_api.fetch(doc.id)
        dest = dest_dir / f"{doc.id}.pdf"
        dest.write_bytes(pdf_bytes)
        return dest

p = Pipeline("./output")
p.ingest(MySource())
```

---

## Output structure

```
output/
├── documents.db          — SQLite FTS5 (full-text search index)
├── goose/
│   └── documents/        — weji-goose columnar store (analytics, if installed)
├── files/                — downloaded PDFs
├── report.txt            — intelligence query results (human-readable)
└── report.json           — intelligence query results (structured)
```

### `documents` table schema

| Column | Description |
|---|---|
| `doc_id` | unique document identifier |
| `source` | originating collection or organisation |
| `title` | short label / document summary |
| `body` | extracted full text (all pages, joined by `\f`) |
| `year` / `month` | document date |
| `file_path` | local PDF path |
| `source_url` | original URL |
| `page_count` | total pages |
| `ocr_method` | how text was extracted (`pdftotext`, `tesseract`, `hybrid:dN/oM/bK`, `exhausted`) |

---

## Part of the WEJI stack

| Package | Role |
|---|---|
| [weji-goose](https://github.com/wejinortherntechnologiesinc/goose) | Columnar storage engine — analytics layer |
| **weji-gannet** | PDF extraction and indexing pipeline |

---

## License

MIT
