Metadata-Version: 2.4
Name: htpd-reader
Version: 0.1.2.post1
Summary: The HTPD reader: read HTPD files and find the exact text, with pages, that answers a question.
Author: Narendra Champaneri
License: Apache-2.0
Project-URL: Homepage, https://github.com/narendrachampaneri/htpd
Project-URL: Documentation, https://github.com/narendrachampaneri/htpd/blob/main/docs/READER.md
Project-URL: Source, https://github.com/narendrachampaneri/htpd
Project-URL: Issues, https://github.com/narendrachampaneri/htpd/issues
Keywords: pdf,htpd,documents,ai,llm,rag,mcp,datasheet
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Text Processing :: Indexing
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Provides-Extra: yaml
Requires-Dist: PyYAML>=5.4; extra == "yaml"
Dynamic: license-file

<p align="center">
  <img src="docs/assets/banner.svg" alt="HTPD - Hyper Text Portable Document" width="100%">
</p>

<p align="center">
  <a href="https://github.com/narendrachampaneri/htpd/actions/workflows/tests.yml"><img src="https://img.shields.io/github/actions/workflow/status/narendrachampaneri/htpd/tests.yml?branch=main&label=tests&style=for-the-badge" alt="tests"></a>
  <img src="https://img.shields.io/badge/converter-C%2B%2B23-1d4ed8?style=for-the-badge" alt="C++23">
  <img src="https://img.shields.io/badge/reader-Python%203.10%2B-0f766e?style=for-the-badge" alt="Python 3.10+">
  <img src="https://img.shields.io/badge/license-Apache--2.0-475569?style=for-the-badge" alt="Apache-2.0">
  <img src="https://img.shields.io/badge/status-0.1%20alpha-b45309?style=for-the-badge" alt="0.1 alpha">
</p>

<p align="center">
  <a href="#what-is-htpd"><b>What it is</b></a> &nbsp;·&nbsp;
  <a href="https://github.com/narendrachampaneri/htpd/releases"><b>Download</b></a> &nbsp;·&nbsp;
  <a href="#quick-start"><b>Quick start</b></a> &nbsp;·&nbsp;
  <a href="#use-it-with-your-ai"><b>Use with your AI</b></a> &nbsp;·&nbsp;
  <a href="#the-format"><b>Format</b></a> &nbsp;·&nbsp;
  <a href="#results"><b>Results</b></a> &nbsp;·&nbsp;
  <a href="#roadmap"><b>Roadmap</b></a>
</p>

<p align="center">
  <img src="docs/assets/demo.gif" alt="Converting a data sheet with htpd and asking questions with htpd-read" width="88%">
  <br><sub>Real commands on the NXP i.MX 93 data sheet (answer output shortened).</sub>
</p>

---

## What is HTPD?

You have a 300-page datasheet or a 5000-page reference manual and a question
about it. You upload the PDF to an AI, and it answers slowly, misses details,
mixes up tables, cannot tell you the page - or invents an answer that is not
in the document at all.

**HTPD fixes the input instead of the model.** It has two parts:

| | Part | What it does |
|---|---|---|
| **A** | **Converter** (`htpd`, C++) | Turns a PDF into HTPD: the exact text, cut into sections at the document's own headings (or into parts of a few pages when it has none), every paragraph with its page, tables kept as rows, code kept as code, headers and footers removed, figures stored as pictures with their caption, and text inside pictures read by OCR. |
| **B** | **Reader** (`htpd-read`, skill, MCP server, Python) | Lets any AI use an HTPD document: it finds the few sections that answer a question and hands over only that text - with pages - or says **"Not found in this file."** It can also hand over any figure as an image. |

```mermaid
flowchart LR
    PDF["📄 PDF<br/>56 MB, 5652 pages"] -->|htpd convert| H["HTPD<br/>sections · pages · tree"]
    H -->|upload| CHAT["AI chat<br/>(file + skill)"]
    H -->|htpd-read / MCP| AGENT["AI agent<br/>(Claude Code, Antigravity, Codex)"]
    CHAT --> ANS["Answer from the file<br/>with page numbers"]
    AGENT --> ANS
```

<p align="center">
  <img src="docs/assets/features.svg" alt="Exact text - Every fact has a page - Says not found - 100 % local" width="100%">
</p>

<p align="center">
  <img src="docs/assets/before-after.svg" alt="A PDF page becomes a clean HTPD section with its place in the document and page marks" width="100%">
</p>

---

## Quick start

### Download (no build needed)

Get the files from the [**release page**](https://github.com/narendrachampaneri/htpd/releases):

| File | What it is |
|---|---|
| `htpd-0.1.2-linux-x64.tar.gz` | The converter, ready to run - Linux x64, Ubuntu 24.04 or newer, also WSL on Windows |
| `htpd-skill.zip` | The skill for AI apps ([how to add it](#2-apps-with-skills---the-htpd-skill)) |

```sh
tar xzf htpd-0.1.2-linux-x64.tar.gz
cd htpd-0.1.2-linux-x64
./htpd convert manual.pdf --single           # writes manual.htpd - give it to your AI
```

### Build from source

**Requirements:** Linux (tested on Ubuntu 24.04, also in WSL), g++ 13+, make or CMake, Python 3.10+.

```sh
git clone https://github.com/narendrachampaneri/htpd.git
cd htpd

./scripts/get_pdfium.sh        # downloads the PDF engine once (PDFium, about 4 MB)
make                           # builds build/htpd
make test                      # optional: the C++ tests
```

**Convert a PDF**

```sh
./build/htpd convert manual.pdf --single     # one file: manual.htpd  (for AI chats)
./build/htpd convert manual.pdf              # a folder: manual.htpd/ (for tools and agents)
```

**Ask a question from the command line**

```sh
python3 -m pip install -e .                  # installs htpd-read and htpd-mcp

# try it on the sample file that comes with the repository
htpd-read ask examples/single-file/sample.htpd "What is the maximum bus voltage?"
```

```
[page 2, 2.1.1 Limits]
  Parameter | Min | Typ | Max | Unit
  Bus voltage | 200 | 400 | 480 | V
...
```

Each answer line is copied word for word from the file, with its page and
section; `--context` adds the whole sections for the AI to read.

```sh
htpd-read ask examples/single-file/sample.htpd "Which OCPP version is supported?"
```

```
Not found in this file. (the file never mentions OCPP)
```

Then use your own file the same way: `htpd-read ask manual.htpd "your question"`.

---

## Use it with your AI

### 1. Any AI chat - just upload the file

`htpd convert manual.pdf --single` writes **one `.htpd` file**. It begins
with the reading rules for the model - *answer only from this file, give the
page of every fact, say "Not found in this file." when the answer is not
there* - followed by the contents and every section with its pages.

Upload it to Claude, ChatGPT or any other assistant instead of the PDF.
If an app does not accept the extension, rename it to `manual.htpd.txt` -
it is plain text.

> Best for documents that fit in the chat's context (a datasheet of a few
> hundred pages). For very large manuals, use the skill below.

For a ChatGPT custom GPT or a Gemini Gem that always has your document and
follows the rules, see [`docs/AI-APPS.md`](https://github.com/narendrachampaneri/htpd/blob/main/docs/AI-APPS.md) (instructions
ready to paste).

### 2. Apps with skills - the `htpd` skill

Download `htpd-skill.zip` from the [release page](https://github.com/narendrachampaneri/htpd/releases),
or build it from the repository:

```sh
python3 scripts/build_skill.py               # writes dist/htpd-skill.zip
```

| App | How to add it |
|---|---|
| **Claude app** | *Customize → Skills*, upload `htpd-skill.zip`, switch it on (code execution must be on). Then upload a `.htpd` file to a chat and ask. |
| **Claude Code** | Install the plugin ([section 4](#4-claude-code---the-htpd-plugin)), or unzip into `~/.claude/skills/` (gives `~/.claude/skills/htpd/`). |
| **Other agents** that read `SKILL.md` | Unzip into their skills folder. |

The skill runs a small script that needs **only Python** - no packages, no
model. It searches the file, returns the exact lines with `[page, section]`,
and says why when something is not in the file. Because the model reads only
what the script returns, this also works for manuals far too large for any
chat.

### 3. Agents with MCP - the `htpd-mcp` server

```sh
pip install htpd-reader        # the reader: htpd-read and htpd-mcp, nothing else needed
```

The server is also listed in the official MCP registry as
`io.github.narendrachampaneri/htpd` (MCP clients start it with `uvx htpd-reader`).

```sh
claude mcp add htpd -- htpd-mcp --bundles ~/manuals        # Claude Code
```

For Antigravity, Codex or any MCP client: command `htpd-mcp`, arguments
`--bundles /path/to/your/htpd/files`. Tools: `list_bundles`, `contents`,
`open_section`, `search`, `ask`, `figure` (the picture of a figure, as an
image), `check`. Standard library only.

### 4. Claude Code - the `htpd` plugin

One install gives Claude Code everything: the skill that answers from
`.htpd` files, a skill that converts PDFs, and the MCP server.

```
/plugin marketplace add narendrachampaneri/htpd
/plugin install htpd@htpd
```

Then ask in plain words - *"convert datasheet.pdf to htpd"*, *"what is the
maximum NPU frequency in datasheet.htpd?"* - or run `/htpd:convert
datasheet.pdf --single`. The first conversion downloads the converter of the
release (Linux x64 and WSL, about 4 MB, checked by SHA-256) once; an `htpd`
already on the PATH is used instead. Only Python 3.10+ is needed; the PDF
never leaves the computer.

---

## The format

An HTPD document is plain UTF-8 text: Markdown with a few fields. Any
model, editor or `grep` can read it.

<details>
<summary><b>See one section of an HTPD file</b> (shortened)</summary>

````markdown
<!-- htpd section {"n":14,"id":"sections/0014-4-1-5-maximum-frequency-of-main-modules","number":"4.1.5","level":3,"pages":[18,18,18],"parent":"sections/0008-4-1-chip-level-conditions"} -->
## [S14] 4.1.5 Maximum frequency of main modules

Path: 4 Electrical characteristics > 4.1 Chip-level conditions > 4.1.5 Maximum frequency of main modules - page 18

<!-- page 18 -->

```text
Cortex®-A55 cores | 0.9 GHz | 1.4 GHz | 1.7 GHz
DRAM | 933 MHz | 1400 MHz | 1866 MHz
NPU | 500 MHz | 800 MHz | 1000 MHz
```
````

</details>

- **Sections** follow the document's own headings (bookmarks, numbered headings, font sizes) and form a tree: parent, children, previous, next.
- **Page marks** `<!-- page N -->` stand above the text of every page.
- **Tables** are rows with ` | ` between cells; **code and listings** keep their spacing.
- **Two forms, same content:** a single `.htpd` file (for chats) and a bundle folder with one file per section (for tools), which is also a valid [Open Knowledge Format](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle.

The full specification - every field and every rule - is in
[`docs/FORMAT.md`](https://github.com/narendrachampaneri/htpd/blob/main/docs/FORMAT.md). Examples:
[`examples/single-file/sample.htpd`](https://github.com/narendrachampaneri/htpd/blob/main/examples/single-file/sample.htpd) and
[`examples/sample.htpd/`](https://github.com/narendrachampaneri/htpd/blob/main/examples/sample.htpd/).

---

## Results

The converter on real documents (single file, figures and OCR on, a 2-core
cloud machine):

| Document | Pages | Sections | Figures | PDF → `.htpd` | Time |
|---|---:|---:|---:|---|---:|
| NXP i.MX 93 Automotive Data Sheet | 109 | 114 | 53 | 1.7 MB → 2.5 MB | 2.1 s |
| NXP i.MX 93 Reference Manual | 5 652 | 5 929 | 637 | 56.3 MB → 39.8 MB | 67 s |
| RISC-V Debug Specification | 119 | 164 | 3 | 2.4 MB → 0.7 MB | 3.4 s |
| Arm AAPCS64 | 65 | 125 | 5 | 0.4 MB → 0.3 MB | 0.3 s |
| RISC-V Unprivileged ISA | 670 | 760 | 25 | 4.5 MB → 2.8 MB | 4.9 s |

Most of a file with figures is their pictures, which stand at its very end:
the text comes first. `--figures off` leaves them out (the i.MX 93 Reference
Manual is then 14 MB). Without the Tesseract program the converter skips
OCR and says which pages it could not read.

The reader on 133 test questions about two public manuals
([`eval/report.md`](https://github.com/narendrachampaneri/htpd/blob/main/eval/report.md), written by the evaluation harness):

| | Right section first | Right section in the first 5 | Right page | Answer in the text for the AI | "Not found" right / wrong |
|---|---:|---:|---:|---:|---:|
| HTPD reader | **0.78** | **0.95** | **0.64** | **0.86** | **0.95 / 0.01** |
| Plain word search over HTPD sections (B3) | 0.64 | 0.87 | 0.55 | 0.83 | never says it |
| Plain word search over 500-token chunks (B1) | 0.32 | 0.72 | 0.51 | 0.83 | never says it |

The text handed to the AI is about 6 % of the whole document.

The converter on 45 PDFs of known text, made by Word, LaTeX, Chrome,
wkhtmltopdf and LibreOffice slides, scanned, and with letters drawn as
shapes (`tests/zoo`): 99 % of the text of the English ones in order, 96 %
of their headings, 89 % of their table rows exact; Hindi and Gujarati from
Word 91 % and 99 % of the text.

**Robustness.** A stress test makes 34 difficult PDFs locally - two
columns, sideways tables, rotated pages, 1500 pages, huge tables, CJK and
Unicode text, encrypted, truncated and corrupted files, a scanned page,
LaTeX and Word exports - and checks that the converter never crashes or
hangs, that every bundle is valid, and that no word is lost.

**Tests.** 96 C++ tests and 150 Python tests run on every change, the
Python tests also without PyYAML, as the skill does in a chat sandbox.

---

## Roadmap

HTPD is version 0.1. It works today; these are the known next steps.

- [x] Converter: sections, pages, tables, code, headers/footers, sideways pages, damaged files
- [x] Structure for documents without headings (parts of a few pages) and a map of the document
- [x] Figures stored as pictures with caption and page; OCR of scanned pages and text in pictures (Tesseract, optional)
- [x] Single-file `.htpd` with reading rules for AI models
- [x] Reader, "not found" rule, `ask`, MCP server, skill for AI apps; figures handed to the AI as images
- [x] A small trained re-ranker (no GPU, a few kilobytes) - the right section first for 78 % of test questions instead of 66 % without it ([`model/README.md`](https://github.com/narendrachampaneri/htpd/blob/main/model/README.md))
- [ ] Other document types: Word, HTML, plain text, images
- [ ] "Not found" for questions in the user's own words (a word the document writes differently)
- [ ] Windows build

**Known limits:** PDF only; what a chart or diagram means is only in its
picture (an AI must look at it); formulas lose their layout; scanned pages
need the free Tesseract program; tables are rows of text (a table whose
first column wraps can come apart); tested mainly with English technical
documents. Details: [`docs/CONVERTER.md`](https://github.com/narendrachampaneri/htpd/blob/main/docs/CONVERTER.md#limits-please-read).

---

## Project layout

<details>
<summary><b>Folders and documentation</b></summary>

```
include/ src/ cli/      the converter (C++23, PDFium is used in one file only)
reader/htpd_reader/     the reader: bundle, check, search, retriever, re-ranker, packer, answer, MCP server
skills/htpd/            the skill for AI apps (scripts/build_skill.py packs it)
skills/convert/         the plugin's skill that converts PDFs
.claude-plugin/         the Claude Code plugin and its marketplace (this repository is both)
eval/                   evaluation harness, baselines, report
model/                  the training of the re-ranker
tests/                  C++ tests, test PDFs, the stress test
docs/                   FORMAT.md · CONVERTER.md · READER.md · BACKGROUND.md
examples/               a converted document in both forms
```

| Document | For |
|---|---|
| [`docs/FORMAT.md`](https://github.com/narendrachampaneri/htpd/blob/main/docs/FORMAT.md) | The format: anyone can write a reader from it |
| [`docs/CONVERTER.md`](https://github.com/narendrachampaneri/htpd/blob/main/docs/CONVERTER.md) | Building, options, how the converter works, what was tested, limits |
| [`reader/README.md`](https://github.com/narendrachampaneri/htpd/blob/main/reader/README.md) | The reader, its commands and its Python API |
| [`docs/READER.md`](https://github.com/narendrachampaneri/htpd/blob/main/docs/READER.md) | How the reader finds the answer, and how it is measured |
| [`docs/BACKGROUND.md`](https://github.com/narendrachampaneri/htpd/blob/main/docs/BACKGROUND.md) | Why HTPD is built the way it is |
| [`AGENTS.md`](https://github.com/narendrachampaneri/htpd/blob/main/AGENTS.md) | Rules for contributors and coding agents |

</details>

---

## Contributing

Issues and pull requests are welcome. Before you change anything, read
[`AGENTS.md`](https://github.com/narendrachampaneri/htpd/blob/main/AGENTS.md); the short version:

- All tests pass before a change is merged: `make test` and `python3 -m pytest`.
- Every new rule in the code comes with a test that fails without it.
- Never lose text and never make text up.
- A change to what is written into an HTPD file updates `docs/FORMAT.md` in the same commit.

## License

HTPD is open source under the [Apache License 2.0](https://github.com/narendrachampaneri/htpd/blob/main/LICENSE): you may use,
change and share it, also in commercial products, as long as you keep the
copyright and license notices ([`NOTICE`](https://github.com/narendrachampaneri/htpd/blob/main/NOTICE)) and mark what you changed.
The license includes a patent grant from every contributor.

PDFium, which the converter uses, is not part of this repository and has its
own license (BSD-style with an Apache-2.0 part); after
`scripts/get_pdfium.sh` its license texts are in
`third_party/pdfium/licenses/`. If you give the program to others together
with `libpdfium.so`, give them those files too.

---

<div align="center">

Made by **Narendra Champaneri**

</div>

<!-- mcp-name: io.github.narendrachampaneri/htpd -->
