Metadata-Version: 2.4
Name: docsift
Version: 0.1.0
Summary: Convert documents once. Give agents only what they need.
Project-URL: Homepage, https://github.com/anishmoncivarghese/docsift
Project-URL: Repository, https://github.com/anishmoncivarghese/docsift
Project-URL: Changelog, https://github.com/anishmoncivarghese/docsift/blob/main/CHANGELOG.md
Author: Anish Monci Varghese
License: MIT
License-File: LICENSE
Keywords: docling,document-conversion,llm,markdown,markitdown,pdf,rag
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Text Processing :: Markup :: Markdown
Requires-Python: >=3.11
Requires-Dist: pydantic>=2.7
Requires-Dist: tiktoken>=0.8
Requires-Dist: typer>=0.12
Provides-Extra: all
Requires-Dist: docling>=2.0; extra == 'all'
Requires-Dist: markitdown[docx,pdf,pptx,xlsx]>=0.1.1; extra == 'all'
Provides-Extra: docling
Requires-Dist: docling>=2.0; extra == 'docling'
Provides-Extra: markitdown
Requires-Dist: markitdown[docx,pdf,pptx,xlsx]>=0.1.1; extra == 'markitdown'
Description-Content-Type: text/markdown

# DocSift

> Convert documents once. Give agents only what they need.

DocSift converts PDFs and Office documents into clean, structured, AI-ready
Markdown and JSON — locally, with no cloud APIs. Docling handles PDFs;
MarkItDown handles the breadth formats; both sit behind one interface.

**v0.1.0**

## Install

DocSift needs at least one conversion engine:

    pip install "docsift[markitdown]"   # Word, Excel, PowerPoint, HTML, CSV, EPUB
    pip install "docsift[docling]"      # PDFs (large download: ML layout models)
    pip install "docsift[all]"          # both

`pip install docsift` alone installs the CLI but no engine, and conversion will
fail with an install hint.

For local development from a clone:

    uv sync --all-extras

## Usage

    docsift convert report.pdf
    docsift convert report.pdf --engine markitdown
    docsift --version
    docsift compare report.pdf
    docsift compare report.pdf --output ./comparison

`compare` runs every engine on the same document and writes
`<name>.compare.json` (machine-readable metrics) and `<name>.compare.md`
(human-readable report) alongside per-engine output folders.

Output defaults to `./output/` and can be changed with `--output DIR`.

## Chunking and cleaning

Conversion cleans the Markdown (repeated headers/footers, page numbers, image
references) and splits it into token-budgeted chunks with heading context:

    docsift convert report.pdf --max-tokens 800 --overlap 100
    docsift convert report.pdf --keep-image-refs
    docsift convert report.pdf --keep-furniture
    docsift convert report.pdf --no-cache

Results are cached in `~/.cache/docsift` (override with `DOCSIFT_CACHE_DIR`);
an unchanged file with unchanged settings returns instantly.

## Known limitations in 0.1.0

- PDF chunks come from Docling's HybridChunker and are built from the raw
  document, so the cleaning stages (header/footer removal, page-number
  stripping) apply to the exported Markdown but **not** to the chunk text on
  that path.
- `--overlap` has no effect on PDFs chunked by Docling; it applies to the
  fallback Markdown chunker used for other file types.
- The result cache in `~/.cache/docsift` has no automatic eviction; delete
  the directory to reclaim space.

## License

MIT
