Metadata-Version: 2.4
Name: quarry-kb
Version: 0.4.0
Summary: A config-driven knowledge-ingestion harness — the deterministic half of turning a source into a linked wiki article.
Project-URL: Homepage, https://github.com/asachs/quarry-kb
Author: André Sachs
License-Expression: MIT
License-File: LICENSE
Keywords: cli,ingestion,knowledge,markdown,wiki
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Text Processing :: Markup :: Markdown
Requires-Python: >=3.11
Requires-Dist: pyyaml>=6.0
Provides-Extra: all
Requires-Dist: curl-cffi>=0.7; extra == 'all'
Requires-Dist: faster-whisper>=1.0; extra == 'all'
Requires-Dist: gitingest>=0.1; extra == 'all'
Requires-Dist: pymupdf4llm>=0.0.17; extra == 'all'
Requires-Dist: trafilatura>=1.6; extra == 'all'
Requires-Dist: youtube-transcript-api>=1.0; extra == 'all'
Requires-Dist: yt-dlp>=2024.0; extra == 'all'
Provides-Extra: dev
Requires-Dist: coverage>=7.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Provides-Extra: discovery
Provides-Extra: github
Requires-Dist: gitingest>=0.1; extra == 'github'
Provides-Extra: instagram
Requires-Dist: curl-cffi>=0.7; extra == 'instagram'
Requires-Dist: yt-dlp>=2024.0; extra == 'instagram'
Provides-Extra: pdf
Requires-Dist: pymupdf4llm>=0.0.17; extra == 'pdf'
Provides-Extra: web
Requires-Dist: trafilatura>=1.6; extra == 'web'
Provides-Extra: whisper
Requires-Dist: faster-whisper>=1.0; extra == 'whisper'
Provides-Extra: youtube
Requires-Dist: youtube-transcript-api>=1.0; extra == 'youtube'
Requires-Dist: yt-dlp>=2024.0; extra == 'youtube'
Description-Content-Type: text/markdown

# Quarry

A config-driven **knowledge-ingestion harness** — a CLI that owns the *deterministic*
half of turning a source (a YouTube video, a web page, …) into a linked article in a
markdown knowledge wiki.

Quarry resolves a source adapter, fetches, writes immutable raw material, hands a
machine-readable compile-spec to an LLM (or human) for the one irreducibly-generative
step — writing the article — then verifies provenance, lints, and commits. **It never
calls a model**, so it is runnable and testable without API keys.

> You extract raw material from the world and refine it into something structured.
> That is the pipeline, and the name.

## Why

The deterministic mechanics of disciplined knowledge ingestion — fetch, immutable raw,
a verifiable manifest, provenance checking, structural lint, semantic link discovery —
are worth doing the same way every time. Quarry packages exactly that machinery and
nothing else: it is **not** an LLM wrapper, not a note-taking app, and not opinionated
about your wiki's shape. Every convention is configuration.

## Install

```bash
pip install quarry-kb[all]          # all shipped adapters
pip install quarry-kb               # core only (PyYAML); add adapters as extras
```

Adapters (each an extra): `youtube` (transcript + Whisper fallback), `web`,
`github` (gitingest), `pdf` (PyMuPDF4LLM + OCR), `instagram`, plus `whisper` for
local audio transcription. Adapters extract raw content deterministically — no LLM.
The core needs no API key; a few adapters take *optional* credentials (e.g. Instagram
cookies for private posts) — see each adapter's module docstring for setup.

> **No Reddit adapter (removed in 0.4.0).** Reddit became a losing battle for a
> deterministic, no-auth fetcher: it fingerprint-blocks pure-Python clients via TLS/JA3
> (so it needed `curl_cffi` impersonation just to get a `200`), throttles by IP reputation
> with **no `Retry-After`** (so back-off is guesswork), and its Nov-2025 *Responsible
> Builder Policy* gates **all** API apps — including read-only OAuth — behind manual
> approval. The net result was a fragile, perpetually-degrading path that violated Quarry's
> "runnable without keys, deterministically" principle. Rather than ship something that
> mostly doesn't work, it's gone. To ingest a Reddit thread, save the page (or its `.json`)
> and feed it through the `web`/`pdf` adapter, or add your own adapter via the entry-point
> hook. (`curl_cffi` lives on — the `instagram` extra uses it for yt-dlp's impersonation.)

> Installed as `quarry-kb` on PyPI (the bare name was taken); the command and import
> are both `quarry`.

Requires Python 3.11+.

## 60-second quickstart

```bash
cd your-wiki-repo
quarry init                  # scaffold a documented quarry.toml + gitignore .quarry/
# edit quarry.toml to match your wiki's conventions

quarry ingest <url>          # fetch -> raw/ + compile-manifest, then STOP
# ...now write the wiki article using the printed compile-spec...
quarry finish <slug>         # verify provenance -> lint -> commit (no push unless --push)
```

Other commands: `quarry lint`, `quarry adapters`, `quarry related <article>`,
`quarry densify [--apply]`, `quarry doctor`.

The two-call seam is deliberate: `ingest` and `finish` are separate processes, and the
generative step (writing the article) happens *between* them — done by you or your agent,
never by Quarry.

## Configuration reference

Quarry is configured by a `quarry.toml` discovered by walking up from the current
directory (a `[tool.quarry]` table in `pyproject.toml` is a fallback). `quarry init`
writes a fully-commented default. Tables:

| Table | Key | Purpose |
|-------|-----|---------|
| `[store]` | `wiki`, `raw`, `manifest_dir` | directories (relative to the store root) |
| | `raw_layout` | raw path template; tokens: `{year} {month} {date} {slug} {kebab_title} {ext} {topic} {source_id}` |
| | `root` | explicit store root (defaults to the dir containing `quarry.toml`) |
| `[frontmatter]` | `required`, `sources_field`, `related_field` | article frontmatter schema |
| `[ingest]` | `default_ext`, `slug`, `on_duplicate` | slug template; `on_duplicate` = `refuse \| warn \| allow` |
| `[adapters]` | `enabled` | allowlist of adapters to run |
| `[discovery]` | `backend`, `mode`, `collection`, `dedup_threshold`, `densify_topk` | semantic-link backend (`qmd` or `none`) |
| `[lint]` | `broken_links`, `require_sources_on_disk`, `orphan_check`, `index_file`, `fail_on` | which checks run and which fail the build |
| `[finish]` | `run_lint`, `auto_push`, `commit_template` | finish behaviour |

See [`examples/quarry.toml`](examples/quarry.toml) for the annotated default.

## Authoring an adapter

A new source type is one small adapter discoverable via the `quarry.adapters`
entry-point group — no fork. See [`CONTRIBUTING.md`](CONTRIBUTING.md) for the contract,
the hermetic-test requirement, and a worked `pdf` example.

## Status

Pre-1.0, but feature-complete for v1 and fully tested. See [`SPEC.md`](SPEC.md) for the
design and [`ISA.md`](ISA.md) for the ideal-state criteria and build provenance.

## License

MIT — see [`LICENSE`](LICENSE).
