Metadata-Version: 2.4
Name: sprawll
Version: 0.0.1
Summary: Evidence-gated scrape-to-artifact harness: artifacts you can defend
License: MIT
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pydantic>=2.0
Requires-Dist: typer>=0.12
Requires-Dist: rich>=13.0
Requires-Dist: httpx>=0.27
Requires-Dist: warcio>=1.7
Requires-Dist: markitdown[pdf]>=0.1
Requires-Dist: lettucedetect>=0.2
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Provides-Extra: crawl
Requires-Dist: crawl4ai; extra == "crawl"
Provides-Extra: llm
Requires-Dist: litellm; extra == "llm"
Dynamic: license-file

<div align="center">

# sprawll

**Research artifacts you can defend.**

[![CI](https://github.com/Wasif-ZA/sprawll/actions/workflows/ci.yml/badge.svg)](https://github.com/Wasif-ZA/sprawll/actions/workflows/ci.yml)
[![PyPI version](https://img.shields.io/pypi/v/sprawll.svg)](https://pypi.org/project/sprawll/)
[![Python versions](https://img.shields.io/pypi/pyversions/sprawll.svg)](https://pypi.org/project/sprawll/)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)

Every AI research tool can write you a report.<br>
sprawll is the one that <b>refuses to publish a claim it can't pin to an archived source</b>.

</div>

---

## Highlights

- 🧾 **Evidence-gated output** — only claims that survive span-level verification against archived sources ship; everything else is withheld, never silently included
- 📦 **Byte-for-byte archives** — every source captured as WARC, with canonical snapshots every claim offset points into
- 🔢 **Deterministic numeric guard** — every figure in a claim must literally exist in a cited source
- 🔒 **Local-first** — prefers a local Ollama model; sources and drafts never leave your machine
- 💸 **Hard budget cap** — API fallback is checked against a dollar ceiling *before* every call
- 🔁 **Replayable** — re-verify any run from its archive with zero network access

## Quick start

```console
$ pip install sprawll
$ sprawll research-pack "solar e-bike charging on university campuses" \
      -u https://example.edu/solar-project \
      -u https://example.org/cycling-economy
```

What you get back:

| Artifact | What it is |
| --- | --- |
| `report.md` | Only claims that survived verification, each cited; single-source claims flagged |
| `diagnostics.md` | Every claim that failed, with the exact unsupported span named |
| `corpus.warc.gz` | The sources as they existed when fetched, byte-for-byte |
| `snapshots/` | The canonical extracted text every claim offset points into |

> [!NOTE]
> The promise is precise: *defensible against this archived corpus, retrieved on these dates.* Not "true". Nobody can honestly sell you "true"; we can sell you "checkable".

## How it verifies

Source-first pipeline:

```
fetch → archive (WARC) → extract → harvest facts → draft (only from harvested facts) → ground → render
```

The grounding gate has two layers:

1. **Span model** ([LettuceDetect](https://github.com/KRLabsOrg/LettuceDetect), MIT) — checks every claim against source text, chunked with tracked offsets
2. **Numeric guard** (deterministic) — every figure in a claim must literally exist in a cited source. Catches the hedged wrong numbers models gloss over

Publication is binary: `supported` claims ship, everything else goes to diagnostics. Claims one source supports and another contradicts publish as **supported but contested**. If the span model is unavailable, the LLM-judge fallback brands the whole report **DEGRADED**.

## Local-first

sprawll prefers a local LLM. If [Ollama](https://ollama.com) is running, your sources and drafts never leave your machine:

```console
$ ollama pull qwen3:14b   # ~9 GB, fits a 12 GB GPU
```

API keys (`ANTHROPIC_API_KEY` / `OPENAI_API_KEY`) are the fallback, capped by a hard dollar ceiling (`SPRAWLL_BUDGET_USD`, default $2 — checked *before* every call). Pin explicitly:

```console
$ export SPRAWLL_LLM=ollama/qwen3:14b
$ export SPRAWLL_LLM=litellm/anthropic/claude-sonnet-5
```

No LLM at all? sprawll refuses to draft undefended text — `sprawll audit` still grounds claims you wrote yourself.

## Replay and audit

```console
$ sprawll research-pack --from-archive runs/<run-dir>   # re-verify, no network
$ sprawll audit claims.json snapshots/                  # just the grounding gate
```

Failed sources are taxonomized (`fetch_failed`, `robots_blocked`, `paywalled`, `timeout`, `extract_empty`) and shown in the report's source table — a source that couldn't be archived is a fact about the corpus, not a rounding error.

## Install notes

> [!TIP]
> Core is pure-Python pip. Fetching uses [crawl4ai](https://github.com/unclecode/crawl4ai) (real browser) when installed — `pip install sprawll[crawl]` then `playwright install chromium` — and falls back to plain HTTP (marked DEGRADED in fetch notes) when not.

- First grounding run downloads the LettuceDetect weights (~1.5 GB).

## License

[MIT](LICENSE)

---

<div align="center">
<sub>Defensible beats plausible.</sub>
</div>
