Metadata-Version: 2.5
Name: llm-web-crawler
Version: 2.7.2
Summary: LLM data collection and synthetic fine-tuning dataset pipeline
License-File: LICENSE
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Python: >=3.11
Requires-Dist: aiofiles>=23
Requires-Dist: beautifulsoup4>=4.12
Requires-Dist: datasets>=2.20
Requires-Dist: httpx[http2]>=0.27
Requires-Dist: huggingface-hub>=0.23
Requires-Dist: jinja2>=3.1
Requires-Dist: kaggle>=1.6
Requires-Dist: keyring>=24
Requires-Dist: litellm>=1.40
Requires-Dist: loguru>=0.7
Requires-Dist: lxml>=5
Requires-Dist: markdownify>=0.12
Requires-Dist: psutil>=5.9
Requires-Dist: pyarrow>=16
Requires-Dist: pydantic-settings>=2.3
Requires-Dist: pydantic>=2.7
Requires-Dist: python-dotenv>=1.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: questionary>=2.0
Requires-Dist: rich>=13
Requires-Dist: sqlmodel>=0.0.18
Requires-Dist: tenacity>=8.3
Requires-Dist: tiktoken>=0.7
Requires-Dist: truststore>=0.10
Requires-Dist: typer>=0.12
Requires-Dist: xmltodict>=0.13
Provides-Extra: dev
Requires-Dist: mcp<3,>=2.2; extra == 'dev'
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pip-audit>=2.7; extra == 'dev'
Requires-Dist: pyinstaller>=6; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest-cov>=5; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: respx>=0.21; extra == 'dev'
Requires-Dist: ruff>=0.4; extra == 'dev'
Provides-Extra: mcp
Requires-Dist: mcp<3,>=2.2; extra == 'mcp'
Provides-Extra: unsloth
Requires-Dist: unsloth>=2024.0; extra == 'unsloth'
Description-Content-Type: text/markdown

# DataForge

**Turn any website into a fine-tuning dataset — in one command.**

[![PyPI](https://img.shields.io/pypi/v/llm-web-crawler)](https://pypi.org/project/llm-web-crawler/)
[![Python](https://img.shields.io/badge/python-3.11%20%7C%203.12%20%7C%203.13%20%7C%203.14-blue)](https://www.python.org/)
[![License](https://img.shields.io/badge/license-MIT-green)](LICENSE)

**Current release: 2.7.2.** **Pages I scraped earlier** now accepts the
run folder, a file in it, or `scrape/` (the newest run is used), and says
exactly where it looked when nothing is found. 2.7.1 added live feedback
while scraping, a working Back, and building a dataset from pages you
already scraped. Tested on Linux, Windows and macOS with Python 3.11 to
3.14, with standalone binaries for Windows, Linux and Apple Silicon Macs.
See the [changelog](CHANGELOG.md#272---2026-09-24).

Fine-tuning needs data, and good domain data is trapped in documentation
sites, knowledge bases and public archives. Getting it out usually means
writing a throwaway scraper, a chunker, a prompt loop and an exporter — then
doing it all again for the next domain.

DataForge is that pipeline, already built, tuned, and checkpointed end to
end:

```
sitemap ──▶ crawl ──▶ clean + chunk ──▶ LLM generates Q&A ──▶ score ──▶ JSONL / Parquet / HF Hub
```

```bash
pip install llm-web-crawler
dataforge init-recipe fema.yaml   # write a starter config
dataforge run fema.yaml           # crawl, generate, score, export — unattended
```

You get a versioned, deduplicated, quality-scored dataset in ChatML JSONL,
Parquet and CSV, ready for Unsloth, Axolotl, TRL or HuggingFace `datasets`.

```
Discovery → Collection → Processing → Generation → Quality → Export
```

See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for what each stage does.

## What you'll need

Before you run anything, here's the honest list of what's required vs.
optional:

| | Required? | Notes |
|---|---|---|
| **Python 3.11 to 3.14** | ✅ Required | Tested on Linux, Windows and macOS. Or skip it entirely with a [standalone binary](docs/INSTALLATION.md#standalone-executables-no-python-required) — no interpreter needed. |
| **An LLM to generate and score samples** | ✅ Required (one of the below) | This is what actually writes the Q&A pairs. |
| — A hosted provider API key (OpenAI, Anthropic, Google, Groq, or Together) | One of these, or the local option below | `dataforge config` walks you through it; stored in `.env`. Content leaves your machine for generation. |
| — A local model, no key | *or* fully local | **Ollama:** `ollama serve && ollama pull llama3.2`, then `dataforge config` → `ollama`. **Any OpenAI-compatible server** (LM Studio, vLLM, llama.cpp, Lemonade): set `DATAFORGE_LLM_PROVIDER=openai_compatible`, `DATAFORGE_LOCAL_BASE_URL=http://localhost:1234/v1` and `DATAFORGE_LLM_MODEL` to a model ID the server lists. Nothing leaves your machine. |
| **A HuggingFace account** | ❌ Optional | Only if you set `export.targets: [huggingface]` to publish the finished dataset to the Hub. Needs `HUGGINGFACE_TOKEN`. |
| **A Kaggle account** | ❌ Optional | Only if you set `export.targets: [kaggle]`. Needs `KAGGLE_USERNAME` / `KAGGLE_KEY`. |
| **Nothing else** | — | Local JSONL/Parquet/CSV export (the default) needs no account at all — the whole pipeline runs against just an LLM provider. |

In short: **one LLM (hosted key, or a local model through Ollama or any
OpenAI-compatible server) is the
only hard requirement.** Everything else — which provider, which export
target, whether you need a HuggingFace or Kaggle account — is a choice you
make in the recipe, not a prerequisite to get started. Full setup for each
option is in [docs/INSTALLATION.md](docs/INSTALLATION.md) and
[docs/CONFIGURATION.md](docs/CONFIGURATION.md).

## Why DataForge

| | |
|---|---|
| **Configuration as code** | A YAML recipe captures every decision. Commit it, review it in a PR, re-run it in CI — the same input produces the same dataset. |
| **Unattended by default** | `dataforge run` needs no prompts. Cron it, or drive it interactively with `dataforge` when exploring a new site. |
| **Streaming pipeline** | Generation starts on the first page instead of waiting for the last one, so the LLM and the crawler work at the same time. |
| **Resumable, not restartable** | Every page, chunk and sample is checkpointed to SQLite. Interrupt a 5,000-page crawl and resume exactly where it stopped — nothing is re-fetched or re-billed. |
| **Any model** | OpenAI, Anthropic, Google, Groq, Together, or fully local via Ollama or any OpenAI-compatible server (LM Studio, vLLM, llama.cpp), so sensitive content never has to leave your machine. |
| **Leak-free by design** | Train/validation/test splits are page-aware, not sample-aware, so paraphrases of the same source never land on both sides of a split. |
| **Polite by construction** | `robots.txt` honoured, per-domain rate limiting, URL sanitisation, and PII/copyright guidance built in. |

## A worked example

Building a U.S. disaster-preparedness dataset from Ready.gov. The full
recipe ships with the project at
[`examples/ready-gov.yaml`](examples/ready-gov.yaml):

```yaml
version: 1
name: ready-gov-preparedness
stream: true

source:
  urls:
    - https://www.ready.gov/sitemap.xml
  language: en
  exclude: ['/node/', '/press-release', 're:toolkit']
  max_urls: 150

generation:
  format: qa
  goal: >
    Accurate Q&A about U.S. disaster preparedness, grounded strictly in
    official Ready.gov and FEMA guidance.
  n_per_chunk: 3

quality:
  threshold: 0.5

export:
  targets: [local]
  split:
    train: 0.8
    validation: 0.1
    test: 0.1
    group_by: page
```

```bash
dataforge run examples/ready-gov.yaml --dry-run   # validate + preview the plan
dataforge run examples/ready-gov.yaml             # execute
```

One sitemap becomes a few hundred curated pages, a few thousand chunks, and
a scored, deduplicated, leak-free Q&A dataset — without answering a single
prompt.

## Get started

```bash
pip install llm-web-crawler
dataforge init-recipe my.yaml       # write an annotated starter recipe
dataforge run my.yaml --dry-run     # validate and print the plan, run nothing
dataforge run my.yaml               # execute end-to-end, no prompts
```

**Just need the pages, or a table on them?** `dataforge scrape` fetches pages
and saves their text as Markdown and every table as CSV and JSON. It needs no
AI, no API key and spends nothing, and it obeys `robots.txt` and rate limits
like everything else:

```bash
dataforge scrape https://www.husd.us/schools/middle-schools -o schools
# schools/page_001_table_1.csv:
# Middle Schools,Address,Phone,Fax
# Anthony Ochoa,"2121 Depot Road Hayward, CA 94545-2428",(510) 723-3130,(510) 786-0559
# ...
```

Add `--check` for rule-based checks (empty or very short pages, duplicate
text), `-f csv` to write only the tables, and `--json` before the command for
a machine-readable summary. AI agents get the same through the MCP tool
`scrape_page`. To turn a scrape folder into training data later, start
**Build an AI training dataset** in `dataforge` and choose **Pages I scraped
earlier** (a run folder, or `scrape/` for the newest run): the pages are not
fetched again and the run starts at processing.

To update or remove DataForge, run `dataforge update` or `dataforge uninstall`
yourself, in your own terminal. Both close DataForge first, so nothing is
replaced while in use; `uninstall` asks whether to keep your data. Read the
[release notes](https://github.com/ianktoo/data-forge/releases) before
updating: a new version can change recipes, outputs or commands. See
[Updating and uninstalling](docs/INSTALLATION.md#updating-and-uninstalling-from-dataforge).

Prefer to explore a site interactively first? `dataforge` launches the
guided wizard with a per-stage review. Full install options (uv, pip,
standalone binaries, local models via Ollama), every CLI command, all
environment variables and the complete recipe schema are in the docs:

- **[Installation](docs/INSTALLATION.md)** — uv / pip / source / standalone binaries / Ollama, plus [known issues and workarounds](docs/INSTALLATION.md#known-issues-and-workarounds)
- **[Configuration](docs/CONFIGURATION.md)** — environment variables and full recipe reference
- **[Architecture](docs/ARCHITECTURE.md)** — how discovery, streaming, quality and export fit together
- **[Development](docs/DEVELOPMENT.md)** — project layout, testing, releasing
- **[Ethics & data residency](docs/ETHICS.md)** — responsible use, PII, copyright, rate limits
- **[Third-party libraries](docs/THIRD_PARTY.md)** — full dependency and license list
- **[Technical writeup](docs/TECHNICAL.tex)** — design rationale and references

## Using DataForge from an AI agent

Claude Code, Codex, Cursor and other coding agents can drive DataForge through
its non-interactive commands. Point the agent at the built-in guide:

```bash
dataforge agent-guide
```

It lists which commands are safe to run without a terminal, how to configure a
provider through environment variables, the recipe workflow and exit codes, and
the responsible-use rules an agent must follow before crawling a site.

For clients that support the [Model Context Protocol](https://modelcontextprotocol.io),
DataForge also runs as a local MCP server. Nothing is hosted; the client starts
it on your machine:

```bash
pip install "llm-web-crawler[mcp]"
claude mcp add dataforge -- dataforge mcp   # run from your project directory
```

Claude Code picks up a newly added server only in a new session: exit with
`/exit`, then run `claude --continue` in your terminal to reopen the same
conversation with the DataForge tools loaded.

The agent then gets typed tools (scrape a page's text and tables, explore a site, validate a recipe, start and
monitor a run, read session stats and samples) instead of parsing terminal
output. Runs started this way must have a spending cap, and a recipe that
disables `robots.txt` is refused.

Agents may never update, upgrade or uninstall DataForge. `dataforge update` and
`dataforge uninstall` refuse to run without a terminal or inside an agent
session, and the agent guide tells agents to leave it to you, after you have
read the release notes.

## What makes it different

Most scrapers stop at "here's the text." DataForge is built for the part
that actually determines whether a fine-tune works:

- **It knows a scraped dataset isn't i.i.d.** Several Q&A pairs come from
  one chunk, several chunks from one page — so a naive random split leaks
  paraphrases across train/test and quietly inflates your eval score.
  DataForge splits by page and verifies the property before writing.
- **It treats web failure modes as data, not noise.** `429` isn't `404`,
  `Retry-After` isn't ignored, and `robots.txt` `Crawl-delay` is honoured
  automatically — so a run against a real, imperfect site finishes instead
  of half-failing silently.
- **It doesn't make you choose between fast and resumable.** Streaming mode
  overlaps crawling and generation for throughput; every stage still
  checkpoints to SQLite, so a killed process loses only the item in flight.
- **The quality bar is layered, not a single score.** Length heuristics,
  a source-reference filter, deduplication, and an LLM judge that fails
  closed — with a rejection breakdown so you can see what's actually being
  filtered out.

## License

MIT — see [LICENSE](LICENSE) for details.

If you use DataForge-generated datasets in a publication or project,
attribution is appreciated but not required:

```
Too, I. K. (2026). DataForge: A Streaming, Leak-Aware Pipeline for Synthetic
LLM Fine-Tuning Datasets (Technical Note). Zenodo.
https://doi.org/10.5281/zenodo.22906071
```

```bibtex
@techreport{too2026dataforge,
  author      = {Too, Ian K.},
  title       = {{DataForge}: A Streaming, Leak-Aware Pipeline for Synthetic
                 {LLM} Fine-Tuning Datasets},
  type        = {Technical Note},
  institution = {Zenodo},
  year        = {2026},
  doi         = {10.5281/zenodo.22906071},
  url         = {https://doi.org/10.5281/zenodo.22906071}
}
```

GitHub's "Cite this repository" button reads the same details from
[`CITATION.cff`](CITATION.cff).

That DOI covers every version of the technical note and always resolves to
the latest. To cite one exact version, use its own DOI: v1.1.0 is
[10.5281/zenodo.22908858](https://doi.org/10.5281/zenodo.22908858) and v1.0.0
is [10.5281/zenodo.22906072](https://doi.org/10.5281/zenodo.22906072).
