Metadata-Version: 2.4
Name: scrape-toolkit
Version: 0.1.0
Summary: Resilient web scraper and crawler CLI
Author-email: gmassello <gmassello@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/gmassello/web-scraper-toolkit
Project-URL: Issues, https://github.com/gmassello/web-scraper-toolkit/issues
Keywords: scraping,crawler,cli,robots.txt,playwright
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Topic :: Software Development :: Libraries
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: httpx>=0.27
Requires-Dist: lxml>=5.2
Requires-Dist: cssselect>=1.2
Requires-Dist: typer>=0.12
Requires-Dist: rich>=13.7
Provides-Extra: browser
Requires-Dist: playwright>=1.44; extra == "browser"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Dynamic: license-file

# web-scraper-toolkit

You give it a URL and tell it which data you want off the page. It gives you back a table (JSON, CSV or SQLite), following links along the way if you ask it to.

It is built for the real web: sites defend themselves against bots, so the tool goes slow, retries when it gets throttled, rotates User-Agents and, when it has to, opens a real browser. And it respects `robots.txt` by default.

## Installation

```bash
pip install scrape-toolkit
```

That gives you the `scrape` command, and it is enough for most sites. If you also want browser mode (for sites that build the page with JavaScript), install the `browser` extra. It is kept separate on purpose: it downloads a whole Chromium, and you will not always need it.

```bash
pip install "scrape-toolkit[browser]"
playwright install chromium
```

### From source

If you are going to work on the tool itself:

```bash
python3 -m venv .venv
.venv/bin/pip install -e ".[dev]"
```

## Your first scrape

There is a ready-made example pointing at `books.toscrape.com`, a site built precisely for practising scraping. Save this as `scraper.example.toml` (it also ships in the repo):

```toml
[crawl]
seeds = ["https://books.toscrape.com/"]
max_depth = 1
max_pages = 25
same_domain = true
follow_selector = "h3 > a"
extract_pattern = "/catalogue/[^/]+/index\\.html$"

[fetch]
fetcher = "http"
delay = 1.0

[selectors]
title = "//h1/text()"
price = ".price_color"
availability = ".availability"
description = "#product_description ~ p"

[export]
formats = ["json", "csv", "sqlite"]
path = "output/books"
```

Run it as is:

```bash
scrape run scraper.example.toml
```

You will see a log line for every page it visits and, at the end, a table with what it wrote. Have a look at `output/books.json`:

```json
{
  "title": "A Light in the Attic",
  "price": "£51.77",
  "availability": "In stock (22 available)",
  "description": "It's hard to imagine a world without A Light in the Attic...",
  "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"
}
```

That is 20 books, also written to `output/books.csv` and `output/books.db`. That is all the tool does: turn pages into rows.

## The concepts you need

**Seed** (`seeds`): the URL it starts from. You can give it several.

**Crawling** (`max_depth`): on top of scraping the seed, follow the links it finds there and scrape those pages too. The depth is how many hops it is allowed: `0` means "the seeds only", `1` means "the seeds and what they link to", and so on. Careful, it grows fast.

**Selector**: how you point at *where* the data lives inside the HTML. There are two languages for that, and the tool accepts both:
- **CSS**, the same one you use to style a page: `.price_color` means "the element with class `price_color`".
- **XPath**, more powerful, good for things CSS cannot do (like reading an attribute): `//h1/text()` means "the text of the `h1`".

The rule that tells them apart: if the selector starts with `/`, `./` or `(`, it is read as XPath. Otherwise, as CSS. You do not have to declare anything.

**Following is not the same as extracting.** This is the point that confuses people the most, and the one that makes a scraper return garbage. In the example, the `books.toscrape.com` home page is a *list* of 20 books: it is the page you reach each book from, but it is not a book itself. If you ask the tool to extract data from every page it visits, you get a row with `title = "All products"` and the price of the first book in the grid, mixed in among the real books. That is why there are two separate options:
- `follow_selector`: which links to **follow** (in the example, `h3 > a`, the links to each book).
- `extract_pattern`: which URLs to **extract data from**, as a regular expression. Pages that do not match are still visited — that is how you reach the ones that do — but they do not pollute the output.

**Assets** (`[assets]`): optionally, on top of the data, download each page's images or PDFs to disk.

## Scraping your own site

**1. Copy the example and change the seed.**

```bash
cp scraper.example.toml my-site.toml
```

Put your URL in `seeds`.

**2. Get the selectors from the browser.** Open the page, F12, right-click the value you want → *Copy* → *Copy selector*. That gives you a CSS selector that works.

But do not paste it as is: Chrome tends to give you something like `div:nth-child(3) > div > span`, which breaks the moment the site moves a row around. Look at the HTML and find a class that describes the data (`.price`, `.product-title`). Those hold up.

**3. Try a single page before you set off crawling.** That way you check your selectors without hammering the site with hundreds of requests. Set `seeds` to the URL of **one** detail page (a product, an article: whatever it is you want to extract), and use:

```toml
max_depth = 0        # do not follow any link
max_pages = 1
# extract_pattern = ...   ← comment it out while you are testing
```

```bash
.venv/bin/scrape run my-site.toml
```

Look at the JSON. If a field comes out `null`, that selector did not match — and when *none* of them match, the tool tells you so in the log ("no selector matched on ..."). Once the data looks right, put the real seed back, raise `max_depth` and uncomment `extract_pattern`.

**4. If junk rows show up, add `extract_pattern`.** Work out what the URLs of the pages you actually want have in common (almost always a shared prefix, like `/product/`) and write the regex that matches them.

## When the site blocks you

It will happen. Most defences are already handled; the table says what the tool does on its own and what you have to touch yourself.

| You see this | What is going on | What to do |
|---|---|---|
| `403` | The site spotted you as a bot | Try `--fetcher browser`. The error message already suggests it |
| `429` | You are asking too fast | It retries on its own, honouring the wait the site asks for. If it persists, raise `delay` |
| `503` | Site down, or Cloudflare-style protection | It retries on its own, waiting longer each time. If it persists, `--fetcher browser` |
| The data comes out `null` | The selector does not match | Check the selector. If you can see it in the browser but not here, the page loads it with JavaScript → `--fetcher browser` |
| It visits nothing | `robots.txt` does not let you in | That is on purpose. Read the disclaimer before overriding it |

If a site blocks you by IP, `proxies` takes a list and rotates through it.

## CLI usage

```bash
scrape run config.toml                  # run with the config
scrape run config.toml --max-depth 3    # override the depth
scrape run config.toml --fetcher browser  # use the headless browser
scrape run config.toml --out data/output  # change where it writes
scrape run config.toml --log-file scrape.log  # save the log to a file
scrape run config.toml --ignore-robots  # see the disclaimer
```

## Config reference

```toml
[crawl]
seeds = ["https://books.toscrape.com/"]  # where it starts (one or many)
max_depth = 1              # link hops allowed. 0 = the seeds only
max_pages = 25             # hard page cap: your safety net
same_domain = true         # do not wander off to other domains
follow_selector = "h3 > a" # which links to follow
extract_pattern = "/catalogue/[^/]+/index\\.html$"  # regex: which URLs to extract from

[fetch]
fetcher = "http"           # "http" (fast) or "browser" (handles JS)
delay = 1.0                # seconds to wait between requests
timeout = 20.0             # how long to wait for a response
max_retries = 3            # retries before giving up
respect_robots = true      # honour robots.txt
proxies = []               # list of proxies, rotated through
# user_agents = [...]      # omit it and a list of real browsers is used

[selectors]                # field_name = selector (CSS or XPath)
title = "//h1/text()"
price = ".price_color"

[export]
formats = ["json", "csv", "sqlite"]   # one or more
path = "output/books"                 # no extension: each format adds its own

[assets]                   # optional: download images/PDFs to disk
selectors = ["#product_gallery img"]
dir = "output/assets"
```

## How it defends against blocks

- **It goes slow**: waits `delay` between requests. If `robots.txt` asks for a longer `Crawl-delay`, the site's number wins.
- **It retries with a brain**: on a `429` or `503` it honours the `Retry-After` header the server sends; if there is none, it waits longer each time (exponential backoff with jitter). A `404` is not retried: insisting will not make it appear.
- **It blends in**: rotates User-Agents from real browsers on every request.
- **It rotates proxies**: round-robin, sharing cookies across all of them.
- **It keeps the session**: cookies persist for the whole crawl.
- **A real browser**: `fetcher = "browser"` drives Chromium via Playwright. The two fetchers are genuinely interchangeable: `robots.txt` and assets travel over the chosen transport too, with its cookies and its proxies.

## Tests

```bash
.venv/bin/pytest
```

HTTP responses are mocked, so the tests never hit a real site.

## Legal disclaimer

This tool is for responsible scraping of sites that allow themselves to be scraped.

- `robots.txt` **is respected by default**. The `--ignore-robots` flag exists for cases where you have explicit permission from the site owner; using it is a deliberate decision and the responsibility lies with whoever runs the tool.
- It is your responsibility to review the site's **Terms of Service** and the applicable law (intellectual property, personal data, unauthorised access) before scraping.
- Set a reasonable `delay`. An aggressive scraper is indistinguishable from a denial-of-service attack.
- Do not scrape personal data or content behind a login without a legal basis for doing so.

## Out of scope

- **CAPTCHA solving**: deliberately not included (ToS grey area).
- **Concurrent crawling**: the crawl is sequential. With delays between requests and a `Crawl-delay` to honour, concurrency buys little and complicates rate limiting.
