Metadata-Version: 2.4
Name: scrape-plan
Version: 0.1.2
Summary: Find the fastest legal way to extract data from any page: robots.txt for your exact path, the internal JSON API the page actually calls, and ready-to-run code.
Project-URL: Homepage, https://github.com/sjh9714/scrape-plan-cli
Project-URL: Issues, https://github.com/sjh9714/scrape-plan-cli/issues
Author-email: Jinhyuk Sung <jinhyuk9714@gmail.com>
License: MIT
License-File: LICENSE
Keywords: api-discovery,cli,crawler,robots-txt,scraping,web-scraping
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.9
Requires-Dist: requests>=2.28
Provides-Extra: browser
Requires-Dist: playwright>=1.40; extra == 'browser'
Description-Content-Type: text/markdown

# scrape-plan

[![PyPI](https://img.shields.io/pypi/v/scrape-plan)](https://pypi.org/project/scrape-plan/)
[![Python](https://img.shields.io/pypi/pyversions/scrape-plan)](https://pypi.org/project/scrape-plan/)
[![License](https://img.shields.io/badge/license-MIT-blue)](LICENSE)

**Before you write a scraper, find out which of three approaches this page actually needs — and whether you're allowed to.**

```console
$ scrape-plan https://quotes.toscrape.com/scroll

https://quotes.toscrape.com/scroll

  robots.txt   allowed   no robots.txt
  body in HTML no        0% of the visible content is already in the raw HTML

  Tier 1  Call the internal JSON API
  This page calls a JSON endpoint directly. Hitting it without a browser is the
  fastest option and survives redesigns far better than parsing markup.

  internal JSON API (1)
    https://quotes.toscrape.com/api/quotes?page=1
      GET 200 · application/json  seen 19x
      shape {has_next, page, quotes, tag, top_ten_tags}
      pagination query parameter ?page=

# ready-to-run code follows...
```

That page has 95 characters of HTML and 16,816 characters on screen. Every
rendering-based tool concludes "client-side rendered, you need a browser" and
you accept ~2 seconds per page. But the page is just calling a JSON API, and
calling it directly takes ~0.1 seconds. **Over 1,000 pages that is 33 minutes
versus 1 minute 40.**

Nothing told you the API was there. That is the gap this fills.

## The three tiers

| | Situation | What to do | Cost per page |
|---|---|---|---|
| **1** | The page calls an internal JSON API | Call it directly | ~0.1s, rarely breaks |
| **2** | No API, but the body is in the raw HTML | Plain HTTP + a parser | ~0.3s |
| **3** | Neither | Headless browser | ~2s, breaks often |

Checking 1 and 2 before reaching for 3 is the highest-leverage decision in
extraction work. Most people skip straight to 3 because nothing tells them not to.

## Install

```bash
pip install scrape-plan[browser]
playwright install chromium
```

The browser is only needed to detect internal APIs (tier 1) — that requires
watching what the page actually requests. Without it, `scrape-plan --no-browser`
still evaluates robots.txt and can confirm tier 2 when the body is already in the
raw HTML. When the raw HTML is nearly empty it reports `Tier ?` rather than
guessing, because at that point the page is either JavaScript-rendered or a block
page and there is no way to tell from the HTML alone:

```bash
pip install scrape-plan     # no browser, no Playwright download
```

## Usage

```bash
scrape-plan https://example.com/products      # full analysis + code
scrape-plan URL --code                        # only the generated code
scrape-plan URL --json                        # machine-readable
scrape-plan URL --no-browser                  # skip tier-1 detection
scrape-plan URL --scrolls 6                   # more scrolling for lazy loaders
```

Exit codes, so it composes in scripts:

| Code | Meaning |
|---|---|
| `0` | Analyzed, crawling permitted |
| `2` | `robots.txt` disallows this path |
| `3` | Could not reach or analyze the target |

```bash
scrape-plan "$URL" --no-browser >/dev/null || echo "skipping $URL"
```

## It tells you when *not* to scrape

`robots.txt` is evaluated for **your exact path**, with correct group
boundaries, wildcards, `$` anchors, and longest-match-wins precedence. The rule
that matched is printed, so you can check the verdict rather than trust it.

```console
$ scrape-plan "https://apps.shopify.com/search?q=test" --no-browser

  robots.txt   DISALLOWED   Disallow: *q=*
  ...
  This path is disallowed. Do not crawl it.

  No code generated. Re-run with --force if you have permission.
```

**No runnable code is generated for a disallowed path.** A tool that flags the
problem and then hands you the scraper anyway has not actually flagged anything.

## Generated code

The code is not a sketch. It runs, and it carries the things people add last and
therefore never add: a request interval and a User-Agent with a contact address.

The tier-2 snippet reads `resp.content`, never `resp.text`. When a server
declares no charset, `requests` falls back to ISO-8859-1 per RFC 2616 and
silently corrupts non-Latin text — no exception is raised, the data is just
wrong, and you find out much later from someone who can read the language.

## What it is not

This does not scrape anything for you. Tools like Scrapy, Crawlee and Firecrawl
do that well. This runs *before* them and answers one question: given this page,
what is the cheapest approach that works, and am I permitted to use it?

It also does not defeat bot protection, solve CAPTCHAs, or access anything
behind a login.

## How it works

1. Fetches `robots.txt` from the target's origin and evaluates your exact path.
2. Fetches the raw HTML and measures how much of the rendered page is already
   present in it, by comparing three-word sequences rather than raw lengths.
   Comparing lengths does not work: hidden menus and templates inflate the raw
   side, which made one server-rendered page look 4x larger than its own
   rendered output.
3. Opens the page in Chromium, scrolls a few times to trigger lazy requests, and
   records every same-page XHR/fetch that returns JSON.
4. Groups endpoints by parameter *names* rather than values, so a paginated API
   hit 19 times shows up once rather than 19 times.
5. Picks the tier and emits matching code.

At most 8 endpoints are probed, so the tool never becomes a source of load on
the site being inspected.

## Tests

```bash
python3 tests/test_core.py     # 24 cases, no network
```

The robots parser and the endpoint grouping carry most of the tests, because
both fail *silently*: a wrong answer looks exactly like a right one.

## Related

A Chrome extension with the same core, for pages you already have open:
[scrape-plan extension](https://github.com/sjh9714/scrape-plan)

## License

MIT © Jinhyuk Sung
