Metadata-Version: 2.4
Name: scrape-plan
Version: 0.1.0
Summary: Find the fastest legal way to extract data from any page: robots.txt for your exact path, the internal JSON API the page actually calls, and ready-to-run code.
Project-URL: Homepage, https://github.com/sjh9714/scrape-plan-cli
Project-URL: Issues, https://github.com/sjh9714/scrape-plan-cli/issues
Author-email: Jinhyuk Sung <jinhyuk9714@gmail.com>
License: MIT
License-File: LICENSE
Keywords: api-discovery,cli,crawler,robots-txt,scraping,web-scraping
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.9
Requires-Dist: requests>=2.28
Provides-Extra: browser
Requires-Dist: playwright>=1.40; extra == 'browser'
Description-Content-Type: text/markdown

# scrape-plan

**Before you write a scraper, find out which of three approaches this page actually needs — and whether you're allowed to.**

```console
$ scrape-plan https://quotes.toscrape.com/scroll

https://quotes.toscrape.com/scroll

  robots.txt   allowed   no robots.txt
  body in HTML no        95 raw vs 16,816 rendered (ratio 0.006)

  Tier 1  Call the internal JSON API
  This page calls a JSON endpoint directly. Hitting it without a browser is the
  fastest option and survives redesigns far better than parsing markup.

  internal JSON API (1)
    https://quotes.toscrape.com/api/quotes?page=1
      GET 200 · application/json  seen 19x
      shape {has_next, page, quotes, tag, top_ten_tags}
      pagination query parameter ?page=

# ready-to-run code follows...
```

That page has 95 characters of HTML and 16,816 characters on screen. Every
rendering-based tool concludes "client-side rendered, you need a browser" and
you accept ~2 seconds per page. But the page is just calling a JSON API, and
calling it directly takes ~0.1 seconds. **Over 1,000 pages that is 33 minutes
versus 1 minute 40.**

Nothing told you the API was there. That is the gap this fills.

## The three tiers

| | Situation | What to do | Cost per page |
|---|---|---|---|
| **1** | The page calls an internal JSON API | Call it directly | ~0.1s, rarely breaks |
| **2** | No API, but the body is in the raw HTML | Plain HTTP + a parser | ~0.3s |
| **3** | Neither | Headless browser | ~2s, breaks often |

Checking 1 and 2 before reaching for 3 is the highest-leverage decision in
extraction work. Most people skip straight to 3 because nothing tells them not to.

## Install

```bash
pip install scrape-plan[browser]
playwright install chromium
```

The browser is only needed to detect internal APIs (tier 1) — that requires
watching what the page actually requests. Without it, `scrape-plan --no-browser`
still evaluates robots.txt and distinguishes tier 2 from tier 3:

```bash
pip install scrape-plan     # no browser, no Playwright download
```

## Usage

```bash
scrape-plan https://example.com/products      # full analysis + code
scrape-plan URL --code                        # only the generated code
scrape-plan URL --json                        # machine-readable
scrape-plan URL --no-browser                  # skip tier-1 detection
scrape-plan URL --scrolls 6                   # more scrolling for lazy loaders
```

Exit codes, so it composes in scripts:

| Code | Meaning |
|---|---|
| `0` | Analyzed, crawling permitted |
| `2` | `robots.txt` disallows this path |
| `3` | Could not reach or analyze the target |

```bash
scrape-plan "$URL" --no-browser >/dev/null || echo "skipping $URL"
```

## It tells you when *not* to scrape

`robots.txt` is evaluated for **your exact path**, with correct group
boundaries, wildcards, `$` anchors, and longest-match-wins precedence. The rule
that matched is printed, so you can check the verdict rather than trust it.

```console
$ scrape-plan "https://apps.shopify.com/search?q=test" --no-browser

  robots.txt   DISALLOWED   Disallow: *q=*
  ...
  This path is disallowed. Do not crawl it.

  No code generated. Re-run with --force if you have permission.
```

**No runnable code is generated for a disallowed path.** A tool that flags the
problem and then hands you the scraper anyway has not actually flagged anything.

## Generated code

The code is not a sketch. It runs, and it carries the things people add last and
therefore never add: a request interval and a User-Agent with a contact address.

The tier-2 snippet reads `resp.content`, never `resp.text`. When a server
declares no charset, `requests` falls back to ISO-8859-1 per RFC 2616 and
silently corrupts non-Latin text — no exception is raised, the data is just
wrong, and you find out much later from someone who can read the language.

## What it is not

This does not scrape anything for you. Tools like Scrapy, Crawlee and Firecrawl
do that well. This runs *before* them and answers one question: given this page,
what is the cheapest approach that works, and am I permitted to use it?

It also does not defeat bot protection, solve CAPTCHAs, or access anything
behind a login.

## How it works

1. Fetches `robots.txt` from the target's origin and evaluates your exact path.
2. Fetches the raw HTML and measures the visible text, with scripts and styles
   removed.
3. Opens the page in Chromium, scrolls a few times to trigger lazy requests, and
   records every same-page XHR/fetch that returns JSON.
4. Groups endpoints by parameter *names* rather than values, so a paginated API
   hit 19 times shows up once rather than 19 times.
5. Picks the tier and emits matching code.

At most 8 endpoints are probed, so the tool never becomes a source of load on
the site being inspected.

## Tests

```bash
python3 tests/test_core.py     # 24 cases, no network
```

The robots parser and the endpoint grouping carry most of the tests, because
both fail *silently*: a wrong answer looks exactly like a right one.

## Related

A Chrome extension with the same core, for pages you already have open:
[scrape-plan extension](https://github.com/sjh9714/scrape-plan)

## License

MIT © Jinhyuk Sung
