Metadata-Version: 2.4
Name: nextflight
Version: 0.3.5
Summary: Extract JSON data from Next.js App Router pages: parses React Server Components (RSC) Flight payloads (self.__next_f.push) from server-rendered HTML or raw RSC fetches into clean Python dicts/lists -- for web scraping, crawling, and data extraction with Scrapy, requests, or httpx.
Author: Aly Reda
License: MIT
Project-URL: Homepage, https://github.com/Aly-Reda/nextflight
Project-URL: Issues, https://github.com/Aly-Reda/nextflight/issues
Keywords: nextjs,next.js,react-server-components,rsc,flight,web-scraping,webscraping,scraper,crawler,scrapy,zyte,html-parser,json-extraction,data-extraction,app-router
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Topic :: Text Processing :: Markup :: HTML
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: fast
Requires-Dist: orjson; extra == "fast"
Provides-Extra: pandas
Requires-Dist: pandas; extra == "pandas"
Provides-Extra: async
Requires-Dist: httpx; extra == "async"
Provides-Extra: all
Requires-Dist: orjson; extra == "all"
Requires-Dist: pandas; extra == "all"
Requires-Dist: httpx; extra == "all"
Dynamic: license-file

# nextflight

**Extract JSON data from any Next.js (App Router) page in Python.**

`nextflight` parses the React Server Components ("Flight") payloads that
Next.js embeds in server-rendered HTML — the
`<script>self.__next_f.push([...])</script>` blocks, or the raw RSC
response you get back from a request sent with an `RSC: 1` header — and
turns them into clean, searchable Python dicts and lists. It works on
**any** Next.js 13+ App Router site out of the box, with no per-site
configuration, which makes it a natural fit for **web scraping**,
**crawling**, and structured **data extraction** with Scrapy, requests,
httpx, or the stdlib alone.

Next.js pages don't put their data in one obvious place — it's spread
across dozens of numbered chunks, cross-referenced with `$`-sigils, and
reshuffled every time the site redeploys. Hardcoding array paths like
`data[3]["children"][0][3]...` breaks the moment that happens.
`nextflight` resolves those references for you and lets you *search* for
the shape of data you want instead — `page.find_by_keys({"price",
"title"})` instead of a brittle index chain.

## Install

```bash
pip install nextflight
```

No required dependencies — stdlib only, so it drops into any existing
Scrapy/Zyte project without touching your dependency tree. A few optional
extras unlock extra features automatically if installed; see
[Optional dependencies](#optional-dependencies).

## Quick start

Two steps: see what's on the page, then fetch the shape of data you want.

```python
from nextflight import extract

page = extract(html_text)      # a string, bytes, or response object

page.keys()                    # ['0', '1', '3f', '20', ...] -- what's here
page["3f"]                     # the resolved JSON for one specific chunk

# In practice, chunk ids are arbitrary per build (they change on
# redeploy), so search for the shape of data you want instead:
listing = page.find_by_keys({"price", "title"})       # first match
listings = page.find_all_by_keys({"price", "title"})  # every match
products = page.find_by_type("Product")               # by @type
everything = page.resolve_all()                        # everything, dereferenced
```

## Usage

### In a Scrapy spider

```python
import scrapy
from nextflight import extract

class MySpider(scrapy.Spider):
    name = "my_spider"

    def parse(self, response):
        page = extract(response.text)
        for item in page.find_all_by_keys({"price", "title"}):
            yield {
                "title": item.get("title"),
                "price": item.get("price"),
                "url": response.url,
            }
```

### Fetching a URL directly (no Scrapy needed)

```python
from nextflight import FlightExtractor

page = FlightExtractor.from_url("https://example.com/product/123")
product = page.find_by_keys({"price", "title"})
```

`from_url` uses only the stdlib, for quick exploration or lightweight
crawling. For anything needing retries, proxies, JS rendering, or
robots.txt handling, fetch the page with your own HTTP client and pass
`response.text` to `extract(...)` instead.

### Raw RSC fetches (no HTML at all)

Sending a request with an `RSC: 1` header — the way Next.js's own
client-side navigation does — returns the raw Flight row stream directly
as the response body, with no HTML wrapper. `extract()` detects and
parses this automatically, same as the HTML-embedded form:

```python
from nextflight import FlightExtractor

# Sets RSC:1 and Next-Url for you, and best-effort auto-discovers a
# build-specific _rsc=<id> from the page's own prefetch links
page = FlightExtractor.from_rsc_url("https://example.com/car/search?page=2")

# Or bring your own client:
import requests
resp = requests.get(
    "https://example.com/car/search",
    params={"page": "2", "_rsc": "1p28d"},   # a build-specific cache key
    headers={"RSC": "1", "Next-Url": "/en/car/search"},
)
page = FlightExtractor(resp.text)
```

If a site also requires a `Next-Router-State-Tree` header, grab it once
from a real browser's network tab and reuse it — it's stable for every
request to the *same route* regardless of query params, so it doesn't
need to be regenerated per request.

### Pages Router support

Older or mixed Next.js deployments use the Pages Router's `__NEXT_DATA__`
blob instead of Flight — already plain JSON, no `$`-refs to resolve:

```python
from nextflight import extract, find_next_data, detect_next_router

router = detect_next_router(html_text)   # "app" | "pages" | "both" | "unknown"

if router == "app":
    data = extract(html_text).find_by_keys({"price", "title"})
else:
    data = find_next_data(html_text)["props"]["pageProps"]
```

### Monitoring a page over time

`diff_pages` compares two crawls of the same URL and reports what
changed — handy for a price or stock watcher:

```python
from nextflight import FlightExtractor, diff_pages

old_page = FlightExtractor.from_url(url)
# ...re-fetch later...
new_page = FlightExtractor.from_url(url)

diff_pages(old_page, new_page)
# {"added": {...}, "removed": {...}, "changed": {"path.to.price": (100, 90)}}
```

For a list of records with a stable id, pass `id_key` — otherwise
inserting one new item shifts every later index and makes everything
after it look changed even though it didn't:

```python
diff_pages(old_page, new_page, id_key="listing_id")
# {"changed": {"items[listing_id=7165546].price": (929900, 899900)}, ...}
```

Or from the command line, polling continuously (`--rsc` for the
lighter-weight RSC payload instead of full HTML each poll):

```bash
nextflight https://example.com/product/123 --watch 60
nextflight https://example.com/car/search --rsc --watch 60
```

### Exporting to a DataFrame or CSV

```python
page = extract(html_text)

df = page.to_dataframe(required_keys={"id", "price"})       # requires pandas
page.to_csv("listings.csv", required_keys={"id", "price"})  # works either way
```

### Command line

```bash
nextflight page.html --keys sections,meta
nextflight https://example.com/product/123 --type Product
nextflight page.html --tree                 # shape summary, no full values
nextflight page.html --all > everything.json
```

## API reference

### `extract(html) -> FlightExtractor`

Shorthand constructor. `html` accepts a plain string, bytes, or a
response-like object (Scrapy's `Response`, `requests.Response`, etc.) —
pass `response` straight from a `parse()` method.

### `FlightExtractor(html, *, strict=False)`

`strict=True` raises `FlightParseError` on a row that's neither valid
JSON nor a recognizable `$`-reference, instead of keeping it as a raw
string. Useful while developing a new scraper; leave off in production so
a handful of odd rows never take down extraction of everything else.

**Exploring a page**

| Method | Returns | What it does |
|---|---|---|
| `.keys()` | `list[str]` | Every chunk id on the page, in order |
| `.kind(chunk_id)` | `str \| None` | Row kind: `"json"`, `"text"`, `"module"`, `"preload"` |
| `.json_keys()` | `list[str]` | Chunk ids holding structured JSON (dict/list) |
| `.html_keys()` | `list[str]` | Text-row chunk ids that look like HTML fragments |
| `.text_keys()` | `list[str]` | All text-row chunk ids, HTML-looking or not |
| `.shape(chunk_id=None, max_depth=3)` | structure summary | Key names + value types, not values — get a feel for a new site fast |
| `.stats()` | `dict` | Chunk count, row-kind breakdown, page size |

**Resolving data** (dereferencing `$`-refs)

| Method | Returns | What it does |
|---|---|---|
| `page["id"]` / `.resolve_chunk("id")` | resolved value | One chunk, fully dereferenced (`page[...]` raises `KeyError` if missing) |
| `.resolve_all()` | `dict` | Every chunk, fully dereferenced |
| `.resolve_json()` / `.resolve_html()` / `.resolve_text()` | `dict` | Only one kind of chunk — cheaper than `resolve_all()` when you don't need everything |
| `.iter_resolved()` | iterator | Like `resolve_all()` but lazy, one chunk at a time |
| `.get("path.to.value", default=None)` | value | Tolerant dotted-path lookup (dict keys, list indices, and React element `"props"`) |
| `.select(*paths, default=None)` | `dict` | Resolve just the named paths, e.g. `page.select("3f.props.price", "3f.props.title")` |

`"3f" in page` and `for k in page` also work, like a dict.

**Searching** (schema-free, works across redeploys)

| Method | Returns | What it does |
|---|---|---|
| `.find_by_keys(required_keys, root=None)` | dict or `None` | First dict containing *all* of `required_keys` |
| `.find_all_by_keys(required_keys, root=None)` | `list` | Every matching dict — for repeated cards/listings |
| `.find_any_keys(any_keys, root=None)` | `list` | Every dict containing *any* of `any_keys` |
| `.find_by_key_pattern(pattern, root=None)` | `list` | Every dict with a key matching a regex, e.g. `r"^price_"` |
| `.find_by_type(type_value, key="@type", root=None)` | `list` | Every dict whose `key` field equals `type_value` |
| `.find_text(pattern, root=None)` | `list` | Distinct string values matching a regex (emails, SKUs, ...) |
| `.find_all(predicate, root=None, max_results=None)` | `list` | Fully custom predicate over every node |
| `.find_one(predicate, root=None)` | value or `None` | Like `find_all` but just the first match |

Pass `include_source=True` on any `find_*` method to get `(node,
chunk_id)` tuples instead of bare nodes, so you can trace a match back to
where it came from. `find_all`/`find_one`/`find_by_keys` resolve chunks
lazily and stop the moment `max_results` is hit — they don't pay to
resolve chunks after a match is already found.

**Fetching**

| Classmethod | What it does |
|---|---|
| `.from_url(url, timeout=15.0, headers=None)` | Fetch and parse a URL, stdlib only |
| `.from_url_async(url, ...)` | Async version for `asyncio.gather(...)` crawls — requires `httpx` |
| `.from_rsc_url(url, headers=None, cookies=None, auto_discover=True)` | Fetch the raw RSC payload instead of full HTML — see "Raw RSC fetches" above |

`from_url`/`from_rsc_url` transparently decompress gzip/deflate/br
responses even if the server ignores the default `Accept-Encoding:
identity` request.

**Diffing and exporting**

| Method | Returns | What it does |
|---|---|---|
| `.diff(other_page, id_key=None)` | `dict` | Compare against another crawl — see "Monitoring a page over time" |
| `.to_json(path=None, indent=2)` | `str \| None` | Dump the fully resolved page to a file, or return as a string |
| `.to_dataframe(records=None, required_keys=None)` | `DataFrame` | Requires pandas |
| `.to_csv(path, records=None, required_keys=None)` | — | Falls back to the stdlib `csv` module without pandas |

### Module-level functions

- **`find_json_ld(html, type_=None) -> list`** — parse
  `<script type="application/ld+json">` blocks, optionally filtered by
  `@type`. Often more stable across redesigns than Flight data — worth
  trying first for product/article/breadcrumb structured data.
- **`find_next_data(html) -> dict | None`** — parse a Pages Router
  `__NEXT_DATA__` blob. `None` if the page doesn't have one.
- **`detect_next_router(html) -> str`** — `"app"`, `"pages"`, `"both"`, or
  `"unknown"`. Run this first if you're not sure which extractor to use.
- **`diff_pages(old, new, id_key=None) -> dict`** — module-level form of
  `.diff()`.

### Command-line reference

```
nextflight <file-or-url>
  [--keys a,b | --all-by-keys a,b | --any-keys a,b
   | --type Product | --text PATTERN | --get path.to.value
   | --json-keys | --html-keys | --tree
   | --router | --next-data | --stats | --watch SECONDS | --all]
  [--rsc] [--redact] [--save out.json]
```

- `--tree` prints a `.shape()` summary instead of full values.
- `--router` / `--next-data` cover Pages Router pages.
- `--rsc` fetches the raw RSC payload instead of full HTML (URL sources
  only) — lighter weight, also works with `--watch`.
- `--watch SECONDS` polls a URL and prints only what changed since the
  last poll.
- `--redact` best-effort scrubs email/phone-shaped strings from output,
  for sharing debug dumps.

## Optional dependencies

Nothing below is required to install or use `nextflight` — each is used
automatically if already present in your environment, and raises a clear
`ImportError` only if you call the one method that needs it.

| Package | Unlocks | Install |
|---|---|---|
| `orjson` | Faster JSON decoding everywhere | `pip install nextflight[fast]` |
| `pandas` | `.to_dataframe()`, nicer `.to_csv()` | `pip install nextflight[pandas]` |
| `httpx` | `.from_url_async()` | `pip install nextflight[async]` |

Or `pip install nextflight[all]` for all three.

## How it works

Flight payloads aren't newline-delimited JSON — text rows
(`id:T<hexByteLen>,<raw bytes>`) are byte-length-prefixed and can contain
literal newlines or run straight into the next row with no separator, and
module/preload rows (`id:I[...]`, `id:HL[...]`) need bracket-aware
parsing. `nextflight` implements the actual row grammar rather than
splitting on `\n`, so it holds up on both well-formed pages and payloads
truncated mid-chunk (e.g. by a proxy that cuts a response short).

Parsing and resolution are both designed to scale roughly linearly with
page size: rows are split cheaply up front, each chunk's JSON is decoded
lazily on first access rather than all at once, and searches
(`find_one`/`find_by_keys`) stop resolving chunks the moment a match is
found instead of resolving the whole page first.

## License

MIT
