Metadata-Version: 2.4
Name: nextflight
Version: 0.3.6
Summary: Extract JSON data from Next.js App Router pages: parses React Server Components (RSC) Flight payloads (self.__next_f.push) from server-rendered HTML or raw RSC fetches into clean Python dicts/lists -- for web scraping, crawling, and data extraction with Scrapy, requests, or httpx.
Author: Aly Reda
License: MIT
Project-URL: Homepage, https://github.com/Aly-Reda/nextflight
Project-URL: Issues, https://github.com/Aly-Reda/nextflight/issues
Keywords: nextjs,next.js,react-server-components,rsc,flight,web-scraping,webscraping,scraper,crawler,scrapy,zyte,html-parser,json-extraction,data-extraction,app-router
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Topic :: Text Processing :: Markup :: HTML
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: fast
Requires-Dist: orjson; extra == "fast"
Provides-Extra: pandas
Requires-Dist: pandas; extra == "pandas"
Provides-Extra: async
Requires-Dist: httpx; extra == "async"
Provides-Extra: all
Requires-Dist: orjson; extra == "all"
Requires-Dist: pandas; extra == "all"
Requires-Dist: httpx; extra == "all"
Dynamic: license-file

# nextflight

**Extract JSON data from any Next.js (App Router) page in Python.**

`nextflight` parses the React Server Components ("Flight") payloads that
Next.js embeds in server-rendered HTML — the
`<script>self.__next_f.push([...])</script>` blocks, or the raw RSC
response you get back from a request sent with an `RSC: 1` header — and
turns them into clean, searchable Python dicts and lists. It works on
**any** Next.js 13+ App Router site out of the box, with no per-site
configuration, which makes it a natural fit for **web scraping**,
**crawling**, and structured **data extraction** with Scrapy, requests,
httpx, or the stdlib alone.

Next.js pages don't put their data in one obvious place — it's spread
across dozens of numbered chunks, cross-referenced with `$`-sigils, and
reshuffled every time the site redeploys. Hardcoding array paths like
`data[3]["children"][0][3]...` breaks the moment that happens.
`nextflight` resolves those references for you and lets you *search* for
the shape of data you want instead — `page.find_by_keys({"price",
"title"})` instead of a brittle index chain.

## Install

```bash
pip install nextflight
```

No required dependencies — stdlib only, so it drops into any existing
Scrapy/Zyte project without touching your dependency tree. A few optional
extras unlock extra features automatically if installed; see
[Optional dependencies](#optional-dependencies).

## Quick start

Two steps: see what's on the page, then fetch the shape of data you want.

```python
from nextflight import extract

page = extract(html_text)      # a string, bytes, or response object

page.keys()                    # ['0', '1', '3f', '20', ...] -- what's here
page["3f"]                     # the resolved JSON for one specific chunk

# In practice, chunk ids are arbitrary per build (they change on
# redeploy), so search for the shape of data you want instead:
listing = page.find_by_keys({"price", "title"})       # first match
listings = page.find_all_by_keys({"price", "title"})  # every match
products = page.find_by_type("Product")               # by @type
everything = page.resolve_all()                        # everything, dereferenced
```

## Usage

### In a Scrapy spider

```python
import scrapy
from nextflight import extract

class MySpider(scrapy.Spider):
    name = "my_spider"

    def parse(self, response):
        page = extract(response.text)
        for item in page.find_all_by_keys({"price", "title"}):
            yield {
                "title": item.get("title"),
                "price": item.get("price"),
                "url": response.url,
            }
```

### Fetching a URL directly (no Scrapy needed)

```python
from nextflight import FlightExtractor

page = FlightExtractor.from_url("https://example.com/product/123")
product = page.find_by_keys({"price", "title"})
```

`from_url` uses only the stdlib, for quick exploration or lightweight
crawling. For anything needing retries, proxies, JS rendering, or
robots.txt handling, fetch the page with your own HTTP client and pass
`response.text` to `extract(...)` instead.

### Raw RSC fetches (no HTML at all)

Sending a request with an `RSC: 1` header — the way Next.js's own
client-side navigation does — returns the raw Flight row stream directly
as the response body, with no HTML wrapper. `extract()` detects and
parses this automatically, same as the HTML-embedded form:

```python
from nextflight import FlightExtractor

# Sets RSC:1 and Next-Url for you, and best-effort auto-discovers a
# build-specific _rsc=<id> from the page's own prefetch links
page = FlightExtractor.from_rsc_url("https://example.com/car/search?page=2")

# Or bring your own client:
import requests
resp = requests.get(
    "https://example.com/car/search",
    params={"page": "2", "_rsc": "1p28d"},   # a build-specific cache key
    headers={"RSC": "1", "Next-Url": "/en/car/search"},
)
page = FlightExtractor(resp.text)
```

If a site also requires a `Next-Router-State-Tree` header, grab it once
from a real browser's network tab and reuse it — it's stable for every
request to the *same route* regardless of query params, so it doesn't
need to be regenerated per request.

### Pages Router support

Older or mixed Next.js deployments use the Pages Router's `__NEXT_DATA__`
blob instead of Flight — already plain JSON, no `$`-refs to resolve:

```python
from nextflight import extract, find_next_data, detect_next_router

router = detect_next_router(html_text)   # "app" | "pages" | "both" | "unknown"

if router == "app":
    data = extract(html_text).find_by_keys({"price", "title"})
else:
    data = find_next_data(html_text)["props"]["pageProps"]
```

### Monitoring a page over time

`diff_pages` compares two crawls of the same URL and reports what
changed — handy for a price or stock watcher:

```python
from nextflight import FlightExtractor, diff_pages

old_page = FlightExtractor.from_url(url)
# ...re-fetch later...
new_page = FlightExtractor.from_url(url)

diff_pages(old_page, new_page)
# {"added": {...}, "removed": {...}, "changed": {"path.to.price": (100, 90)}}
```

For a list of records with a stable id, pass `id_key` — otherwise
inserting one new item shifts every later index and makes everything
after it look changed even though it didn't:

```python
diff_pages(old_page, new_page, id_key="listing_id")
# {"changed": {"items[listing_id=7165546].price": (929900, 899900)}, ...}
```

Or from the command line, polling continuously (`--rsc` for the
lighter-weight RSC payload instead of full HTML each poll):

```bash
nextflight https://example.com/product/123 --watch 60
nextflight https://example.com/car/search --rsc --watch 60
```

### Exporting to a DataFrame or CSV

```python
page = extract(html_text)

df = page.to_dataframe(required_keys={"id", "price"})       # requires pandas
page.to_csv("listings.csv", required_keys={"id", "price"})  # works either way
```

### Command line

```bash
nextflight page.html --keys sections,meta
nextflight https://example.com/product/123 --type Product
nextflight page.html --tree                 # shape summary, no full values
nextflight page.html --all > everything.json
```

## API reference

### `extract(html) -> FlightExtractor`

Shorthand constructor. `html` accepts a plain string, bytes, or a
response-like object (Scrapy's `Response`, `requests.Response`, etc.) —
pass `response` straight from a `parse()` method.

### `FlightExtractor(html, *, strict=False)`

`strict=True` raises `FlightParseError` on a row that's neither valid
JSON nor a recognizable `$`-reference, instead of keeping it as a raw
string. Useful while developing a new scraper; leave off in production so
a handful of odd rows never take down extraction of everything else.

**Exploring a page**

| Method | Returns | What it does |
|---|---|---|
| `.keys()` | `list[str]` | Every chunk id on the page, in order |
| `.kind(chunk_id)` | `str \| None` | Row kind: `"json"`, `"text"`, `"module"`, `"preload"` |
| `.json_keys()` | `list[str]` | Chunk ids holding structured JSON (dict/list) |
| `.html_keys()` | `list[str]` | Text-row chunk ids that look like HTML fragments |
| `.text_keys()` | `list[str]` | All text-row chunk ids, HTML-looking or not |
| `.shape(chunk_id=None, max_depth=3)` | structure summary | Key names + value types, not values — get a feel for a new site fast |
| `.stats()` | `dict` | Chunk count, row-kind breakdown, page size |

**Resolving data** (dereferencing `$`-refs)

| Method | Returns | What it does |
|---|---|---|
| `page["id"]` / `.resolve_chunk("id")` | resolved value | One chunk, fully dereferenced (`page[...]` raises `KeyError` if missing) |
| `.resolve_all()` | `dict` | Every chunk, fully dereferenced |
| `.resolve_json()` / `.resolve_html()` / `.resolve_text()` | `dict` | Only one kind of chunk — cheaper than `resolve_all()` when you don't need everything |
| `.iter_resolved()` | iterator | Like `resolve_all()` but lazy, one chunk at a time |
| `.get("path.to.value", default=None)` | value | Tolerant dotted-path lookup (dict keys, list indices, and React element `"props"`) |
| `.select(*paths, default=None)` | `dict` | Resolve just the named paths, e.g. `page.select("3f.props.price", "3f.props.title")` |

`"3f" in page` and `for k in page` also work, like a dict.

**Searching** (schema-free, works across redeploys)

| Method | Returns | What it does |
|---|---|---|
| `.find_by_keys(required_keys, root=None)` | dict or `None` | First dict containing *all* of `required_keys` |
| `.find_all_by_keys(required_keys, root=None)` | `list` | Every matching dict — for repeated cards/listings |
| `.find_any_keys(any_keys, root=None)` | `list` | Every dict containing *any* of `any_keys` |
| `.find_by_key_pattern(pattern, root=None)` | `list` | Every dict with a key matching a regex, e.g. `r"^price_"` |
| `.find_by_type(type_value, key="@type", root=None)` | `list` | Every dict whose `key` field equals `type_value` |
| `.find_text(pattern, root=None)` | `list` | Distinct string values matching a regex (emails, SKUs, ...) |
| `.find_all(predicate, root=None, max_results=None)` | `list` | Fully custom predicate over every node |
| `.find_one(predicate, root=None)` | value or `None` | Like `find_all` but just the first match |

Pass `include_source=True` on any `find_*` method to get `(node,
chunk_id)` tuples instead of bare nodes, so you can trace a match back to
where it came from. `find_all`/`find_one`/`find_by_keys` resolve chunks
lazily and stop the moment `max_results` is hit — they don't pay to
resolve chunks after a match is already found.

**Fetching**

| Classmethod | What it does |
|---|---|
| `.from_url(url, timeout=15.0, headers=None)` | Fetch and parse a URL, stdlib only |
| `.from_url_async(url, ...)` | Async version for `asyncio.gather(...)` crawls — requires `httpx` |
| `.from_rsc_url(url, headers=None, cookies=None, auto_discover=True)` | Fetch the raw RSC payload instead of full HTML — see "Raw RSC fetches" above |

`from_url`/`from_rsc_url` transparently decompress gzip/deflate/br
responses even if the server ignores the default `Accept-Encoding:
identity` request.

**Diffing and exporting**

| Method | Returns | What it does |
|---|---|---|
| `.diff(other_page, id_key=None)` | `dict` | Compare against another crawl — see "Monitoring a page over time" |
| `.to_json(path=None, indent=2)` | `str \| None` | Dump the fully resolved page to a file, or return as a string |
| `.to_dataframe(records=None, required_keys=None)` | `DataFrame` | Requires pandas |
| `.to_csv(path, records=None, required_keys=None)` | — | Falls back to the stdlib `csv` module without pandas |

### Module-level functions

- **`find_json_ld(html, type_=None) -> list`** — parse
  `<script type="application/ld+json">` blocks, optionally filtered by
  `@type`. Often more stable across redesigns than Flight data — worth
  trying first for product/article/breadcrumb structured data.
- **`find_next_data(html) -> dict | None`** — parse a Pages Router
  `__NEXT_DATA__` blob. `None` if the page doesn't have one.
- **`detect_next_router(html) -> str`** — `"app"`, `"pages"`, `"both"`, or
  `"unknown"`. Run this first if you're not sure which extractor to use.
- **`diff_pages(old, new, id_key=None) -> dict`** — module-level form of
  `.diff()`.

### Command-line reference

```
nextflight <file-or-url>
  [--keys a,b | --all-by-keys a,b | --any-keys a,b
   | --type Product | --text PATTERN | --get path.to.value
   | --json-keys | --html-keys | --tree
   | --router | --next-data | --stats | --watch SECONDS | --all]
  [--rsc] [--redact] [--save out.json]
```

- `--tree` prints a `.shape()` summary instead of full values.
- `--router` / `--next-data` cover Pages Router pages.
- `--rsc` fetches the raw RSC payload instead of full HTML (URL sources
  only) — lighter weight, also works with `--watch`.
- `--watch SECONDS` polls a URL and prints only what changed since the
  last poll.
- `--redact` best-effort scrubs email/phone-shaped strings from output,
  for sharing debug dumps.

## Optional dependencies

Nothing below is required to install or use `nextflight` — each is used
automatically if already present in your environment, and raises a clear
`ImportError` only if you call the one method that needs it.

| Package | Unlocks | Install |
|---|---|---|
| `orjson` | Faster JSON decoding everywhere | `pip install nextflight[fast]` |
| `pandas` | `.to_dataframe()`, nicer `.to_csv()` | `pip install nextflight[pandas]` |
| `httpx` | `.from_url_async()` | `pip install nextflight[async]` |

Or `pip install nextflight[all]` for all three.

## How it works

Flight payloads aren't newline-delimited JSON — text rows
(`id:T<hexByteLen>,<raw bytes>`) are byte-length-prefixed and can contain
literal newlines or run straight into the next row with no separator, and
module/preload rows (`id:I[...]`, `id:HL[...]`) need bracket-aware
parsing. `nextflight` implements the actual row grammar rather than
splitting on `\n`, so it holds up on both well-formed pages and payloads
truncated mid-chunk (e.g. by a proxy that cuts a response short).

It also doesn't assume one `<script>self.__next_f.push(...)</script>`
call is one complete, self-contained set of rows. On large real-world
pages, Next.js's own streaming buffer can flush mid-string, splitting a
single row's raw text across two or more separate `push()` calls with no
separator between the pieces — all `push()` payloads are reassembled into
one continuous stream before being split into rows, so this doesn't
silently corrupt chunk ids on pages large enough to trigger it (confirmed
against production pages where over half of all `push()` calls turned out
to be mid-row continuations).

Chunk ids aren't guaranteed unique, either — Next.js deliberately emits
preload (`HL`) rows with a completely empty id (`:HL["/path.css","style"]`)
since nothing ever needs to `$`-ref them individually, and real pages have
had dozens of these sharing the same empty id. Rather than the later ones
silently overwriting the earlier ones, only the first occurrence of a
duplicated id keeps its real id; later ones get a synthesized, clearly
distinguishable key (`"id#2"`, `"id#3"`, ...) so nothing gets lost.

Parsing and resolution are both designed to scale roughly linearly with
page size: rows are split cheaply up front, each chunk's JSON is decoded
lazily on first access rather than all at once, and searches
(`find_one`/`find_by_keys`) stop resolving chunks the moment a match is
found instead of resolving the whole page first.

## License

MIT
