Metadata-Version: 2.4
Name: nextflight
Version: 0.3.3
Summary: Parse Next.js App Router 'Flight' (__next_f.push) payloads embedded in server-rendered HTML -- works on any Next.js 13+ site.
Author: Aly Reda
License: MIT
Project-URL: Homepage, https://github.com/Aly-Reda/nextflight
Project-URL: Issues, https://github.com/Aly-Reda/nextflight/issues
Keywords: nextjs,scraping,scrapy,flight,rsc,react-server-components,web-scraping,zyte
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: fast
Requires-Dist: orjson; extra == "fast"
Provides-Extra: pandas
Requires-Dist: pandas; extra == "pandas"
Provides-Extra: async
Requires-Dist: httpx; extra == "async"
Provides-Extra: all
Requires-Dist: orjson; extra == "all"
Requires-Dist: pandas; extra == "all"
Requires-Dist: httpx; extra == "all"
Dynamic: license-file

# nextflight

A general-purpose parser for the data Next.js (App Router) embeds in
`<script>self.__next_f.push([...])</script>` tags — the React Server
Components "Flight" wire format. Works on **any** Next.js 13+ App Router
site, not just one particular project.

Instead of hardcoding array indices like `data[3]["children"][0][3]...`,
which break the moment a site's component tree reshuffles on redeploy,
`nextflight` resolves the `$`-sigil references Next.js uses internally
and lets you *search* for the shape of data you want.

## Install

```bash
pip install nextflight
```

## Quick start

The core workflow is two steps: hand it any HTML, see what keys are on
the page, then fetch the resolved JSON for whichever key you want.

```python
from nextflight import extract

# Step 1: send any HTML, get the list of keys (one per __next_f.push chunk)
page = extract(html_text)
print(page.keys())          # e.g. ['0', '1', '3f', '20', ...]

# Step 2: fetch the resolved JSON for a specific key
data = page["3f"]           # same as page.resolve_chunk("3f")
```

Chunk ids are arbitrary per build though (a redeploy can renumber them),
so in practice you'll usually skip straight to *searching* for the shape
of data you want instead of a specific id:

```python
from nextflight import extract

page = extract(html_text)

# Find the first object anywhere in the page that has all of these keys,
# wherever this build's component tree happened to put it:
listing = page.find_by_keys({"sections", "meta"})

# Find every node with a given @type (or any custom key):
products = page.find_by_type("Product")

# Or search with a fully custom predicate:
items = page.find_all(lambda n: isinstance(n, dict) and "price" in n)

# Or grab everything, fully dereferenced, and inspect by hand:
everything = page.resolve_all()
```

### Command line

For quick, no-script exploration of a page you've already saved (or a live URL):

```bash
nextflight page.html --keys sections,meta
nextflight https://example.com/product/123 --type Product
nextflight page.html --all > everything.json
```

### In a Scrapy / Zyte spider

```python
import scrapy
from nextflight import extract

class MySpider(scrapy.Spider):
    name = "my_spider"

    def parse(self, response):
        page = extract(response.text)

        items = page.find_all(
            lambda n: isinstance(n, dict) and "price" in n and "title" in n
        )
        for item in items:
            yield {
                "title": item.get("title"),
                "price": item.get("price"),
                "url": response.url,
            }
```

### Fetching a URL directly (no Scrapy needed)

```python
from nextflight import FlightExtractor

page = FlightExtractor.from_url("https://example.com/product/123")
product = page.find_by_keys({"price", "title"})
```

(`from_url` uses only the stdlib for quick one-off exploration. For
production crawling — retries, proxies, JS rendering, robots.txt — fetch
the page with your own HTTP client / Scrapy / Zyte and pass
`response.text` to `FlightExtractor(...)` / `extract(...)` instead.)

### Raw RSC fetches (no HTML at all)

Sending a request with an `RSC: 1` header — the way Next.js's own
client-side navigation does it — gets back the raw Flight row stream
directly as the response body, with no surrounding HTML and no
`self.__next_f.push(...)` wrapper. `extract()` detects and parses this
automatically, exactly the same as the HTML-embedded form:

```python
from nextflight import FlightExtractor

# Convenience constructor: sets the RSC header for you
page = FlightExtractor.from_rsc_url("https://example.com/car/search?page=2")

# Or bring your own client (requests, httpx, Scrapy, ...):
import requests
resp = requests.get(
    "https://example.com/car/search",
    params={"page": "2", "_rsc": "1p28d"},  # build-specific cache key from the page's own JS
    headers={"RSC": "1", "Next-Url": "/en/car/search"},
)
page = FlightExtractor(resp.text)
```

Some deployments require extra headers to serve the RSC payload instead
of redirecting to the full HTML page or rejecting the request — a
matching `Next-Router-State-Tree` header, a `next-url` header pointing at
the page itself, or a build-specific `_rsc=<id>` query parameter (grab it
from the page's own client-side JS/network tab; it changes across
deploys). Copy whatever a real browser sends for that specific site if
the bare `RSC: 1` header alone doesn't work.

### Pages Router support

Not every Next.js site uses the App Router. Older / mixed deployments
often use the Pages Router's `__NEXT_DATA__` blob instead, which is
already plain JSON:

```python
from nextflight import extract, find_next_data, detect_next_router

router = detect_next_router(html_text)  # "app" | "pages" | "both" | "unknown"

if router == "app":
    page = extract(html_text)
    data = page.find_by_keys({"price", "title"})
else:
    data = find_next_data(html_text)["props"]["pageProps"]
```

### Monitoring a page over time

`diff_pages` compares two crawls of the same URL and reports what
changed, by dotted path — handy for a price/stock watcher:

```python
from nextflight import FlightExtractor, diff_pages

old_page = FlightExtractor.from_url(url)
# ...re-fetch later...
new_page = FlightExtractor.from_url(url)

changes = diff_pages(old_page, new_page)
# {"added": {...}, "removed": {...}, "changed": {"path.to.price": (100, 90)}}
```

Or from the command line, polling continuously:

```
nextflight https://example.com/product/123 --watch 60
```

### Exporting to a DataFrame / CSV

```python
page = extract(html_text)

df = page.to_dataframe(required_keys={"id", "price"})   # requires pandas
page.to_csv("listings.csv", required_keys={"id", "price"})  # works either way
```

## API

- **`extract(html) -> FlightExtractor`** — shorthand constructor. `html`
  accepts a plain string, bytes, or a response-like object (Scrapy's
  `Response`, `requests.Response`, etc.) — pass `response` straight from a
  `parse()` method without writing `response.text` yourself.
- **`FlightExtractor(html, *, strict: bool = False)`**
  - `.keys() -> list[str]` — every chunk id found on the page, in order.
  - `.json_keys() -> list[str]` — chunk ids whose raw value is structured
    JSON (a dict or list), i.e. the ones you almost always want.
  - `.html_keys() -> list[str]` — chunk ids from Flight text (`T`) rows
    that look like an HTML fragment (contain a tag) — suspense fallbacks,
    error boundaries, inlined SVGs, and other raw markup Next.js streams
    outside the JSON chunks.
  - `.text_keys() -> list[str]` — every chunk id from a text (`T`) row,
    HTML-looking or not (plain copy, translated strings, etc).
  - `.kind(chunk_id) -> str | None` — which Flight row kind a chunk came
    from (`"json"`, `"text"`, `"module"`, `"preload"`), or `None` if the id
    doesn't exist. This is what `json_keys()` / `html_keys()` / `text_keys()`
    filter on.
  - `.find_any_keys(any_keys, root=None, include_source=False) -> list` —
    like `find_all_by_keys` but matches a dict containing ANY of `any_keys`
    rather than requiring all of them.
  - `.find_by_key_pattern(pattern, root=None, include_source=False) -> list`
    — find every dict with at least one key matching a regex `pattern`
    (e.g. `r"^price_"` to catch `price_usd`, `price_aed`, ...).
  - `include_source=True` on any `find_*` method returns `(node, chunk_id)`
    tuples instead of bare nodes, so you can trace a match back to roughly
    where it came from (or re-fetch just that chunk on a future crawl).
  - `.iter_resolved() -> Iterator[(chunk_id, value)]` — like `resolve_all()`
    but lazy, one chunk at a time, for bailing out early on very large pages.
  - `.shape(chunk_id=None, max_depth=3) -> Any` — a compact summary of the
    resolved data's *structure* (key names + value types, lists collapsed to
    their first element) instead of full values — for getting a feel for an
    unfamiliar site fast. Omit `chunk_id` to summarize every chunk.
  - `.diff(other_page) -> dict` / `diff_pages(old, new) -> dict` — compare
    two crawls of the same URL and report `{"added", "removed", "changed"}`
    by dotted path — handy for price/stock-monitoring pipelines. See the
    caveat about chunk ids reshuffling across redeploys in the docstring.
  - `.to_dataframe(records=None, required_keys=None)` / `.to_csv(path, ...)`
    — build a pandas DataFrame or write a CSV from a list of dict records
    (or run `find_all_by_keys(required_keys)` for you first). `to_csv` falls
    back to the stdlib `csv` module if pandas isn't installed.
  - `.from_url_async(url, ...)` (async classmethod) — async counterpart to
    `from_url`, for concurrent multi-page crawls with `asyncio.gather(...)`.
    Requires `httpx` (optional).
  - `.from_rsc_url(url, headers=None, cookies=None, ...)` — fetch a raw RSC
    payload directly (sets the `RSC: 1` header for you) instead of the full
    HTML page. See "Raw RSC fetches" above.
  - `page["3f"]` / `.resolve_chunk("3f")` — the resolved JSON for one
    specific chunk id (`page[...]` raises `KeyError` if it doesn't exist;
    `resolve_chunk` returns `None`). `"3f" in page` and `for k in page`
    also work, like a dict.
  - `.resolve_all() -> dict` — every chunk, fully dereferenced.
  - `.resolve_json() -> dict` — only the chunks whose raw value is
    structured JSON (see `json_keys()`). Skips top-level text/HTML-only
    chunks that aren't referenced from any JSON chunk, so it's cheaper
    than `resolve_all()` on pages with a lot of raw markup/text rows.
  - `.resolve_html() -> dict` / `.resolve_text() -> dict` — the mirror
    image: resolve only `html_keys()` / `text_keys()`.
  - `.find_all(predicate, root=None, max_results=None) -> list` — walk the
    resolved tree and collect every node matching `predicate`. With no
    `root`, chunks are resolved lazily one at a time (via
    `iter_resolved()`), so `max_results` stops *resolving* further chunks
    the moment enough matches are found, not just stops searching.
  - `.find_one(predicate, root=None) -> Any | None`
  - `.find_by_keys(required_keys, root=None) -> dict | None` — find the
    first dict containing all of `required_keys`.
  - `.find_all_by_keys(required_keys, root=None) -> list` — like
    `find_by_keys` but returns every match, for pages with repeated
    cards/listings that share the same shape.
  - `.find_by_type(type_value, key="@type", root=None) -> list` — find
    every dict whose `key` field equals `type_value`.
  - `.find_text(pattern, root=None) -> list` — regex-search every string
    value on the page and return the distinct whole values that contain a
    match (emails, prices, phone numbers, SKUs, ...) without needing to
    know which object they live on.
  - `.get("path.to.value", default=None) -> Any` — tolerant dotted-path
    lookup into the resolved page (dict keys and/or list indices), once
    you already know roughly where something lives on this site.
  - `.stats() -> dict` — quick diagnostic snapshot (chunk count, ids,
    value type counts, Flight row kind counts, json/html chunk counts,
    page size) for exploring a new site.
  - `.to_json(path=None, indent=2) -> str | None` — dump the fully
    resolved page to a file, or return it as a JSON string.
  - `.from_url(url, timeout=15.0, headers=None) -> FlightExtractor`
    (classmethod) — fetch and parse a URL using only the stdlib.
  - `strict=True` raises `FlightParseError` on a row that's neither valid
    JSON nor a recognizable `$`-reference marker, instead of silently
    keeping it as a raw string (useful while developing a new scraper;
    leave off in production so a handful of odd rows never take down
    extraction of everything else on the page).
- **`find_json_ld(html, type_=None) -> list`** — parse any
  `<script type="application/ld+json">` blocks on the page, optionally
  filtered by `@type`. Also accepts response-like objects.
- **`find_next_data(html) -> dict | None`** — parse the **Pages Router**'s
  `__NEXT_DATA__` JSON blob (Next.js's pre-App-Router data mechanism).
  Already plain JSON, no `$`-ref resolution needed, so there's no
  extractor class for it, just this function. Returns `None` if the page
  doesn't have one (e.g. it's an App Router page — use `extract()` there
  instead).
- **`detect_next_router(html) -> str`** — best-effort guess at which
  router rendered a page: `"app"`, `"pages"`, `"both"` (rare, e.g.
  mid-migration sites), or `"unknown"`. Run this first if you're not sure
  which of `extract()` / `find_next_data()` to reach for.
- **`diff_pages(old, new) -> dict`** — module-level version of `.diff()`,
  same thing.
- **CLI**: `nextflight <file-or-url> [--keys a,b | --all-by-keys a,b | --any-keys a,b | --type Product | --text PATTERN | --get path.to.value | --json-keys | --html-keys | --tree | --router | --next-data | --stats | --watch SECONDS | --all] [--redact] [--save out.json]`
  - `--tree` prints `.shape()` instead of full values.
  - `--router` / `--next-data` cover Pages Router pages (see above).
  - `--watch SECONDS` polls a URL and prints only what changed since the
    last poll (via `diff_pages`) — a quick way to eyeball whether a site's
    data-monitoring pipeline is worth building before you build it.
  - `--redact` best-effort scrubs email/phone-shaped strings from output,
    for sharing debug dumps.

No *required* runtime dependencies — stdlib only (`json`, `re`, `urllib`,
`argparse`) — so it's safe to drop into any existing Scrapy/Zyte project
without touching the rest of your dependency tree. Optional accelerators/
integrations, used automatically if already installed, otherwise skipped
or raising a clear `ImportError` only if you call the specific method that
needs them:

| Package  | Used for                                    | Install                       |
|----------|----------------------------------------------|-------------------------------|
| `orjson` | faster JSON decoding everywhere               | `pip install nextflight[fast]`   |
| `pandas` | `.to_dataframe()`, nicer `.to_csv()`          | `pip install nextflight[pandas]` |
| `httpx`  | `.from_url_async()`                           | `pip install nextflight[async]`  |

(or `pip install nextflight[all]` for all three.)

### Upgrading from `nextjs-flight-extractor` / `NextFlightExtractor`

The old names still work but emit a `DeprecationWarning`:

| Old (0.1.x)                          | New (0.2.x+)                     |
|---------------------------------------|----------------------------------|
| `from nextjs_flight_extractor import NextFlightExtractor` | `from nextflight import FlightExtractor` |
| `extractor.find_first(...)`           | `page.find_one(...)`            |
| `extract_json_ld(html, schema_type=…)`| `find_json_ld(html, type_=…)`   |

## Performance

Parsing is designed to be roughly linear in the size of the page, even on
pages with hundreds or thousands of Flight rows:

- **Text (`T`) rows no longer re-encode the rest of the payload on every
  row.** Earlier versions read a text row's body with
  `payload[body_start:].encode("utf-8")[:hex_len]`, which re-encodes the
  *entire remainder* of the payload for every single text row. A page with
  many text rows (translated copy, repeated card fragments, inlined SVGs)
  made parsing effectively O(n²). It now reads only the bytes each row
  actually needs, so parsing is O(n) again — roughly a 10x–20x speedup on
  pages with thousands of text rows, and the gap grows with page size.
- **No more O(n) string copies for bare/ref rows.** Locating the end of an
  unbracketed value (a bare `$`-ref marker, number, etc.) used to slice
  `payload[i:]` — copying the rest of the payload — before searching it.
  It now searches in place with a `pos` argument instead.
- **Optional `orjson` acceleration.** If `orjson` is already installed in
  your environment (common in scraping stacks), `nextflight` will use it
  automatically for JSON decoding — no configuration needed, and it's not
  a required dependency. Falls back to the stdlib `json` module otherwise.
- **`find_one`/`find_by_keys` stop *resolving*, not just searching, at the
  first match.** These used to build `resolve_all()` — resolving every
  chunk on the page — before the search even started, so `max_results=1`
  only cut short the walk, not the resolution work leading up to it. They
  now resolve chunks lazily (via `iter_resolved()`) and stop the moment a
  match is found. On a page with thousands of unrelated chunks and the
  match near the front, this measured 300x+ faster in testing (0.056s →
  0.0002s on a 3,000-chunk synthetic page) — the gap scales with how many
  chunks come *after* the match and how expensive they are to resolve.
- **`resolve_json()` / `resolve_html()` / `resolve_text()`** for when you
  only care about one kind of chunk and don't want to pay to resolve (and
  allocate copies of) everything else on the page via `resolve_all()`.

## Why not just `str.split('\n')`?

Two of the Flight row kinds break that assumption:

- **Text rows** (`id:T<hexByteLen>,<raw text>`) are byte-length-prefixed
  blobs, not newline-terminated, and can contain literal newlines or run
  directly into the next row's id with zero separator.
- **Module / preload rows** (`id:I[...]` / `:HL[...]`) need bracket-aware
  parsing.

`nextflight` implements the real row grammar, quote/escape aware, so it
holds up on both well-formed and truncated payloads (e.g. from a proxy
that cuts a response off mid-chunk).


## License

MIT
