Metadata-Version: 2.5
Name: pydj
Version: 0.1.1
Summary: Dynamic JS File Extractor - discover dynamically loaded JavaScript and source maps from websites
Project-URL: Homepage, https://github.com/ejfkdev/pydj
Project-URL: Repository, https://github.com/ejfkdev/pydj
Project-URL: Issues, https://github.com/ejfkdev/pydj/issues
Project-URL: Changelog, https://github.com/ejfkdev/pydj/releases
Project-URL: Original Go implementation, https://github.com/ejfkdev/dj
Author: pydj contributors
License-Expression: MPL-2.0
License-File: LICENSE
Keywords: cli,extractor,javascript,nextjs,recon,source-map,vite,webpack
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Mozilla Public License 2.0 (MPL 2.0)
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Security
Classifier: Topic :: Software Development :: Debuggers
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: curl-cffi>=0.7
Requires-Dist: httpx[http2]>=0.27
Provides-Extra: dev
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Provides-Extra: tls
Description-Content-Type: text/markdown

# pydj - Dynamic JS File Extractor (Python)

[中文](./README.zh.md) | English

[![Python](https://img.shields.io/badge/Python-3.10%2B-3776AB.svg?style=flat-square&logo=python&logoColor=white)](https://www.python.org/)
[![License](https://img.shields.io/badge/License-MPL%202.0-blue.svg?style=flat-square)](./LICENSE)
[![PyPI](https://img.shields.io/badge/PyPI-pydj-3775A9.svg?style=flat-square&logo=pypi&logoColor=white)](https://pypi.org/project/pydj/)
[![pipx](https://img.shields.io/badge/pipx-ready-2C3E50.svg?style=flat-square)](https://pipx.pypa.io/)
[![uvx](https://img.shields.io/badge/uvx-ready-DE5FE9.svg?style=flat-square)](https://docs.astral.sh/uv/)
[![Port of](https://img.shields.io/badge/port%20of-ejfkdev%2Fdj-00ADD8.svg?style=flat-square)](https://github.com/ejfkdev/dj)

`pydj` statically analyses a website's HTML and JavaScript to find the JS files
that are loaded **dynamically** — webpack chunks, `import()` lazy loading, Vite
preloads, Module Federation remotes, Next.js Flight chunks, microfrontend
sub-applications — then locates the source maps and **restores the original
source code**.

A Python port of the Go project [`ejfkdev/dj`](https://github.com/ejfkdev/dj),
keeping the same CLI surface and output formats.

> **Scope:** CLI and library only. There is no HTTP API and no MCP server.

## Why this is useful

A plain `curl` of a page gets you the entry bundle and nothing else. The other
300 chunk files exist only as strings assembled at runtime by the bundler's chunk
loader. `pydj` recognises those loaders and reconstructs the URLs — which is what
makes the whole application graph, and its source maps, reachable.

```console
$ pydj https://example.com
## Summary
- **JS files**: 74
- **Source maps**: 12 (found)
- **Restored sources**: 1204 files (restored)
```

## Installation

```bash
pipx install pydj              # isolated CLI install (recommended)
uvx pydj --help                # run without installing
pip install pydj               # library use
```

This pulls in [`curl-cffi`](https://github.com/lexiforest/curl_cffi) for browser
TLS impersonation, so there is no extra to remember. Prebuilt `abi3` wheels cover
macOS, Linux (glibc and musl) and Windows, so no compiler is normally needed. If
it cannot be installed, pydj still runs with a plain fingerprint — see
[TLS fingerprints](#tls-fingerprints).

<details>
<summary>Install from source</summary>

```bash
git clone https://github.com/ejfkdev/pydj && cd pydj
pip install -e '.[dev]'
```
</details>

## CLI usage

```bash
pydj <url>                    # scan a website (same as: pydj scan <url>)
pydj scan <url>               # canonical subcommand form
pydj <url> -f json            # structured output
pydj <url> -f text            # bare URL list
```

### Options

| Option | Description |
|--------|-------------|
| `-v, --version` | Print version and exit |
| `-d, --debug` | Enable debug output |
| `-f, --format <fmt>` | Output format: `md` (default), `json`, `text` (bare URL list) |
| `--json` | Global flag: emit raw JSON instead of the rendered report |
| `--no-cache` | Disable cache reads (downloads are still saved to disk) |
| `--cache[=bool]` | Legacy spelling: `--cache`, `--cache=false`, `--cache=yes`, `--cache no` |
| `--useragent <UA>` | Custom User-Agent (non-ASCII supported) |
| `--ua <UA>` | Short alias for `--useragent` |
| `-x, --proxy <URL>` | Proxy URL: `http://`, `https://`, `socks5://` |
| `--cookie <cookies>` | Cookies to get past Cloudflare, e.g. `"cf_clearance=xxx"` |
| `-H, --header <K: V>` | Custom header, repeatable; later values override earlier ones |
| `--no-random-tls` | Pin the TLS fingerprint to Chrome instead of randomising |
| `-o, --output <dir>` | Also write artefacts here, without the per-site subdirectory |
| `--no-js-cache` | Do not persist downloaded JS bodies (URL list and sources unaffected) |
| `--js-cache-dir <dir>` | Write JS bodies here instead of the cache's `js/`, flat and shareable |
| `-t, --timeout <secs>` | Per-request timeout (default: 30) |
| `-c, --concurrency <N>` | Max concurrent HTTP requests (default: 8) |
| `-h, --help` | Show help |

Exit codes are `0` for success and `1` for usage or runtime errors.

### Examples

```bash
# Basic scan
pydj https://example.com

# JSON output, piped into jq
pydj -f json https://example.com | jq -r '.jsURLs[]'

# Fresh scan (ignore the cache), export artefacts, use a SOCKS5 proxy
pydj --no-cache -o ./output -x socks5://127.0.0.1:1080 -t 60 https://example.com

# Bypass a Cloudflare challenge
pydj --cookie "cf_clearance=xxx" --useragent "Mozilla/5.0 ..." https://example.com

# Custom headers (later value wins for duplicate keys)
pydj -H 'Referer: https://google.com' -H 'X-Token: abc' https://example.com

# Keep JS bodies off disk; export only the URL list and restored sources
pydj --no-js-cache -f json https://example.com

# Proxy from the environment, with a bypass list
HTTPS_PROXY=http://127.0.0.1:7890 NO_PROXY=localhost pydj https://example.com
```

## Library usage

`scan` is a facade over a set of composable pieces. Use it for the whole job, or
drop to the pieces when you want to control part of it.

### The one-call version

```python
from pydj import scan

result = scan("https://example.com")

print(result.summary.js_count)      # 74
print(result.summary.source_count)  # 1204 restored source files
print(result.cache_dirs.sources)    # /tmp/ejfkdev/dj/https_example.com/sources

for url in result.js_urls:          # ScanResult iterates over JS URLs
    print(url)

# Structured data
data = result.to_dict()
json_text = result.to_json()

# Rendered report, matching the CLI byte for byte
print(scan("https://example.com", format="md").text)
```

`ScanResult` carries both the structured `OutputResult` (`result.result`) and the
rendered report (`result.text`).

### `scan()` parameters

| Parameter | Default | Description |
|-----------|---------|-------------|
| `url` | required | Target site; must include a scheme |
| `format` | `"text"` | `text`, `json` or `md` |
| `transport` | `None` | Your own HTTP layer — see [below](#bringing-your-own-http-layer) |
| `concurrency` | `8` | Max concurrent HTTP requests (shared by downloads, probes, HEAD and RSC fetches) |
| `timeout` | `30.0` | Per-request timeout in seconds |
| `proxy` | `""` | Proxy URL, overrides environment proxies |
| `user_agent` | `""` | Custom User-Agent |
| `cookie` | `""` | Cookie header string |
| `headers` | `None` | `dict` or a list of `"Key: Value"` strings |
| `no_random_tls` | `False` | Pin TLS fingerprint to Chrome |
| `cache` | `True` | Read from cache (`False` still writes) |
| `cache_js` | `True` | Write downloaded JS bodies to disk — see [JS on disk](#js-on-disk) |
| `js_cache_dir` | `""` | Write JS bodies here instead of the cache's `js/` |
| `output_dir` | `""` | Extra flat output directory |
| `cache_dir` | `""` | Override the cache root |
| `debug` | `False` | Emit debug logging |
| `registry` | `None` | Custom plugin registry |

> [!NOTE]
> The fetch-related parameters (`concurrency`, `timeout`, `proxy`, `user_agent`,
> `cookie`, `headers`, `no_random_tls`) configure *pydj's own* HTTP client.
> Passing a `transport` replaces that client, so those parameters are then
> ignored — apply them inside your transport instead.

### Bringing your own HTTP layer

pydj never touches the network directly. Every request — the landing page, JS
files, source maps, `.map` probes, Next.js RSC payloads — goes through a
*transport*: any object with two methods.

```python
class HttpTransport(Protocol):
    def get(self, url, *, headers=None) -> HttpResponse: ...
    def head(self, url, *, headers=None) -> HttpResponse: ...
```

That is the entire contract, and it is what makes TLS handling your choice rather
than pydj's. Implement it to plug in a different TLS library, an existing
authenticated session, a caching proxy, or a request budget you own:

```python
import curl_cffi
from pydj import scan


class ImpersonatingTransport:
    """curl-cffi with a fingerprint pydj does not ship."""

    def __init__(self, impersonate: str = "chrome124") -> None:
        self._session = curl_cffi.requests.Session(impersonate=impersonate)

    def get(self, url, *, headers=None):
        return self._session.get(url, headers=dict(headers or {}), timeout=30)

    def head(self, url, *, headers=None):
        return self._session.head(url, headers=dict(headers or {}), timeout=30)


result = scan("https://example.com", transport=ImpersonatingTransport("safari17_0"))
print(result.js_urls)
```

**Your return value does not have to be an `HttpResponse`.** pydj coerces
whatever comes back, so `requests.Response`, `httpx.Response` and curl-cffi
responses work as-is — anything with `status_code`, `content`, `headers` and
`url`:

```python
import requests
from pydj import scan


class SessionTransport:
    def __init__(self):
        self.session = requests.Session()
        self.session.headers["User-Agent"] = "MyBot/1.0"

    def get(self, url, *, headers=None):
        return self.session.get(url, headers=headers or {}, timeout=30)

    def head(self, url, *, headers=None):
        return self.session.head(url, headers=headers or {}, timeout=30)


result = scan("https://example.com", transport=SessionTransport())
```

| Behaviour | Detail |
|-----------|--------|
| Required methods | Only `get`. Without `head`, pydj issues a GET when checking for a `.map` — it still works, it just transfers a little more |
| Lifecycle | pydj **never closes your transport**; a transport pydj built itself *is* closed |
| Configuration | None injected — no User-Agent, cookie or proxy reaches inside your client |
| Bad transports | A `TypeError` is raised up front, not deep inside a scan |

### The fine-grained API

Everything `scan` does is also available as a standalone function. None of them
hold state, read configuration or write to disk, so you can assemble your own
pipeline:

```python
from pydj import fetch_content, analyze_content, fetch_source_map, restore_sources

# 1. Fetch a page.
page = fetch_content(transport, "https://example.com/")
if page is None:
    raise SystemExit("fetch failed")

# 2. Analyse it. No network, no disk — just plugins over bytes.
for result in analyze_content("https://example.com/", page.content, "html",
                              headers=page.headers):
    for found in result.urls:
        print(result.from_plugin, "found", found.url)

# 3. Fetch a JS file and find what it loads.
app = fetch_content(transport, "https://example.com/js/app.js")
for result in analyze_content("https://example.com/js/app.js", app.content, "js"):
    print(result.urls)

# 4. Recover original sources from its map.
map_bytes = fetch_source_map(transport, "https://example.com/js/app.js")
for source in restore_sources(map_bytes, minified_content=app.content):
    print(source.path, source.mode)
```

| Function | What it does |
|----------|--------------|
| `fetch_content(transport, url, headers=None, retries=3, require_2xx=True)` | GET with retry; returns `HttpResponse` or `None` |
| `probe_content(transport, url, headers=None)` | HEAD probe (falls back to GET) |
| `analyze_content(source_url, content, content_type, headers=None, registry=None, plugins=None)` | Run plugins over a payload; returns `list[Result]` |
| `discover_js_urls(transport, url, registry=None, concurrency=8, headers=None)` | Full crawl, **no disk and no source maps** |
| `fetch_source_map(transport, js_url, head_probe=True)` | Download and validate a `.map` |
| `restore_sources(map_content, minified_content=None)` | Map bytes to `list[SourceFile]` |
| `probe_fragment(fragment, source_url, knowledge=None, base_url="")` | Expand a chunk path into candidate URLs |
| `detect_content_type(header, content)` | Sniff `html` / `js` / `json` |
| `decode_content(text)` | Un-escape JS, URL, Unicode and HTML-entity encodings |
| `is_source_map_url(url)` / `is_likely_static_resource(url)` | URL predicates |

`discover_js_urls` is the interesting one: the whole crawl with none of the side
effects, so `summary.source_map_count` and `summary.source_count` stay zero and
nothing is written anywhere.

### JS on disk

JS bodies dominate the disk usage of a scan and are usually the least useful
artefact afterwards — you wanted the URL list and the sources. Three ways to
control that:

```python
# Default: bodies land in the cache's js/ directory.
scan("https://example.com")

# Keep them off disk entirely. The URL list, source maps and restored
# sources are unaffected.
scan("https://example.com", cache_js=False)

# Or send them somewhere specific, flat and without a per-origin level,
# so several sites can share one folder.
scan("https://example.com", js_cache_dir="./js-bodies")
```

With `cache_js=False`, `meta.json` records no `local_path` for JS entries and
`cache_dirs.js` is empty — the metadata never claims a file that is not there.

### Runnable examples

[`examples/`](./examples) holds six scripts covering the whole library surface,
each runnable offline against a bundled fixture site:

| Example | What it shows |
|---------|---------------|
| [`01_basic_scan.py`](./examples/01_basic_scan.py) | One call, and reading the summary |
| [`02_structured_output.py`](./examples/02_structured_output.py) | Options that matter in practice; the JSON result |
| [`03_fine_grained.py`](./examples/03_fine_grained.py) | Each building block used on its own |
| [`04_custom_transport.py`](./examples/04_custom_transport.py) | Custom HTTP layer: logging, another stack, a request budget |
| [`05_custom_plugin.py`](./examples/05_custom_plugin.py) | Adding a plugin for a bundler pydj does not know |
| [`06_batch_scan.py`](./examples/06_batch_scan.py) | Scanning several sites, writing results to a file |

```bash
cd examples
python 01_basic_scan.py
python 04_custom_transport.py

# or without installing pydj at all
PYTHONPATH=../src python 05_custom_plugin.py
```

To develop against your local checkout so edits take effect immediately:

```bash
pip install -e .          # then `python examples/01_basic_scan.py`
pipx install --editable . # same, but the CLI is on PATH too
```

See [`examples/README.md`](./examples/README.md) for the fixture's contents and
the wheel / `uvx` variants.

### Custom plugins

```python
from pydj import PluginRegistry, scan
from pydj.types import AnalyzeInput, ContentType, Result, DiscoveredJS


class MyBundlerPlugin:
    def name(self) -> str:
        return "MyBundlerPlugin"

    def precheck(self, input: AnalyzeInput) -> bool:
        return input.content_type is ContentType.JS and b"myBundler" in input.content

    def analyze(self, input: AnalyzeInput) -> Result:
        result = Result()
        result.urls.append(
            DiscoveredJS(url="https://example.com/my-chunk.js", from_url=input.source_url)
        )
        return result


registry = PluginRegistry()
registry.register(MyBundlerPlugin())

result = scan("https://example.com", registry=registry)
```

A plugin implements three methods:

- `name()` — identifier reported as `from_plugin` in provenance output.
- `precheck(input)` — cheap gate; only if it returns `True` is `analyze` called.
- `analyze(input)` — returns a `Result` with `urls` (finished URLs),
  `probe_targets` (path fragments needing resolution), `intermediates` (config
  files to download), `inline_scripts`, `rsc_probes`, `public_paths` and
  `prepend_urls`.

Use `pydj.plugins.build_registry()` to start from the built-ins and add yours.

## Supported patterns

26 built-in plugins.

| Plugin | What it recognises |
|--------|--------------------|
| `HTMLScriptPlugin` | `<script src>`, `<link rel=modulepreload>`, `<link rel=prefetch>`, `<script type=module>`, inline script bodies |
| `DynamicImportPlugin` | `import()` with any quoting, and `import(/* webpackChunkName */)` comments |
| `WebpackPlugin` | `__webpack_require__.e()`, chunk maps, `webpackChunk_*`, string chunk ids, webpack 4 `HASH.TIMESTAMP` fingerprints with `{"chunk-xxx":1}` markers, rspack runtimes |
| `NextJSPlugin` | App/Pages Router chunks, `_buildManifest`, Turbopack `otherChunks`, RSC Flight `I[...]`/`:HL[...]` data |
| `NuxtJSPlugin` | `/_nuxt/` paths, build assets |
| `VitePlugin` | `__vitePreload()`, `__vite__mapDeps`, modulepreload, `.vite/manifest.json` |
| `SvelteKitPlugin` | `/_app/immutable/nodes/` and `/chunks/` |
| `RequireJSPlugin` | `require([...])`, `define([...])`, `data-main` |
| `ModuleFederationPlugin` | `remoteEntry.js`, `manifest.json`, MF remote entry pages |
| `ModuleFederationManifestPlugin` | Both current `@module-federation/enhanced` manifests and the legacy CDN/region format |
| `HelMicroPlugin` | `metadata.json` component configs, `COMPONENT_CDN_PREFIX` |
| `ESMImportPlugin` | Static `import ... from "..."` |
| `ScriptCreatePlugin` | `document.createElement('script')`, `script.src = ...`, `new URL(...)` |
| `ModernJSPlugin` | ByteDance Modern.js route manifest, `b.p` publicPath |
| `URLPatternPlugin` | Protocol-relative CDN prefixes, quoted `.js` strings |
| `SourceMapPlugin` | `sourceMappingURL`, `X-SourceMap` header, inline `data:` maps |
| `UmiJSPlugin` | `preload_helper` route/file tables, `{id:name}`/`{id:hash}` maps |
| `QiankunPlugin` | qiankun `entry` / vite-plugin-qiankun `proEntry` |
| `GarfishPlugin` | Garfish `apps[].entry` |
| `MicroAppPlugin` | micro-app `start({url})` |
| `WujiePlugin` | wujie `startApp({url})` |
| `IcestarkPlugin` | icestark `AppRoute({url})` |
| `TrunkPlugin` | Trunk (Rust/wasm) `sitemap.json` |
| `HTMLPivotPlugin` | Same-origin `<a>/<link>/<iframe>` entries and quoted `.html` literals |
| `EmpPlugin` | EMP `emp.json` federation manifest |
| `UniversalURLPlugin` | Encoding-aware catch-all: un-escapes JS/URL/Unicode/HTML-entity encodings, then matches `<script src>`, `import()`, `require()`, `loader!path`, SystemJS dependency arrays and runtime string concatenation |

## How it works

1. Download the landing page HTML.
2. If `meta.json` exists for that origin, restore the previous run's JS list and
   sources from cache and stop — no network requests at all.
3. Otherwise dispatch every plugin against the payload. Each result is one of: a
   finished URL (queued), a path fragment (probed later), an intermediate config
   file (downloaded and re-analysed), or a follow-up request.
4. Probe path fragments: webpack chunk paths have no origin, so candidates are
   assembled from known-good JS directories, the source file's directory, known
   CDN prefixes and every known domain, then de-duplicated.
5. For every JS file, HEAD-probe the sibling `.map` and download it if real.
6. Restore sources: prefer `sourcesContent` from the map; fall back to
   re-assembling fragments via `mappings` (VLQ) when it is absent.
7. Write `meta.json` recording every JS URL, its provenance, its source map and
   the restored sources.

### Cache reuse

Caching is on by default and lives under the system temp directory:

| OS | Cache directory |
|----|-----------------|
| Linux/macOS | `/tmp/ejfkdev/dj/` |
| Windows | `%TEMP%\ejfkdev\dj\` |

```
<temp>/ejfkdev/dj/<normalized-origin>/
├── js/           downloaded JS
├── source_map/   .map files
├── sources/      restored original sources (directory tree preserved)
├── html/         landing page (web.html)
└── meta.json     site metadata
```

The second run on a site reuses all of it. `--no-cache` forces a network scan
while still writing what it downloads.

With `-o/--output` every artefact is *also* written to that directory without the
origin level, giving a flat export:

```
<output_dir>/
├── js/
├── source_map/
├── sources/
├── html/
└── meta.json
```


## Output formats

### `text` (bare URL list)

```
https://example.com/js/main.js
https://example.com/js/chunk-abc123.js

--- Summary ---
JS files: 2
Source maps: 1 (found)
Restored sources: 15 files (restored)
```

### `md` (default)

A summary, the JS URL list, per-JS provenance, cache directories and any HTML
entries discovered through multi-page pivoting.

### `json`

```json
{
  "summary": { "jsCount": 3, "sourceMapCount": 1, "sourceCount": 15 },
  "jsURLs": ["https://example.com/js/main.js"],
  "jsDetails": [
    {
      "url": "https://example.com/js/main.js",
      "from_url": "https://example.com/",
      "from_plugin": "HTMLScriptPlugin"
    }
  ],
  "cacheBase": "/tmp/ejfkdev/dj/example.com",
  "cacheDirs": {
    "js": "/tmp/ejfkdev/dj/example.com/js",
    "sourceMap": "/tmp/ejfkdev/dj/example.com/source_map",
    "source": "/tmp/ejfkdev/dj/example.com/sources",
    "html": "/tmp/ejfkdev/dj/example.com/html/web.html"
  }
}
```

## Verified against real sites

Numbers below come from scans of live sites, one per supported framework family.

| Site | Stack | JS URLs | Distinct files |
|------|-------|---------|----------------|
| [react.dev](https://react.dev) | Next.js (App Router) | 42 | 42 |
| [vuejs.org](https://vuejs.org) | VitePress | 63 | 63 |
| [svelte.dev](https://svelte.dev) | SvelteKit | 83 | 83 |
| [arco.design](https://arco.design) | UmiJS | 478 | 478 |
| [chat.z.ai](https://chat.z.ai) | Turbopack | 359 | 183 |

Two things worth reading off that table:

- **A URL count is not a file count.** chat.z.ai's 359 URLs are 183 distinct
  files, 176 of which are served from the site's own origin *and* its CDN. Dedupe
  by path when you want the file inventory and not every route to it.
- **Cross-origin bundles are followed.** A chunk resolved from a CDN host keeps
  its own absolute URL, so the output records where a file actually lives instead
  of assuming the scanned origin.

> Totals drift between runs on sites that generate chunks on demand -- VitePress
> builds `*.md.<hash>.lean.js` per request, so vuejs.org lands anywhere between 56
> and 63. Treat these as approximate.

Three real defects were found this way and fixed:

- **Source maps for cross-origin bundles were silently dropped.** A script hosted
  on a CDN has no site-relative path, and the map writer skipped it -- so the
  sources were restored (1012 files on react.dev) while the summary reported zero
  maps. Now stored under a host-qualified name.
- **The cache root was computed two different ways.** The summary rebuilt it from
  `base_dir + origin`, which differs from the cache's own root when the scanned
  URL has no trailing slash (`https_x` vs `https_x_`), so the summary looked in a
  directory nothing had written to.
- **A global request lock serialised every download.** It was added on the
  assumption that curl-cffi's session is not thread-safe; measuring showed 12
  threads sharing one session give a 4.7x speedup with no errors. Removing it
  took a 390-request scan from 185s to 35s.

## TLS fingerprints

Modern bot detection fingerprints the TLS ClientHello, not just the headers, so a
plain Python client is refused by Cloudflare-era sites while a browser is let
through. `pydj` presents a real browser fingerprint using
[`curl-cffi`](https://github.com/lexiforest/curl_cffi) (curl-impersonate) —
**installed by default**, picking a random browser per scan and honouring
`--no-random-tls` to pin Chrome.

| Situation | Behaviour |
|-----------|-----------|
| Normal install | A random browser fingerprint per scan; `--no-random-tls` pins Chrome |
| `curl-cffi` unavailable | The scan proceeds with a plain Python TLS fingerprint. Nothing is printed; `-d/--debug` reports the reason. Sites that fingerprint TLS may return no JavaScript |
| You pass a [`transport`](#bringing-your-own-http-layer) | pydj does no TLS handling at all — yours decides |

The dependency is declared rather than optional because the fingerprint is the
tool's entire value proposition: without it, a Cloudflare-protected site yields an
empty result while still "succeeding". It is still non-fatal when it cannot be
installed, because some platforms have no wheel and a toolchain may not be
available — refusing to run would be worse than running with a weaker
fingerprint.

If you need a fingerprint pydj does not offer, or your own session, pass a
custom transport.

## Development

```bash
git clone https://github.com/ejfkdev/pydj
cd pydj
pip install -e '.[dev]'
python -m pytest
```

Python 3.10 is the supported floor. See [CONTRIBUTING.md](./CONTRIBUTING.md) for
what a change needs to pass, and [CHANGELOG.md](./CHANGELOG.md) for what has
changed. Security reports go through GitHub's private advisory flow, described in
[SECURITY.md](./SECURITY.md) -- please do not open a public issue for those.

### Continuous integration

`.github/workflows/ci.yml` runs on every push and pull request:

| Job | What it does |
|-----|--------------|
| `test` | The suite on Python 3.10–3.14 (Linux), plus 3.13 on macOS and Windows |
| `verify-cli` | Builds a fixture site, serves it, and runs `python -m pydj` against it — the packaged entry point finding real JS over real HTTP, which unit tests do not cover |
| `differential` | Builds the Go implementation from source and checks that both tools discover the same URL set on a fixture site; skipped with a warning when the upstream checkout is unavailable |
| `lint` | `ruff check`, plus assertions on the built wheel: `curl-cffi` unconditional, correct project URLs, both console scripts present, README as `long_description` |

Python 3.10 is the declared floor and is tested as such; the newer interpreters are
there because a decompression bug once appeared only on 3.14.

```bash
ruff check src tests     # what CI runs
ruff check src tests --fix
```

### Releasing

Publishing is tag-driven — pushing a tag is the only action required:

```bash
# 1. Bump the version in pyproject.toml and src/pydj/__init__.py
# 2. Commit, then tag and push
git tag -a v0.2.0 -m "v0.2.0"
git push origin v0.2.0
```

`.github/workflows/release.yml` then:

1. **verify** — runs the suite and fails if the tag, `pyproject.toml` and
   `__init__.py` versions disagree. PyPI does not allow re-uploading a version, so
   a mismatch has to be caught before publishing, not after.
2. **build** — `python -m build` plus `twine check`.
3. **pypi** — publishes with [trusted publishing][tp] (OIDC), so there is no
   long-lived token in the repository. Falls back to a `PYPI_API_TOKEN` secret if
   one is set.
4. **github-release** — creates the GitHub Release from the annotated tag, with an
   install snippet in the run summary.

[tp]: https://docs.pypi.org/trusted-publishers/

One-time setup for trusted publishing, at
<https://pypi.org/manage/account/publishing/>:

| Field | Value |
|-------|-------|
| PyPI project name | `pydj` |
| Owner | `ejfkdev` |
| Repository name | `pydj` |
| Workflow name | `release.yml` |
| Environment name | `pypi` |

The environment name must match the `environment: pypi` in the workflow; create it
under the repository's Settings → Environments (adding required reviewers there is
worth doing, and gives you a manual gate on publication).

## License

[MPL-2.0](./LICENSE), matching the original project.