Metadata-Version: 2.4
Name: fetchquest
Version: 0.1.0
Summary: The web fetcher that always finds a way
Keywords: browser,fetch,http,playwright,web-scraping
Author: Stanislav Lukyanov
Author-email: Stanislav Lukyanov <stan@slukyano.dev>
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Operating System :: MacOS
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Utilities
Requires-Dist: httpx>=0.28
Requires-Dist: click>=8.1
Requires-Dist: playwright>=1.49
Requires-Dist: html2text>=2025.4.15
Requires-Dist: trafilatura>=2.0
Requires-Dist: truststore>=0.10.4
Requires-Python: >=3.14
Project-URL: Homepage, https://github.com/slukyano/fetchquest
Project-URL: Repository, https://github.com/slukyano/fetchquest
Project-URL: Issues, https://github.com/slukyano/fetchquest/issues
Project-URL: Changelog, https://github.com/slukyano/fetchquest/blob/main/CHANGELOG.md
Description-Content-Type: text/markdown

# FetchQuest

<p align="center">
  <img src="https://raw.githubusercontent.com/slukyano/fetchquest/main/docs/assets/logo.png" alt="FetchQuest" width="760">
</p>

<p align="center">
  <a href="https://github.com/slukyano/fetchquest/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/slukyano/fetchquest/actions/workflows/ci.yml/badge.svg"></a>
  <a href="https://www.python.org/"><img alt="Python 3.14" src="https://img.shields.io/badge/python-3.14-3776AB.svg"></a>
  <a href="https://github.com/slukyano/fetchquest/blob/main/LICENSE"><img alt="License: MIT" src="https://img.shields.io/badge/license-MIT-C9A227.svg"></a>
</p>

FetchQuest is a resilient command-line web fetcher that escalates from plain
HTTP to browser-like requests and, when needed, a real Playwright browser. It
returns readable content instead of leaving the caller to diagnose blocks,
JavaScript shells, rendering delays, and vanished pages.

```console
$ fetchquest https://example.com/app --strategy playwright --wait-selector "#results" -v
# https://example.com/app → playwright (status 200)
```

## Why FetchQuest

- **Automatic escalation** — try lightweight HTTP first, add browser headers
  when needed, and launch Chromium for JavaScript-rendered pages.
- **Content-aware results** — preserve raw responses, convert pages to readable
  text, or extract the main article content without surrounding boilerplate.
- **Bounded browser readiness** — require load, network-idle, or a visible CSS
  selector while keeping all waits inside one hard deadline.
- **Reliable bulk work** — fetch URL lists, continue through individual
  failures, avoid filename collisions, and write a Markdown report.
- **Careful archive recovery** — the opt-in Wayback strategy runs only after the
  original resource returns 404 or 410.
- **Agent-friendly** — use the CLI directly or install the included Claude Code
  plugin as a resilient replacement for WebFetch.

## Install

Run a tagged release without installing it:

```bash
uvx fetchquest https://example.com
```

For a persistent command:

```bash
uv tool install fetchquest
fetchquest https://example.com
fetchquest --version
```

To run the latest unreleased source directly from `main`:

```bash
uvx --from git+https://github.com/slukyano/fetchquest fetchquest https://example.com
```

The Playwright strategy needs a one-time Chromium installation:

```bash
uvx playwright install chromium
```

Python 3.14 or newer and [uv](https://docs.astral.sh/uv/) are required.

## Quick start

```bash
# Fetch readable text to stdout
fetchquest https://example.com

# Save raw HTML and readable text
fetchquest https://example.com \
  --output pages/ \
  --formats raw,text

# Extract only the main article content
fetchquest https://example.com --formats text-trim

# Require JavaScript-rendered content to appear
fetchquest https://example.com/app \
  --strategy playwright \
  --wait-selector "#results article"

# Fetch a list and recover genuinely missing pages from Wayback
fetchquest --input urls.txt \
  --output pages/ \
  --formats raw,text \
  --strategy smart-cache
```

Each input-file line contains a URL and an optional output prefix:

```text
https://example.com/pricing pricing
https://example.com/about
```

## How the smart chain works

1. Fetch with plain httpx.
2. If that does not return HTTP 200, retry with browser-like headers.
3. When a successful HTML response is only an empty JavaScript shell, render it
   with Playwright.
4. When both HTTP strategies fail because of TLS, try Playwright.
5. With `smart-cache`, consult Wayback only if an original HTTP response was 404
   or 410.

Retries apply to transient failures such as HTTP 429, HTTP 503, and network
timeouts. The per-URL `--timeout` remains the hard deadline across strategy
attempts, retries, browser startup, required waits, render settling, and capture.

## Strategies and formats

| Strategy | Behavior |
|---|---|
| `smart` | Automatic HTTP, browser-header, and Playwright chain |
| `smart-cache` | Smart chain plus Wayback for original 404/410 responses |
| `simple` | Plain httpx request |
| `ua-spoof` | httpx with browser-like request headers |
| `playwright` | Headless Chromium |
| `cache` | Wayback Machine only |

| Format | Behavior |
|---|---|
| `text` | Readable Markdown-like text; the default |
| `text-trim` | Main content without navigation, footers, or ads |
| `raw` | Original response bytes |

Run `fetchquest --help` for all options. The
[CLI specification](https://github.com/slukyano/fetchquest/blob/main/docs/cli-spec.md)
documents exact defaults, constraints, output behavior, readiness semantics,
and exit codes.

## Claude Code plugin

Install FetchQuest as a Claude Code plugin:

```bash
claude plugin marketplace add slukyano/fetchquest
claude plugin install fetchquest@fetchquest
```

The included fetch skill teaches Claude when to use FetchQuest and how to select
formats, strategies, and browser-readiness controls.

## Development

Setup, validation, network-test, documentation, and pull-request guidance lives
in
[CONTRIBUTING.md](https://github.com/slukyano/fetchquest/blob/main/CONTRIBUTING.md).
Project documentation starts at
[docs/index.md](https://github.com/slukyano/fetchquest/blob/main/docs/index.md),
and user-visible changes are tracked in
[CHANGELOG.md](https://github.com/slukyano/fetchquest/blob/main/CHANGELOG.md).

## License

[MIT](https://github.com/slukyano/fetchquest/blob/main/LICENSE)
