Metadata-Version: 2.5
Name: webscraping_ai
Version: 4.1.0
Summary: Official Python client for the WebScraping.AI API.
Project-URL: Homepage, https://webscraping.ai
Project-URL: Documentation, https://webscraping.ai/docs
Project-URL: Source, https://github.com/webscraping-ai/webscraping-ai-python
Project-URL: Changelog, https://github.com/webscraping-ai/webscraping-ai-python/blob/master/CHANGELOG.md
Project-URL: Issues, https://github.com/webscraping-ai/webscraping-ai-python/issues
Author-email: "WebScraping.AI Support" <support@webscraping.ai>
License: MIT License
        
        Copyright (c) WebScraping.AI
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: api,crawler,llm,scraping,webscraping
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.9
Requires-Dist: httpx<1.0,>=0.27
Provides-Extra: dev
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: respx>=0.21; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Description-Content-Type: text/markdown

# webscraping_ai

[![PyPI](https://img.shields.io/pypi/v/webscraping_ai.svg)](https://pypi.org/project/webscraping_ai/)
[![CI](https://github.com/webscraping-ai/webscraping-ai-python/actions/workflows/ci.yml/badge.svg)](https://github.com/webscraping-ai/webscraping-ai-python/actions/workflows/ci.yml)

Official Python client for the [WebScraping.AI](https://webscraping.ai) API —
web scraping with Chromium JavaScript rendering, rotating
datacenter/residential/stealth proxies, and AI-powered question answering and
structured field extraction on any page. Sync and async clients included. See
the [API documentation](https://webscraping.ai/docs) for the full parameter
reference.

> **4.0 is a hard break from 3.x.** See [CHANGELOG.md](CHANGELOG.md) for the
> migration notes. If you cannot update your call sites yet, stay on
> `webscraping_ai == 3.2.1`.

## Install

```bash
pip install webscraping_ai
```

Requires Python 3.9 or newer.

## Quick start

[Sign up](https://webscraping.ai/auth/sign_up) to get an API key — the free
trial includes 2,000 credits, no credit card required. Your key lives in the
[dashboard](https://webscraping.ai/dashboard).

```python
from webscraping_ai import Client

client = Client(api_key="YOUR_API_KEY")

# Page HTML
html = client.html("https://example.com")

# Visible text, optionally as a structured JSON response
text = client.text("https://example.com", text_format="json", return_links=True)

# CSS-selected HTML
heading = client.selected("https://example.com", selector="h1")
multiple = client.selected_multiple("https://example.com", selectors=["h1", "p"])

# LLM-powered helpers
answer = client.question("https://example.com", question="What is the page title?")
fields = client.fields(
    "https://example.com",
    fields={"title": "Main product title", "price": "Current product price"},
)

# Google search results (SERP) for a query
results = client.serp("coffee machines", gl="us", hl="en", page=1)

# Account quota
info = client.account()
```

The client is also a context manager, which closes the underlying connection
pool on exit:

```python
with Client(api_key="...") as client:
    client.html("https://example.com")
```

## Async usage

`AsyncClient` mirrors `Client` but uses `async def` methods backed by
`httpx.AsyncClient`:

```python
import asyncio
from webscraping_ai import AsyncClient

async def main():
    async with AsyncClient(api_key="YOUR_API_KEY") as client:
        html = await client.html("https://example.com")
        print(html)

asyncio.run(main())
```

## Error handling

Every non-2xx response is mapped to a typed exception so you can `except` on
the situation you actually care about rather than parsing status codes:

```python
from webscraping_ai import (
    Client,
    AuthenticationError,
    RateLimitError,
    PaymentRequiredError,
    APITimeoutError,
    APIConnectionError,
)

client = Client(api_key="YOUR_API_KEY")

try:
    client.html("https://example.com")
except AuthenticationError:
    ...  # 403 — wrong or missing API key
except PaymentRequiredError:
    ...  # 402 — out of credits
except RateLimitError:
    ...  # 429 — too many concurrent requests
except APITimeoutError:
    ...  # request did not complete in time
except APIConnectionError:
    ...  # transport-level failure
```

All exceptions inherit from `WebScrapingAIError`, so you can catch everything
the client raises with a single `except` if you prefer. API errors expose the
parsed error envelope (`message`, `status`, `status_code`, `status_message`,
`body`, `response_body`).

`APITimeoutError` and `APIConnectionError` are raised without chaining the
underlying httpx exception (it holds the request URL, which contains your API
key); the original exception type is named in the message instead.

### Logging and your API key

The API key travels in the query string, and httpx logs every request URL at
`INFO` on the `httpx` logger. Importing `webscraping_ai` installs a
`logging.Filter` on that logger that rewrites `api_key=<value>` to
`api_key=[REDACTED]`, so enabling `INFO` logging does not leak the key. The
filter only covers the `httpx` logger; if you log request URLs yourself, redact
them too.

## Endpoint reference

| Method                          | HTTP route          | Returns                       |
| ------------------------------- | ------------------- | ----------------------------- |
| `client.html(...)`              | `GET /html`         | `str` (page HTML)             |
| `client.text(...)`              | `GET /text`         | `str` or `dict` (JSON)        |
| `client.selected(...)`          | `GET /selected`     | `str`                         |
| `client.selected_multiple(...)` | `GET /selected-multiple` | `list`                   |
| `client.question(...)`          | `GET /ai/question`  | `str`                         |
| `client.fields(...)`            | `GET /ai/fields`    | `dict` (wrapped under `result`) |
| `client.serp(...)`              | `GET /serp`         | `dict` (`SerpResult`)         |
| `client.account()`              | `GET /account`      | `dict`                        |

Every page-fetch method accepts the full set of API parameters as keyword
arguments: `headers`, `timeout`, `js`, `js_timeout`, `wait_for`, `proxy`,
`country`, `custom_proxy`, `device`, `error_on_404`, `error_on_redirect`,
`js_script`, plus the per-endpoint extras (`return_script_result`, `format`,
`text_format`, `return_links`, `selector`, `selectors`, `question`, `fields`).
See the [API documentation](https://webscraping.ai/docs) for the full
parameter reference.

### SERP

`client.serp(q, *, engine=None, gl=None, hl=None, page=None)` returns parsed
search engine results for a query. It is query-shaped rather than URL-shaped,
so none of the page-fetch parameters above apply. Flat 15 credits per search;
failed searches are not charged. Raises `ValueError` before any request when
`q` is not a non-blank `str` or `page` is not an `int` >= 1 (the server also
rejects it with a 400, not billed; checking client-side saves the round trip).
`q` is sent as given.

| Parameter | Type  | Default    | Description                                   |
| --------- | ----- | ---------- | --------------------------------------------- |
| `q`       | `str` | —          | Search query (required)                       |
| `engine`  | `str` | `"google"` | Search engine; currently only `google`        |
| `gl`      | `str` | `"us"`     | Two-letter country code for the search        |
| `hl`      | `str` | `"en"`     | Two-letter language code for the results      |
| `page`    | `int` | `1`        | Results page number (10 per page); 1–100, server rejects > 100 with a 400 |

```python
results = client.serp("coffee machines", gl="gb", page=2)
print(results["search_information"]["organic_results_state"])  # "Results for exact spelling"
for r in results["organic_results"]:
    print(r["position"], r["title"], r["link"])
print(results["pagination"])  # {"current": 2, "next": 3}
```

The response dict has `search_parameters` (`engine`, `q`, `gl`, `hl`, `page`),
`search_information` (`query_displayed`, `organic_results_state`, optional
`showing_results_for` and `total_results`), `organic_results` (`position` —
1-based within the page — `title`, `link`, `domain`, `displayed_link`,
optional `snippet` and `date`), optional `related_searches` (`query`), and
`pagination` (`current`, optional `next`). Optional keys are absent when the
engine does not show them, so use `.get()` for those.

### API response-shape notes

Two endpoints return shapes that differ from the OpenAPI spec examples. The
client returns the raw response unchanged, so:

- `/ai/fields` wraps the extracted fields under a `result` key:
  `{"result": {"title": "...", "price": "..."}}`.
- `/selected-multiple` returns `list[list[str]]`, not a flat `list[str]`.

## Development

```bash
mise install                    # or use python 3.13 from any source
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytest
ruff check .
mypy src/webscraping_ai
```

## Smoke testing

`bin/smoke.py` hits every endpoint once against the live API through the sync `Client`, plus one
`account` call through `AsyncClient`. It puts `src/` first on `sys.path`, so it always tests the
working tree (you still need the runtime deps, e.g. from `pip install -e ".[dev]"`). It is not
part of the pytest suite and costs ~32 credits per run: the four page calls run with `js=False`
and `proxy="datacenter"` (1 credit each), `question` and `fields` cost 6 each, and the SERP call
is 15. Each case checks the result shape, not just that no exception was raised (SERP must return
organic results for the query sent, `selected_multiple` must match something, and so on), and
FAIL lines redact the API key.

```bash
WEBSCRAPING_AI_API_KEY=... python bin/smoke.py
```

Each call prints an `ok` or `FAIL` line (any exception counts as a failure, and the sweep
continues); the script exits non-zero if any call fails.

## Links

- [WebScraping.AI](https://webscraping.ai) — features, pricing, signup
- [API documentation](https://webscraping.ai/docs)
- [Dashboard](https://webscraping.ai/dashboard) — API key, usage, request builder
- Other official clients: [JavaScript](https://github.com/webscraping-ai/webscraping-ai-js) · [Ruby](https://github.com/webscraping-ai/webscraping-ai-ruby) · [PHP](https://github.com/webscraping-ai/webscraping-ai-php) · [Go](https://github.com/webscraping-ai/webscraping-ai-go) · [Java](https://github.com/webscraping-ai/webscraping-ai-java) · [.NET](https://github.com/webscraping-ai/webscraping-ai-dotnet) · [CLI](https://github.com/webscraping-ai/webscraping-ai-cli) · [MCP server](https://github.com/webscraping-ai/webscraping-ai-mcp-server) · [n8n node](https://github.com/webscraping-ai/webscraping-ai-n8n)
- Support: [support@webscraping.ai](mailto:support@webscraping.ai)

## License

[MIT](LICENSE).
