Metadata-Version: 2.4
Name: web-scraping-api-sdk
Version: 0.1.0
Summary: Web scraping API client for ScrapingBee: HTML API, Auto-Mode cost control, and the Google, Amazon, Walmart, YouTube, ChatGPT, Gemini, Shopee and agentic search endpoints.
Author: wordstotech
License: MIT
Project-URL: Homepage, https://www.scrapingbee.com/features/ai-web-scraping-api/
Project-URL: Documentation, https://www.scrapingbee.com/documentation/
Project-URL: Repository, https://github.com/ScrapingBee/ScrapingBee
Keywords: web-scraping-api,scraping-api,scrapingbee,web-scraping,scraper-api,serp-api,proxy-rotation,headless-browser,data-extraction,ai-scraping
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.25.0
Dynamic: license-file

# web-scraping-api-sdk

A Python client for the **ScrapingBee** web scraping API. It wraps the HTML API, Auto-Mode
cost control, and all nine dedicated scraper endpoints behind one class.

> This package talks to ScrapingBee (`app.scrapingbee.com`). It is an independent client,
> published separately from ScrapingBee's own `scrapingbee` SDK. You need a ScrapingBee API
> key to use it: [app.scrapingbee.com](https://app.scrapingbee.com/), 1,000 free credits on signup.

## Install

```bash
pip install web-scraping-api-sdk
```

## Quickstart

```python
from web_scraping_api_sdk import ScrapingBeeAPI

api = ScrapingBeeAPI("YOUR-API-KEY")

page = api.scrape("https://news.ycombinator.com", render_js=False)
print(page.status_code, page.cost, page.request_id)
print(page.text[:500])
```

Authentication uses the `Authorization: Bearer` header. ScrapingBee still accepts the
`api_key` query parameter but marks it deprecated, so this client does not use it.

## Auto-Mode

Choosing a proxy tier by hand means overpaying on easy pages or getting blocked on hard ones.
`auto()` asks ScrapingBee to try configurations from cheapest to most expensive and charge only
for the one that works:

```python
page = api.auto("https://example.com", max_cost=25)
print(page.auto_cost)   # one of 1, 5, 10, 25 or 0 if every tier failed
```

`max_cost=25` allows premium proxy with JavaScript but never the 75-credit stealth tier. When
every configuration fails, the request costs nothing.

## Endpoint reference

| Method | Endpoint | Credits |
| --- | --- | --- |
| `scrape(url, **params)` | `/api/v1` | 1 to 75, plus 5 for AI parameters |
| `auto(url, max_cost=None)` | `/api/v1` with `mode=auto` | only the tier that succeeds, 0 on total failure |
| `screenshot(url, full_page=False)` | `/api/v1` with `screenshot` | same ladder as `scrape` |
| `google(search, search_type=...)` | `/api/v1/google` | 15, or 10 light |
| `fast_search(search)` | `/api/v1/fast_search` | 10 |
| `amazon_search(query)` | `/api/v1/amazon/search` | 5 light, 15 standard |
| `amazon_product(asin)` | `/api/v1/amazon/product` | 5 light, 15 standard |
| `amazon_pricing(asin)` | `/api/v1/amazon/pricing/` | 5 light, 15 standard |
| `walmart_search(query)` | `/api/v1/walmart/search` | 10 light, 15 standard |
| `walmart_product(product_id)` | `/api/v1/walmart/product` | 10 light, 15 standard |
| `youtube_search(search)` | `/api/v1/youtube/search` | 5 |
| `youtube_metadata(search)` | `/api/v1/youtube/metadata` | 5 |
| `youtube_subtitles(search)` | `/api/v1/youtube/subtitles` | 5 |
| `chatgpt(prompt)` | `/api/v1/chatgpt` | 15 |
| `gemini(prompt)` | `/api/v1/gemini` | 15 |
| `shopee(url)` | `/api/v1/shopee` | 75 |
| `agentic_search(prompt, limit)` | `/api/v1/agentic_search` | 3,750 |
| `usage()` | `/api/v1/usage` | free, 6 calls per minute |

## Google surfaces

One parameter switches between eight result types:

```python
api.google("web scraping api", search_type="news", country_code="us")
api.google("running shoes", search_type="shopping", min_price=50, sort_by="price_low_to_high")
api.google("how does virtual interlining work", search_type="ai_mode")
```

Accepted values: `classic`, `news`, `maps`, `images`, `lens`, `shopping`, `ai_mode`, `ads`.
`news` is unavailable with `device="mobile"`, `lens` expects an image URL, and `ai_mode` caps
the query at 400 characters.

## Reading the cost

Every response exposes the ScrapingBee headers as attributes:

```python
page = api.scrape("https://example.com")
page.cost          # Spb-cost, credits charged
page.auto_cost     # Spb-auto-cost, present only for auto() calls
page.request_id    # Spb-request-id, quote this in support requests
page.resolved_url  # Spb-resolved-url, the final URL after redirects
```

## Errors

Non-2xx responses raise `ScrapingBeeError`, carrying the status, the body and the request id.
HTTP 500 responses are not charged credits, so retrying on 500 is safe.

```python
from web_scraping_api_sdk import ScrapingBeeAPI, ScrapingBeeError

try:
    page = api.scrape("https://example.com", stealth_proxy=True)
except ScrapingBeeError as error:
    print(error.status_code, error.request_id)
```

## Scope

Public, pre-login pages only. Scraping behind login credentials is prohibited by the
[ScrapingBee terms](https://www.scrapingbee.com/terms-and-conditions/). Keep your API key out
of shared environments, including AI coding assistants.

## Links

- [ScrapingBee documentation](https://www.scrapingbee.com/documentation/)
- [Plans and credits](https://www.scrapingbee.com/pricing/)
- [Examples in eight languages](https://github.com/ScrapingBee/ScrapingBee)

MIT licensed.
