Metadata-Version: 2.4
Name: web-scraping-api-sdk
Version: 0.1.3
Summary: Web scraping API client for ScrapingBee: HTML API, Auto-Mode cost control, and the Google, Amazon, Walmart, YouTube, ChatGPT, Gemini, Shopee and agentic search endpoints.
Author: wordstotech
License: MIT
Project-URL: Homepage, https://www.scrapingbee.com/features/ai-web-scraping-api/
Project-URL: Documentation, https://www.scrapingbee.com/documentation/
Project-URL: Repository, https://github.com/ScrapingBee/ScrapingBee
Keywords: web-scraping-api,scraping-api,scrapingbee,web-scraping,scraper-api,serp-api,proxy-rotation,headless-browser,data-extraction,ai-scraping
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.25.0
Dynamic: license-file

# web-scraping-api-sdk

<p align="center">
  <a href="https://www.scrapingbee.com/">
    <img src="https://raw.githubusercontent.com/ScrapingBee/ScrapingBee/main/assets/banner.png" alt="ScrapingBee Web Scraper API" width="100%" />
  </a>
</p>

A Python client for the [ScrapingBee](https://www.scrapingbee.com/) web scraping API. One class
covers the HTML API, Auto-Mode cost control, and nine dedicated scraper endpoints that return
parsed JSON instead of HTML you have to select against.

> Independent client, published separately from ScrapingBee's own `scrapingbee` SDK. You need a
> ScrapingBee API key: [app.scrapingbee.com](https://app.scrapingbee.com/), 1,000 free credits on
> signup, no card.

Every endpoint, parameter, credit cost and response field documented below was verified against
the live API on 2026-08-25.

## Install

```bash
pip install web-scraping-api-sdk
```

Requires Python 3.8+ and `requests`.

## Quickstart

```python
from web_scraping_api_sdk import ScrapingBeeAPI

api = ScrapingBeeAPI("YOUR-API-KEY")

page = api.scrape("https://news.ycombinator.com", render_js=False)
print(page.status_code, page.cost, page.request_id)
print(page.text[:500])
```

## Authentication

The client sends `Authorization: Bearer YOUR-API-KEY` on every request. ScrapingBee still accepts
an `api_key` query parameter, but its documentation marks that method deprecated, so this client
does not use it.

## Cost control with Auto-Mode

Picking a proxy tier by hand means overpaying on pages that need nothing special, or getting
blocked on pages that need a lot. `auto()` asks ScrapingBee to try configurations from cheapest to
most expensive and charge only for the one that actually returned the page.

```python
page = api.auto("https://example.com", max_cost=25)
print(page.auto_cost)   # 1, 5, 10 or 25, or 0 if every configuration failed
```

`max_cost` caps the climb: 25 reaches premium proxy with JavaScript and never touches the
75-credit stealth tier. **If no configuration succeeds, the request costs 0 credits.**

## The HTML API

`scrape()` fetches any public URL and accepts the full parameter set from the
[ScrapingBee documentation](https://www.scrapingbee.com/documentation/).

**JavaScript rendering.** On by default. A real headless browser executes the page, which is what
makes single-page apps and lazy-loaded content work. Turn it off with `render_js=False` to drop
from 5 credits to 1. Control timing with `wait`, `wait_for` (a CSS selector) or `wait_browser`.

**Proxies.** Rotating datacenter IPs by default. `premium_proxy=True` switches to residential IPs,
which is also what `country_code` geotargeting requires. `stealth_proxy=True` uses the anti-bot
tier. `session_id` keeps the same IP across a sequence of requests.

**Output shape.** Ask for text with `return_page_text=True`, markdown with
`return_page_markdown=True` (useful for RAG and LLM pipelines), a PNG with `screenshot()`, or
structured JSON via [data extraction rules](https://www.scrapingbee.com/features/data-extraction/):

```python
page = api.scrape(
    "https://news.ycombinator.com",
    extract_rules={"titles": {"selector": ".titleline > a", "type": "list"}},
)
print(page.json())
```

**AI extraction.** Describe the fields in plain language instead of writing selectors, via
[AI web scraping](https://www.scrapingbee.com/features/ai-web-scraping-api/). Adds 5 credits.

```python
page = api.scrape("https://example.com", ai_query="the main heading and the first paragraph")
```

## Dedicated scrapers

Nine endpoints return parsed JSON, so there are no selectors to maintain when a layout changes.
Response fields listed below are the real top-level keys returned by the live API.

### Google search API

```python
results = api.google("web scraping api", country_code="us").json()
for r in results["organic_results"]:
    print(r["position"], r["title"], r["url"])
```

Returns `organic_results`, `ai_overviews`, `top_ads`, `bottom_ads`, `shopping_ads`,
`related_queries`, `questions`, `top_stories`, `news_results`, `local_results`, `hotel_results`,
`knowledge_graph`, `related_searches` and `meta_data`. Costs 15 credits, or 10 with
`light_request=True`.

One `search_type` parameter switches surface:

| `search_type` | Returns | Scraper page |
| --- | --- | --- |
| `classic` | Organic results, the default | [Google scraper](https://www.scrapingbee.com/features/google/) |
| `news` | News results | [Google News API](https://www.scrapingbee.com/scrapers/google-news-scraper-api/) |
| `maps` | Local listings | [Google API docs](https://www.scrapingbee.com/documentation/google-api/) |
| `images` | Image results | [Google Images scraper](https://www.scrapingbee.com/scrapers/google-image-scraper/) |
| `lens` | Visual matches for an image URL | [Google Lens API](https://www.scrapingbee.com/scrapers/google-lens-api/) |
| `shopping` | Product listings, sortable and price-filtered | [Google Shopping API](https://www.scrapingbee.com/scrapers/google-shopping-api/) |
| `ai_mode` | The AI-generated answer | [Google AI Mode API](https://www.scrapingbee.com/scrapers/google-ai-mode-api/) |
| `ads` | Paid placements | [Google Ads API](https://www.scrapingbee.com/scrapers/google-ads-api/) |

`news` is unavailable with `device="mobile"`, `lens` expects an image URL as the search term, and
`ai_mode` accepts at most 400 characters. Shopping supports `sort_by`, `min_price` and `max_price`,
priced in the currency implied by `country_code`.

### Fast Search API

```python
api.fast_search("web scraping api").json()
```

A lighter search endpoint returning `answer_box`, `organic`, `people_also_ask`,
`related_searches`, `top_stories` and `search_parameters`. Flat 10 credits. See
[Fast Search](https://www.scrapingbee.com/features/fast-search/).

### Amazon scraper API

```python
search = api.amazon_search("laptop stand").json()
asin = search["products"][0]["asin"]

product = api.amazon_product(asin).json()
print(product["title"], product["buybox"], product["rating"])

pricing = api.amazon_pricing(asin).json()
```

`amazon_search()` returns `products`, `products_count`, `refinements`, `page` and `url`.
`amazon_product()` takes an ASIN or a product URL as `query` and returns `asin`, `brand`,
`bullet_points`, `buybox`, `category`, `coupon`, `currency`, `delivery`, `description` and more.
An unknown ASIN returns HTTP 404 with `{"error": "Product not found"}`. All three cost 5 credits,
or 15 with `light_request=False`.

Related pages: [Amazon search](https://www.scrapingbee.com/scrapers/amazon-search-api/),
[ASIN API](https://www.scrapingbee.com/scrapers/amazon-asin-api/),
[review API](https://www.scrapingbee.com/scrapers/amazon-review-api/),
[offers API](https://www.scrapingbee.com/scrapers/amazon-offers-api/),
[best sellers](https://www.scrapingbee.com/scrapers/amazon-best-sellers-api/),
[seller API](https://www.scrapingbee.com/scrapers/amazon-seller-api/),
[Amazon feature page](https://www.scrapingbee.com/features/amazon/).

### Walmart scraper API

```python
api.walmart_search("coffee maker").json()
api.walmart_product("10450114").json()
```

`walmart_product()` returns `price`, `price_strikethrough`, `currency`, `rating`, `rating_count`,
`fulfillment`, `out_of_stock`, `gtin`, `seller_id`, `images` and more. Search supports
`start_page`, `sort_by`, `min_price`, `max_price`, `store_id` and `delivery_zip`. Costs 10 credits,
or 15 with `light_request=False`.

Related pages: [Walmart price API](https://www.scrapingbee.com/scrapers/walmart-price-api/),
[search results](https://www.scrapingbee.com/scrapers/walmart-search-engine-results-api/),
[inventory](https://www.scrapingbee.com/scrapers/walmart-inventory-api/),
[seller API](https://www.scrapingbee.com/scrapers/walmart-seller-api/),
[Walmart feature page](https://www.scrapingbee.com/features/walmart/).

### YouTube scraper API

```python
api.youtube_search("web scraping tutorial").json()
meta = api.youtube_metadata("dQw4w9WgXcQ").json()
subs = api.youtube_subtitles("dQw4w9WgXcQ").json()
```

Metadata and subtitles take a **video id**, not a URL. Metadata returns `title`, `description`,
`channel_id`, `channel_url`, `duration`, `like_count`, `comment_count`, `upload_date`,
`categories`, `tags`, `thumbnails`, `formats`, `age_limit` and `is_live`. Subtitles returns
`subtitles`. Search filters include `type`, `duration`, `upload_date`, `sort_by`, `hd`, `4k`,
`live`, `creative_commons` and `subtitles`. Flat 5 credits.

Related pages: [YouTube video scraper](https://www.scrapingbee.com/scrapers/youtube-video-scraper-api/),
[transcript scraper](https://www.scrapingbee.com/scrapers/youtube-transcript-scraper-api/),
[comment scraper](https://www.scrapingbee.com/scrapers/youtube-comment-scraper-api/),
[captions API](https://www.scrapingbee.com/scrapers/youtube-captions-api/),
[YouTube feature page](https://www.scrapingbee.com/features/youtube/).

### ChatGPT and Gemini

```python
api.chatgpt("what is a web scraping api").json()
api.gemini("what is a web scraping api").json()
```

ChatGPT returns `results_text`, `results_markdown`, `results_json`, `full_html`, `llm_model` and
`prompt`. Gemini returns `results_text`, `results_markdown`, `citations`, `full_html` and `prompt`.
Both cost 15 credits. ScrapingBee notes ChatGPT citations are not returned on every call, so treat
them as best effort. See [ChatGPT](https://www.scrapingbee.com/features/chatgpt/) and
[Gemini](https://www.scrapingbee.com/scrapers/gemini-api/).

### Shopee and agentic search

```python
api.shopee("https://shopee.co.id/product-name-i.93014939.1881883105")
api.agentic_search("senior data scientists at large tech companies in the US", limit=10)
```

[Shopee](https://www.scrapingbee.com/scrapers/shopee-api/) costs 75 credits per product page and
takes a product URL. Agentic employee search costs **3,750 credits per call**, so check `usage()`
before putting it in a loop. Both parameter sets come from the official documentation. The Shopee
endpoint returned HTTP 500 for the placeholder URL used in the docs, charged at 0 credits, which
is the documented behaviour for 500 responses; agentic search was not run because of its cost.

## Credit reference

| Configuration | Credits |
| --- | --- |
| Rotating proxy, no JavaScript | 1 |
| Rotating proxy with JavaScript (default) | 5 |
| Premium proxy, no JavaScript | 10 |
| Premium proxy with JavaScript | 25 |
| Stealth proxy with JavaScript | 75 |
| `ai_query` or `ai_extract_rules` | +5 |
| Google, Walmart | 10 light, 15 standard |
| Amazon | 5 light, 15 standard |
| YouTube | 5 |
| Fast Search | 10 |
| ChatGPT, Gemini | 15 |
| Shopee | 75 |
| Agentic employee search | 3,750 |

Check consumption with `api.usage()`, which returns `max_api_credit`, `used_api_credit`,
`max_concurrency`, `current_concurrency` and `renewal_subscription_date`. It is free and limited to
6 calls per minute. Full pricing at [scrapingbee.com/pricing](https://www.scrapingbee.com/pricing/).

## Response metadata

```python
page.cost           # Spb-cost, credits charged
page.auto_cost      # Spb-auto-cost, only on auto() calls, 0 when all tiers failed
page.request_id     # Spb-request-id, quote this in support requests
page.resolved_url   # Spb-resolved-url, final URL after redirects
page.status_code
page.headers
```

## Errors

Non-2xx responses raise `ScrapingBeeError` carrying `status_code`, `body` and `request_id`. HTTP
500 responses are not charged credits, so retrying on 500 is safe.

```python
from web_scraping_api_sdk import ScrapingBeeAPI, ScrapingBeeError

try:
    page = api.scrape("https://example.com", stealth_proxy=True)
except ScrapingBeeError as error:
    print(error.status_code, error.request_id)
```

## Scope

Public, pre-login pages only. Scraping behind login credentials is prohibited by the
[ScrapingBee terms](https://www.scrapingbee.com/terms-and-conditions/). Keep your API key out of
shared environments, including AI coding assistants.

## Links

- [ScrapingBee documentation](https://www.scrapingbee.com/documentation/)
- [Pricing and credits](https://www.scrapingbee.com/pricing/)
- [Examples in eight languages](https://github.com/ScrapingBee/ScrapingBee)

MIT licensed.
