Metadata-Version: 2.4
Name: scrapenest
Version: 0.5.2
Summary: Official Python client for the ScrapeNest API — web scraping, search, AI answers, screenshots, PDFs, Google Maps reviews, and transcription.
Project-URL: Homepage, https://scrapenest.dev
Project-URL: Documentation, https://scrapenest.dev/docs
Author-email: ScrapeNest <support@scrapenest.dev>
License: MIT
License-File: LICENSE
Keywords: api,crawler,scrapenest,scraping,screenshot,search,serp,web-scraping
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Typing :: Typed
Requires-Python: >=3.9
Requires-Dist: httpx>=0.27.0
Provides-Extra: dev
Requires-Dist: pytest-asyncio>=0.24; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: respx>=0.21; extra == 'dev'
Description-Content-Type: text/markdown

# scrapenest

Official Python client for the [ScrapeNest](https://scrapenest.dev) API — web scraping, search, AI answers, screenshots, PDFs, Google Maps reviews, gas prices, and audio transcription.

- Docs: https://scrapenest.dev/docs
- Get an API key: https://scrapenest.dev

## Install

```bash
pip install scrapenest
```

Requires Python 3.9+.

## Quick start

```python
from scrapenest import Client

client = Client("sn_your_key_here")

result = client.search("python web scraping", num_results=5)
for r in result.results:
    print(r.title, r.url)
```

Every method returns a typed dataclass. Pass `include_usage=True` (where supported) to get a `usage` block with the credits charged.

Options you leave out are not sent, so the API applies its own defaults.

## Search

```python
# SERP results. depth = "fast" | "balanced" | "thorough" | "stealth"
hits = client.search("openai news", depth="balanced", num_results=10)

# Search + fetch & clean the top pages
deep = client.search_deep("what is rust lang", fetch_top=3)
for page in deep.pages:
    print(page.url, len(page.text or ""))

# Cited answer synthesized from live sources
answer = client.search_answer("who wrote the odyssey", max_sources=3)
print(answer.answer)
for c in answer.citations:
    print(c.index, c.url)
```

`timeout_seconds` caps how long `search` and `search_deep` wait for the engines. Search responses report how the engines did:

```python
hits = client.search("rust async runtimes", timeout_seconds=3)
if hits.partial:
    print(hits.error, hits.note)
    for name, status in (hits.engine_status or {}).items():
        print(name, status.status, status.results)
```

## Scrape a page

```python
page = client.scrape_url(
    "https://example.com",
    render="auto",          # auto | direct | browser | stealth
    extract=True,
    return_markdown=True,
)
print(page.title)
print(page.markdown[:500])

# AI extraction
data = client.scrape_url("https://example.com/product", ai_query="Return the product name and price")
print(data.ai_extract)
```

Read a long page in windows with `max_chars` and `offset`, or pull out the passages that match `focus`:

```python
url = "https://example.com/long-article"
part = client.scrape_url(url, max_chars=4000)
text = part.text or ""
while part.next_offset is not None:
    part = client.scrape_url(url, max_chars=4000, offset=part.next_offset)
    text += part.text or ""

passages = client.scrape_url(url, max_chars=4000, focus='"return policy"')
```

## Newer API fields

Every response keeps the JSON it was built from in `.raw`, and every request method takes `extra_body`, merged into the request as-is, so fields the API adds before this SDK names them are still reachable:

```python
res = client.search("query", extra_body={"some_new_option": True})
print(res.raw.get("some_new_field"))
```

## Screenshots & PDF

```python
shot = client.screenshot("https://example.com", full_page=True, format="png")
with open("page.png", "wb") as f:
    import base64
    f.write(base64.b64decode(shot.image_base64))

doc = client.pdf("https://example.com", format="A4", landscape=False)
with open("page.pdf", "wb") as f:
    import base64
    f.write(base64.b64decode(doc.pdf_base64))
```

## Google Maps reviews

Pass the place's name and city, or any Google Maps link of it. If the name matches several places, the call fails free of charge with the error code `place_ambiguous`, and the message lists them with their place IDs. The `place_id` in a response skips the name lookup on later calls.

```python
res = client.maps_reviews("Katz's Delicatessen, New York", max_reviews=50)
print(res.name, res.rating, res.review_count)
for rev in res.reviews:
    print(rev.author_name, rev.rating, rev.text)

# Large pulls run as async jobs
run = client.maps_reviews_async(res.place_id, max_reviews=2000)
final = client.wait_for_run(run.run_id)
print(f"{len(final.result['reviews'])} reviews fetched")
# Need to stop a long job early? client.cancel_run(run.run_id)
```

## Gas prices

```python
gas = client.gas_prices("Los Angeles, CA", grade="regular", limit=10)
for s in gas.stations:
    print(s.brand, s.address, s.price)
if gas.region_stats:
    print(gas.region_stats.region, gas.region_stats.average_price)
```

## Translate & transcribe

```python
tr = client.translate("Hello, how are you?", "es")
print(tr.translated)

audio = client.audio_transcript("https://www.youtube.com/watch?v=...")
print(audio.text)
```

## Async client

```python
import asyncio
from scrapenest import AsyncClient

async def main():
    async with AsyncClient("sn_your_key_here") as client:
        result = await client.search("python web scraping")
        print(result.results[0].title)

asyncio.run(main())
```

`AsyncClient` exposes the same methods as `Client`.

## Error handling

```python
from scrapenest import RateLimitError, AuthenticationError, PaymentRequiredError

try:
    result = client.search("test")
except RateLimitError as e:
    print(f"Rate limited. Retry after {e.retry_after}s")
except PaymentRequiredError:
    print("Out of credits")
except AuthenticationError:
    print("Invalid API key")
```

## License

MIT
