Metadata-Version: 2.4
Name: scrapequeue
Version: 0.1.0
Summary: Polite, resumable URL queues for bounded scraping jobs.
Project-URL: Homepage, https://github.com/Karunasagar12/scrapequeue
Project-URL: Repository, https://github.com/Karunasagar12/scrapequeue
Project-URL: Issues, https://github.com/Karunasagar12/scrapequeue/issues
Project-URL: Changelog, https://github.com/Karunasagar12/scrapequeue/blob/main/CHANGELOG.md
Author-email: Karunasagar Mohansundar <quasarkaruna10@gmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: checkpoint,queue,rate-limit,resumable,retry,scraping,sqlite,web-scraping
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: System :: Archiving
Requires-Python: >=3.11
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == 'dev'
Requires-Dist: pytest>=8.2; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Requires-Dist: twine>=5.1; extra == 'dev'
Provides-Extra: examples
Requires-Dist: httpx>=0.27; extra == 'examples'
Provides-Extra: playwright
Requires-Dist: playwright>=1.45; extra == 'playwright'
Description-Content-Type: text/markdown

# scrapequeue

Polite, resumable URL queues for bounded Python scraping jobs.

`scrapequeue` is for developers who already know *what URLs they want to fetch* and want reliable queue management without adopting a crawler framework. You provide a `fetch(url)` function; `scrapequeue` handles URL ingestion, dedupe, rate limiting, retries, SQLite checkpointing, resume, failed URL inspection, and CSV export.

## When to use it

Use `scrapequeue` when you have:

- hundreds to low-hundreds-of-thousands of known URLs
- a script/notebook that keeps growing retry/checkpoint code
- a need to stop/restart jobs without re-fetching successes
- a custom fetcher using `httpx`, `requests`, Playwright, or another client

## When not to use it

Use something else when you need:

- link discovery / crawling / spider frontiers
- distributed queues across machines
- proxy rotation, CAPTCHA solving, or anti-bot evasion
- HTML parsing helpers baked into the queue

For those, consider Scrapy/Crawlee/Playwright directly. `scrapequeue` intentionally stays smaller.

## Install

```bash
pip install scrapequeue
```

For local development:

```bash
pip install -e '.[dev]'
```

## Five-line quickstart

```python
from scrapequeue import Queue

queue = Queue(rate_limit="1/s", retries=3, checkpoint="jobs.sqlite")
queue.add_urls(["https://example.com/a", "https://example.com/b"])
queue.run(lambda url: {"url": url, "body": "..."})
queue.export_csv("results.csv")
```

## Real fetch function

```python
import httpx
from scrapequeue import Queue

queue = Queue(rate_limit="30/m", retries=3, checkpoint="jobs.sqlite")
queue.add_urls("urls.csv")  # requires a `url` column


def fetch(url: str) -> dict:
    response = httpx.get(url, timeout=10, follow_redirects=True)
    response.raise_for_status()
    return {
        "status": response.status_code,
        "content_type": response.headers.get("content-type", ""),
        "body": response.text,
    }


queue.run(fetch, progress=True)
queue.export_csv("results.csv")
queue.export_failed("failed.csv")
```

## Input sources

```python
queue.add_urls(["https://a.com", "https://b.com"])
queue.add_urls("urls.txt")
queue.add_urls("urls.csv")  # CSV must contain a `url` column
```

Duplicate URLs are ignored, so re-adding the same source is safe.

## State machine

Every URL is persisted in SQLite with one of these states:

```text
pending -> in_progress -> success
pending -> in_progress -> retrying -> pending
pending -> in_progress -> failed
```

On startup, stranded `in_progress` / `retrying` rows are reset to `pending` so interrupted jobs can resume.

## Retry failed URLs

```python
queue.export_failed("failed.csv")
queue.retry_failed()
queue.run(fetch, only_failed=True)
```

This lets you inspect failures, fix your fetcher/network issue, then rerun only failed URLs.

## Backoff and progress

```python
queue = Queue(backoff="exponential", backoff_seconds=0.5)
queue.run(fetch, progress=True)
```

Backoff modes:

- `fixed`
- `linear` default
- `exponential`
- `jitter`

Progress can also be a callback:

```python
events = []
queue.run(fetch, progress=events.append)
```

## CLI

```bash
scrapequeue add urls.csv --checkpoint jobs.sqlite
scrapequeue status --checkpoint jobs.sqlite
scrapequeue run ./fetcher.py:fetch --checkpoint jobs.sqlite --retries 3 --rate-limit 1/s
scrapequeue run ./fetcher.py:fetch --checkpoint jobs.sqlite --only-failed --progress
scrapequeue export results.csv --checkpoint jobs.sqlite
scrapequeue export-failed failed.csv --checkpoint jobs.sqlite
scrapequeue retry-failed --checkpoint jobs.sqlite
```

`run` accepts a fetch callable as either:

```text
module:function
/path/to/file.py:function
```

## Public API

### `Queue(...)`

```python
Queue(
    checkpoint="jobs.sqlite",
    retries=3,
    rate_limit="1/s",
    backoff_seconds=0.25,
    backoff="linear",
)
```

### Methods

```python
queue.add_urls(urls)
queue.run(fetch, only_failed=False, progress=None)
queue.export_csv(path)
queue.export_failed(path)
queue.retry_failed()
queue.counts()
```

## Comparison

| Tool | Best for | Difference |
|---|---|---|
| Scrapy | full crawlers/spiders | much larger framework; scrapequeue is bounded URL queue only |
| Crawlee | browser/crawling automation | scrapequeue avoids crawler abstraction and external complexity |
| tenacity | retrying functions | scrapequeue adds URL persistence, resume, export, and state tracking |
| requests-cache | HTTP caching | scrapequeue is queue/checkpoint oriented, not response caching |

## Examples

See:

```text
examples/basic.py
examples/httpx_fetch.py
examples/playwright_fetch.py
examples/urls.csv
```

## Development

```bash
pytest -q
ruff check .
ruff format --check .
python -m build
twine check dist/*
```

## Non-goals

- HTML parsing
- browser automation abstraction
- distributed queues
- proxy rotation / anti-bot evasion
- crawling/link following

## License

MIT
