Metadata-Version: 2.4
Name: pulsecrawl
Version: 0.2.0
Summary: Local Python crawler and extraction SDK with portable workspaces, Parquet datasets, SQL, and scheduling.
Author: PulseCrawl contributors
License-Expression: AGPL-3.0-only
Project-URL: GitHub, https://github.com/pulsecrawl/PulseCrawl
Project-URL: Documentation, https://github.com/pulsecrawl/PulseCrawl#readme
Project-URL: Changelog, https://github.com/pulsecrawl/PulseCrawl/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/pulsecrawl/PulseCrawl/issues
Project-URL: Security, https://github.com/pulsecrawl/PulseCrawl/security/policy
Keywords: crawler,scraping,sdk,robots,parquet,duckdb
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: aiohttp<4,>=3.14.5
Requires-Dist: httpx<1,>=0.27
Requires-Dist: selectolax<0.5,>=0.3.10
Provides-Extra: sql
Requires-Dist: duckdb>=1.2; extra == "sql"
Requires-Dist: pytz>=2024.1; extra == "sql"
Provides-Extra: workspace
Requires-Dist: duckdb>=1.2; extra == "workspace"
Requires-Dist: pyarrow>=17.0; extra == "workspace"
Requires-Dist: croniter>=2.0; extra == "workspace"
Requires-Dist: pytz>=2024.1; extra == "workspace"
Requires-Dist: tzdata>=2024.1; sys_platform == "win32" and extra == "workspace"
Provides-Extra: cloud
Requires-Dist: keyring>=25.0; extra == "cloud"
Provides-Extra: dev
Requires-Dist: setuptools>=84.0.0; extra == "dev"
Requires-Dist: pytest>=9.0.3; extra == "dev"
Requires-Dist: pytest-asyncio>=1.3; extra == "dev"
Requires-Dist: pytest-cov>=5.0; extra == "dev"
Requires-Dist: duckdb>=1.2; extra == "dev"
Requires-Dist: pyarrow>=17.0; extra == "dev"
Requires-Dist: croniter>=2.0; extra == "dev"
Requires-Dist: pytz>=2024.1; extra == "dev"
Requires-Dist: tzdata>=2024.1; sys_platform == "win32" and extra == "dev"
Requires-Dist: ruff<1,>=0.14; extra == "dev"
Requires-Dist: keyring>=25.0; extra == "dev"
Dynamic: license-file

# PulseCrawl

**[GitHub repository](https://github.com/pulsecrawl/PulseCrawl)**

PulseCrawl is a local open-source Python crawler and structured extraction SDK.
One Python distribution supports in-memory crawling and durable local workspaces.
Run on your own machine with no account required. Requires Python 3.11 or newer.

## Installation

```bash
python -m pip install 'pulsecrawl[workspace]'
```

No repository clone is needed. For in-memory crawling only, use
`python -m pip install pulsecrawl`. The `workspace` extra
adds SQLite-backed execution state, Parquet datasets, embedded DuckDB queries,
and cron scheduling. No Docker, Redis, or database server is required.

Additional extras: `sql` for DuckDB-backed local results and `dev` for
contributors.

## Quickstart (no network)

After installing the workspace extra, save this as `quickstart.py` and run
`python quickstart.py` from any directory. Choose a new workspace path:

```python
from pulsecrawl import Workspace

with Workspace.init("./my-crawls") as pc:
    dataset = pc.datasets.create(
        "pages",
        fields=[{"fieldId": "title", "name": "title", "type": "string"}],
    )
    print(dataset["datasetId"])
    print(pc.validate()["valid"])
```

This creates local definitions only; it does not start a crawl. The installed
CLI can also initialize and validate a fresh workspace:

```bash
pulsecrawl init ./cli-crawls
pulsecrawl --workspace ./cli-crawls validate
```

See the [standalone package examples](https://github.com/pulsecrawl/PulseCrawl/tree/main/examples/installed_package)
for independently runnable workspace setup and explicitly opted-in live crawling.

## Features and boundaries

- CSS, JSON-LD, and metadata extraction with URL-scoped template stacks.
- Robots checks, request pacing, bounded responses, redirect authorization,
  and private-network blocking by default.
- Portable workspaces with dataset history, SQL, compaction, and backup/restore.
- Public Python and CLI interfaces over one hexagonal crawl engine.

Before a live crawl, review authorization, site terms, and robots policies and
identify your operator with a working contact in the user agent. Safeguards do
not grant permission to scrape. The private-network opt-in must never be used
with untrusted URLs or job definitions.

Start with the [guide and public API examples](https://github.com/pulsecrawl/PulseCrawl#readme)
and [offline examples](https://github.com/pulsecrawl/PulseCrawl/tree/main/examples).
The examples are available in the source repository and source distribution,
not as modules installed by the wheel.

## Project information

PulseCrawl is alpha software; APIs and workspace compatibility may evolve.
See the [support policy](https://github.com/pulsecrawl/PulseCrawl/blob/main/SUPPORT.md)
and [changelog](https://github.com/pulsecrawl/PulseCrawl/blob/main/CHANGELOG.md).
It is licensed under [AGPL-3.0-only](https://github.com/pulsecrawl/PulseCrawl/blob/main/LICENSE).
Report vulnerabilities using the [security policy](https://github.com/pulsecrawl/PulseCrawl/security/policy),
not a public issue.
