Metadata-Version: 2.4
Name: klyrek-crawler
Version: 0.1.0
Summary: Website crawling and application mapping for the Klyrek ecosystem
Author: Klyrek Contributors
License: MIT
Keywords: appsec,crawler,pentesting,reconnaissance,security
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Information Technology
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Security
Requires-Python: >=3.10
Requires-Dist: klyrek-core
Requires-Dist: klyrek-http
Provides-Extra: dev
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Description-Content-Type: text/markdown

# klyrek-crawler

Breadth-first crawler that maps a target's pages, forms, and script references into a
`klyrek_core.models.ScanResult`. Every fetch goes through a `KlyrekHTTPClient`
(`klyrek-http`), so scope enforcement and rate limiting are inherited automatically — the
crawler additionally skips *enqueuing* out-of-scope links so it doesn't spend its page budget
chasing hosts it isn't authorized to touch.

```python
from klyrek_core.config import KlyrekConfig
from klyrek_core.models import Target
from klyrek_core.scope import AuthorizationScope
from klyrek_http.client import KlyrekHTTPClient
from klyrek_crawler.crawler import Crawler

scope = AuthorizationScope(authorized_hosts=["target.com", "*.target.com"])
target = Target(base_url="https://target.com/", hosts=["target.com"])

with KlyrekHTTPClient(scope, config=KlyrekConfig(rate_limit_per_host=2)) as client:
    crawler = Crawler(client, max_pages=200, max_depth=3)
    result = crawler.crawl(target)

print(len(result.endpoints), "endpoints discovered")
```

Endpoints are tagged by how they were found: `source="crawler"` for pages,
`"crawler-form"` for form actions (with field names captured as `params`), and
`"crawler-script"` for referenced JavaScript files (the raw material `klyrek-js` consumes).
