Global header Login Pricing

On this page

Overview Authentication Legacy crawler

Sidebar upgrade promo

Agent Session Recovery

Session recovery lets an agent resume a crawl after a process restart without duplicating completed pages.

The recovery token stores the queue cursor, visited URLs, retry attempts, and the latest durable checkpoint.

from agentcrawl import AgentCrawl
crawler = AgentCrawl({"fetcher": "http"})
run = crawler.crawl("https://example.com/docs", max_pages=10)

Operational guarantees

Operators can inspect checkpoint age, pending URLs, and terminal failures before allowing a resumed run.

FieldMeaning
visitedCompleted URLs
pendingQueued URLs