Metadata-Version: 2.5
Name: outcrawl
Version: 0.4.2
Summary: The Python client for Outcrawl: scrape, crawl, search and run agent browser tasks on one credit balance.
Project-URL: Homepage, https://outcrawl.ai
Project-URL: Repository, https://github.com/aeonmindai/outcrawl-browser
Project-URL: Issues, https://github.com/aeonmindai/outcrawl-browser/issues
License-Expression: MIT
License-File: LICENSE
Keywords: agent,browser-automation,crawler,outcrawl,scraping,stealth,web-search
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: httpx<1,>=0.27
Provides-Extra: dev
Requires-Dist: pytest-asyncio>=0.24; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Description-Content-Type: text/markdown

# outcrawl

The Python client for [Outcrawl](https://api.outcrawl.ai): scrape, crawl, search and run agent browser
tasks, on one credit balance. Pure Python — one `py3-none-any` wheel for every operating system,
and no compiler on any of them.

```sh
pip install outcrawl
```

Requires Python 3.10 or newer. One runtime dependency, [`httpx`](https://www.python-httpx.org/),
which is pure Python, as is everything it pulls.

## Usage

```python
import asyncio
from outcrawl import Outcrawl

async def main() -> None:
    async with Outcrawl() as oc:                       # reads OUTCRAWL_API_KEY
        doc = await oc.scrape("https://example.com", formats=["markdown"])
        print(doc.markdown)                            # or doc["markdown"] — it is the wire object
        print(doc.usage)                               # every call returns what it cost

        job = oc.agent(
            task="Report the title of the top story on Hacker News.",
            caps={"budget": "3.00"},                   # the one ceiling a submit must carry
        )
        async for record in job:                       # the event cursor, while it runs
            print(record["kind"], record.get("text"))
        run = await job                                # and the settled run
        print(run["status"], run.get("data"))          # no "data" when a cap stopped it first

asyncio.run(main())
```

`await` and `async for` on **one** job is one run and not two: they are separate requests against
one durable row, so watching a run and having its answer is not a choice.

## Configuration

| Variable | Meaning | Default |
| --- | --- | --- |
| `OUTCRAWL_API_KEY` | your key | required |
| `OUTCRAWL_API_URL` | base url | `https://api.outcrawl.ai` |

The same two names `@outcrawl/sdk`, the `outcrawl` CLI and the Outcrawl MCP server read. Pass
`Outcrawl(api_key=..., base_url=...)` to override them per client.

## Surface

Every one of the 37 capabilities in the registry is reachable, and `tests/test_registry.py` fails
when one is not.

- `oc.scrape(url, **options)` / `oc.crawl(url, **options)` / `oc.search(query, **options)` —
  `crawl` is async-iterable and single-pass. `search` returns `SearchHits`: when one search
  engine refuses the query and another answers, `hits.warnings` carries an `engines_refused`
  entry naming the engine that refused; when every engine refuses it raises
  `SearchRefusedError` (503, retryable after `retry_after_seconds`, a retry is served from
  another exit, nothing charged) rather than returning `[]`.
- Options are keyword arguments with the API's own names, sent as given; `help(oc.scrape)` and
  `help(oc.crawl)` list every one with its default. `maxAge` is how old a cached answer may be, in
  milliseconds, default `172800000` (2 days); `maxAge=0` is always live, neither reading nor
  writing the cache. A hit bills the same `page_load` credit a live fetch does.
- Every scrape result and crawl page says where it came from: `doc.cacheState` is `"hit"`,
  `"miss"` or `"bypass"`, `doc.cachedAt` is when a hit was captured, `doc.usage["cache"]["status"]`
  agrees with it (`HIT` / `MISS` / `DISABLED`), `doc.metadata["servedVia"]` is `"direct"` or
  `"residential"`, the network path that fetched it, and `doc.metadata["egressCountry"]` is the
  country (ISO code) of the address it was fetched from.
- A degraded page still comes back, and says so in `doc["warnings"]`: absent on a clean result,
  otherwise a list of `{"code", "format"?, "message"}` whose `message` names the numbers.
  `partial_document` — the page was read before its main document finished loading (the parser
  was still in it, or the response was still arriving), so `markdown`, `html` and `rawHtml` are a
  prefix of the page; retry with a larger `timeoutMs` or a `waitFor`. `empty_capture` — the
  requested `markdown` or `html` came back empty. `blocked_interstitial` — a bot-protection page
  came back instead of the page. `challenge_budget_exhausted` — a challenge left too little time
  to finish reading. `source_truncated` — a JSON or text body arrived cut off. `json_no_content`,
  `json_empty` and `json_required_empty` describe a `json` extraction: no page text, no data, or
  empty values for fields your schema requires. `egress_country_mismatch` — this destination cannot be reached through our residential
  network, so the page was fetched from our own address in `metadata["egressCountry"]` instead of
  the country you asked for; prices and availability may differ from what a visitor there sees.
  `search_challenge_unsolved` — a search engine's results page came back as its "verify you are
  human" check rather than results; use `oc.search(...)` (`POST /v1/search`) for search results.
- `oc.crawl.job(id)` — the same crawl by id, with `status()`, `results()`, `cancel()`, and
  `oc.crawl.get(id)` / `.results(id)` / `.cancel(id)` for an id on its own. A streaming crawl
  belongs to its connection, so hanging up cancels it: the row settles `cancelled` with every
  page it had already delivered, and these routes are how you read the pages you paid for. The
  id is on `handle.job["id"]` from the first progress frame — store it before you need it.
- `oc.agent(task=..., caps=...)` — `caps` is required at runtime and `caps.budget` is required
  inside it; `caps.steps` and `caps.duration` are optional and unset unless you name them, and
  all three are hard stops when present. The job is awaitable, async-iterable, and carries
  everything a handle does: `status()`, `results()`, `events()`, `cancel()`, `control()`,
  `answer()`, `add_file()`. `oc.agent.get(id)` and friends reach a run by id — an id from a
  webhook needs no submit. `schema` accepts a JSON Schema object and constrains the run's `data` to
  it, the same typed submit `@outcrawl/sdk` documents.
- `oc.profiles` / `oc.secrets` / `oc.rules` / `oc.integrations` / `oc.sessions` / `oc.monitors`.
  `oc.secrets` has deliberately **no `get`**: there is no route to widen. A run reaches a value
  only by naming its HANDLE in the submit, and the substitution happens below the model.
  `oc.secrets.intent(name=..., kind=...)` is for a value a person types into YOUR web page, so it
  never passes through your server: your app server calls it and gets `name`, `kind`, `url`,
  `token` and `expiresAt`; hand `url` and `token` to the page, which sends `PUT url` with
  `Authorization: Bearer <token>` and a body `{"value": "..."}`. The token is single use, lasts
  five minutes and writes `name` and nothing else (creating it, or replacing the value already
  stored under it). It is a credential: give it to that page only and never log it; the result's
  `repr` masks it. A refused `PUT` answers `secret_intent_invalid` (401, the token does not open
  that URL for that name) or `secret_intent_spent` (410, `details.reason` `used` or `expired`),
  rebuilt here as `SecretIntentInvalidError` / `SecretIntentSpentError`; nothing is stored either
  way and the fix is a new intent. Minting is capped per account per minute
  (`QuotaExceededError` with `ceiling == "secret_intents_per_minute"`). A value your server
  already holds goes through `oc.secrets.create`.
- `oc.integrations` — MCP servers a run may call outside the browser. Connecting is not granting:
  a run reaches a connector only when its submit names it (`oc.agent(integrations=["linear"])`).

  ```python
  verdict = await oc.integrations.test(
      url="https://mcp.linear.app/mcp", auth={"kind": "bearer", "secret": "linear_key"}
  )
  if not verdict["ok"]:
      print(verdict["reason"], verdict["message"])  # not_mcp | unauthorized | unreachable | missing_secret
  ```

  `test(url=, auth=)` checks a server and stores nothing; a check that ran always answers, and
  `ok` says how it went. `connect(...)` runs the same check first and raises
  `IntegrationCheckFailedError` (422, `.reason` from the same four) with nothing stored.
  `update(id, tools=..., auth={"secret": ...})` changes the tool pin or the secret handle in
  place; `tools=None` means every tool the server offers, and a new secret is checked the same
  way before it replaces the old one. `list()` rows carry `lastUsedAt`, when a run last called
  the connector successfully (`None` if never). Each call a run makes appears in its event log as
  a record with `kind: "integration"` whose `value` is `{id, name, tool, ok, ms}`, or `{id, name,
  status: "unreachable", reason, message}` for a granted connector the run could not use; both
  are at the default `step` level. No tool arguments or results are logged.
- `oc.usage(**query)` and `oc.credits()` — every credit figure is a decimal **string**; use
  `decimal.Decimal`, never `float()`.
- `CAPABILITY_AVAILABILITY` — whether anything is SERVED behind a declared route. A capability
  nothing serves is refused in your own process, with what is missing, rather than as a 503.

## Errors

Identical to the TypeScript SDK's: same classes, same fields, same sentences. Branch on
`error.code`, never on `str(error)`.

```python
from outcrawl import ProfileInUseError, QuotaExceededError

try:
    ...
except ProfileInUseError as leased:
    retry_at = leased.held_until          # a retry that knows when to retry
except QuotaExceededError as over:
    await asyncio.sleep(over.retry_after_seconds)
```

`tests/error_parity.json` is emitted from the TypeScript SDK's own error reconstruction and read by
both test suites, so neither surface can drift from the other without turning its own suite red.

## What is not here

`oc.browser()`. The live browser needs a CDP connection driven in the caller's own process, which
is why it is deliberately not a registry capability and why the TypeScript SDK is the one place it
exists. The same page capabilities are reachable through `agent` and `scrape`, which run the loop
on our side. That is a difference in kind, not an omission.

The same reason takes `Session.take_control` with it: the TypeScript SDK's `takeControl()` opens a
CDP connection to a running session's own page so a person can drive it directly, and that
connection has to live in the caller's process for the same reason `oc.browser()` does. Everything
else on a session — `get`, `list`, `export`, `live()` — is a row read or a held stream and is here.

## Development

```sh
pip install -e '.[dev]'                  # pytest + pytest-asyncio, never runtime dependencies
python3 scripts/gen_registry.py          # regenerate registry.py from packages/core/src/registry.ts
python3 scripts/gen_registry.py --check  # or just fail if it is stale
bun ../sdk-python/scripts/gen_error_parity.ts   # regenerate the shared error contract
python3 -m pytest tests -q               # the drift, parity and ergonomics tests
```

From the repo root, `npm run test:python` runs the staleness check and the suite together on
the SUPPORTED FLOOR — `uv run --extra dev --python 3.10 pytest` — so the gate exercises 3.10
rather than whatever `python3` happens to be. Without `uv`: `pip install -e '.[dev]'` and
`python3 -m pytest tests -q`.

The two registry-versus-TypeScript tests skip when run from an unpacked sdist, which has no
TypeScript to compare against, and run whenever the monorepo is present. The reachability test —
the one that fails when a registry row has no method — has no such dependency and always runs.
