Metadata-Version: 2.4
Name: geo-crawler
Version: 0.5.19
Summary: AI-powered geographic information crawler with multi-model support
Author: GEO Crawler Team
Requires-Python: <3.14,>=3.11
Description-Content-Type: text/markdown
Requires-Dist: beautifulsoup4>=4.0.0
Requires-Dist: browser-use-volcengine>=0.13.7
Requires-Dist: markdownify>=1.0.0
Requires-Dist: openai>=2.0.0
Requires-Dist: openpyxl>=3.0.0
Requires-Dist: pandas>=2.0.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: python-socks[asyncio]>=2.0.0
Requires-Dist: rich>=14.0.0
Requires-Dist: typer>=0.20.0
Requires-Dist: volcengine-python-sdk==5.0.30
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.21.0; extra == "dev"
Requires-Dist: hypothesis>=6.0.0; extra == "dev"
Requires-Dist: ruff>=0.8.0; extra == "dev"

# GEO Crawler

[中文文档](README.zh-CN.md)

GEO Crawler is an installable Python CLI for running standardized sampling on AI platforms such as Kimi, Doubao, Yuanbao, DeepSeek, and Qianwen. It saves auditable raw outputs and includes the data models, judges, and metrics used for brand evaluation.

## Recommended Runtime

For regular users, `pipx` is the recommended way to run the CLI. It installs the command into an isolated virtual environment, so you can run `geo-crawler` from any project folder without cloning this repository or managing dependencies manually.

```bash
python3 -m pip install --user pipx
python3 -m pipx ensurepath

pipx install geo-crawler
geo-crawler --help
```

Upgrade an installed version:

```bash
pipx upgrade geo-crawler
```

Install from Git before a version is published to PyPI:

```bash
pipx install git+https://github.com/fengclient/geo-crawler.git
```

## Quick Start

For a step-by-step walkthrough of the full CLI loop, see [CLI_WALKTHROUGH.md](CLI_WALKTHROUGH.md). The Chinese version is [CLI_WALKTHROUGH.zh-CN.md](CLI_WALKTHROUGH.zh-CN.md).

### 1. Run Setup

Sampling needs an OpenAI-compatible model for the browser controller and a browser runtime. Run setup once in the project folder where you want `.geo-auth/` and `runs/` to live.

```bash
geo-crawler setup
```

`setup` writes a private `.env.geo-crawler`, creates `.geo-auth/`, and adds both to `.gitignore`. Later runtime commands discover `.env.geo-crawler` automatically. The generic `.env` file is still ignored by default.

### 2. Initialize Platform Auth

Before the first run, prepare a browser-use/Playwright `storage_state` for each target platform. The default auth directory is `.geo-auth/` under the current working directory, so no extra flag is usually needed.

Interactive initialization:

```bash
geo-crawler auth init kimi
geo-crawler doctor auth kimi
```

Initialize every supported platform:

```bash
geo-crawler auth init all
geo-crawler doctor auth all
```

Import an existing storage state in CI, on a remote machine, or when you already have the file:

```bash
geo-crawler auth import kimi ./kimi_storage_state.json
geo-crawler doctor auth kimi
```

`auth import` overwrites the same platform file in the default auth directory. For example, `geo-crawler auth import kimi ./kimi_storage_state.json` writes `.geo-auth/kimi_storage_state.json`. It affects only `kimi`, not other platforms.

`auth status` and `doctor auth` are fast offline checks of the saved file structure. `auth init` additionally opens the real platform page and requires a positive server-side session signal. Sampling repeats the live probe before starting the browser agent, but rejects only an explicit server-side logout with `error_category=login_required`; an inconclusive probe continues into sampling so the platform workflow can decide whether anonymous use is possible. A failed retained Gem Browser session keeps its primary error category, reports cleanup failures separately, and skips storage-state persistence while detaching the local CDP client.

Remote Gem Browser initialization:

```bash
geo-crawler setup --preset volcengine-gem --discover
geo-crawler auth init kimi \
  --browser remote
```

Remote auth init uses Gem Browser managed lifecycle: the CLI creates or obtains a remote browser session, shows the viewer URL for QR code or phone verification login, exports `.geo-auth/`, and later reuses the same provider family for sampling. `--yes` only skips CLI confirmation prompts; it does not bypass platform login.

For an external agent or wrapper that needs to show the viewer URL while the command is still waiting, use the opt-in NDJSON event stream:

```bash
geo-crawler auth init kimi \
  --browser remote \
  --events ndjson
```

Each stdout line is one JSON event. The first successful handoff event is `auth_session_created` with `viewer_url`; terminal events are `auth_completed` or `auth_failed`. A forced managed authentication always creates a fresh browser session and emits the handoff after platform navigation. It waits for a verified account name before exporting state; an initially indeterminate page remains in the login-waiting state instead of blocking the handoff. The first version supports one platform at a time and remote browser mode only. Do not store raw auth event streams in CI logs or shared artifacts, because `viewer_url` is a short-lived bearer login URL.

### 3. Run Sampling

Run commands from the project folder that owns your dataset, auth directory, and outputs. That keeps `.geo-auth/`, env files, and `runs/` in the same workspace.

Single query:

```bash
geo-crawler sample run \
  --platform kimi \
  --query "20万左右电动车推荐" \
  --mode quick \
  --repeat 1 \
  --out runs
```

Dataset:

```bash
geo-crawler sample run \
  --platform kimi \
  --dataset dataset.json \
  --mode quick \
  --repeat 3 \
  --out runs
```

Config file:

```bash
geo-crawler sample run --config sampling.json
```

After sampling, outputs are written to `runs/run-YYYYMMDD-HHMMSS-xxxxxx/`:

```text
runs/run-20260614-143025-a1b2c3/
├── manifest.json
├── inputs.json
├── outputs.json
├── summary.json
└── DONE
```

## Configuration Files

### Dataset

```json
[
  {
    "id": "q1",
    "query": "20万左右电动车推荐"
  },
  {
    "id": "q2",
    "query": "适合家庭使用的混动车有哪些"
  }
]
```

### Sampling Config

A config file is useful when platforms, inputs, and browser settings should be repeatable. The auth directory defaults to `.geo-auth/`; set `browser.profile_dir` or pass `--profile-dir` only when you need an override.

```json
{
  "platforms": ["kimi", "qianwen"],
  "inputs": [
    {
      "id": "q1",
      "query": "20万左右电动车推荐"
    }
  ],
  "mode": "quick",
  "repeat": 3,
  "out": "runs",
  "controller": {
    "model": "deepseek-v4-flash",
    "api_key_env": "OPENAI_API_KEY",
    "input_price_per_million": "1",
    "output_price_per_million": "2",
    "dont_force_structured_output": true,
    "reasoning_models": "deepseek-v4-flash",
    "reasoning_effort": "low"
  },
  "browser": {
    "cdp_url": "wss://example.com/browser",
    "vcpu_price_per_second": "0.0000405",
    "memory_gib_price_per_second": "0.0000102"
  },
  "usage": {
    "currency": "CNY"
  }
}
```

### Env File

`geo-crawler setup` manages `.env.geo-crawler` for the ordinary path. This file is project-local, private, and discovered automatically when `--env-file` is omitted. Do not commit `.geo-auth/`, storage state JSON files, or env files containing real tokens.

```bash
OPENAI_MODEL=deepseek-v4-flash
OPENAI_API_KEY=sk-...
OPENAI_API_BASE=https://ark.cn-beijing.volces.com/api/plan/v3
OPENAI_INPUT_PRICE_PER_MILLION=1
OPENAI_OUTPUT_PRICE_PER_MILLION=2
OPENAI_DONT_FORCE_STRUCTURED_OUTPUT=true
OPENAI_REASONING_MODELS=deepseek-v4-flash
OPENAI_REASONING_EFFORT=low
BROWSER_USE_LOGGING_LEVEL=warning
BROWSER_PROVIDER=local
```

For a low-cost controller that keeps forced `json_schema` structured output, switch only the controller model/prices:

```bash
OPENAI_MODEL=doubao-seed-2.0-mini
OPENAI_INPUT_PRICE_PER_MILLION=0.8
OPENAI_OUTPUT_PRICE_PER_MILLION=2
OPENAI_DONT_FORCE_STRUCTURED_OUTPUT=false
OPENAI_REASONING_MODELS=
OPENAI_REASONING_EFFORT=
```

Advanced users can still pass `--env-file <path>` to use a different environment file. The explicit file replaces `.env.geo-crawler` discovery for that command.

### Judge batch inference

Judge and Metric LLM settings use `JUDGE_METRIC_LLM_*` and are independent of the sampling controller. Set `JUDGE_METRIC_LLM_MODE=batch` to route each Judge request to `<JUDGE_METRIC_LLM_BASE_URL>/batch/chat/completions`; requests remain individually completed and checkpointed while Ark handles queueing and server-side batching. Keep the normal base URL without `/batch` and do not create a separate `.env.batch` file.

For Ark batch inference, Doubao Seed 2.0 Mini (`doubao-seed-2-0-mini-260428`) is the recommended foundation model. Create its batch inference endpoint, use the returned `ep-bi-...` ID as `JUDGE_METRIC_LLM_MODEL`, and configure `JUDGE_METRIC_LLM_TIMEOUT` to 24–72 hours in seconds. When external evaluation does not explicitly set `batch_size` and `max_concurrency`, batch mode uses `JUDGE_METRIC_LLM_BATCH_CONCURRENCY` (100 by default) for both. See the complete commented configuration in [`.env.example`](.env.example).

Remote CDP attach is for trusted private endpoints only:

```bash
geo-crawler setup --preset remote-cdp
```

Gem Browser remote runs support managed lifecycle only. To connect an existing browser, use the generic `remote-cdp` preset.

```bash
geo-crawler setup --preset volcengine-gem --discover
```

The legacy `BROWSER_PROVIDER=volcengine` and Gem attach configurations have been removed. Legacy settings fail before resource creation with a migration hint.

## Common Commands

```bash
# Show CLI help
geo-crawler --help
geo-crawler sample run --help

# Auth status
geo-crawler auth status all
geo-crawler doctor auth all

# Force re-login
geo-crawler auth init kimi --force

# Import from stdin
geo-crawler auth import kimi -

# Print JSON summary
geo-crawler sample run \
  --platform kimi \
  --query "20万左右电动车推荐" \
  --format json
```

## Machine-Readable JSON

CLI output is human-first by default, including when stdout is piped or running in CI. For scripts, CI, and AI agents, pass `--format json` explicitly. JSON mode writes a single JSON document to stdout with `schema_version` and `status`; progress, Rich tables, and human prompts stay out of stdout. `sample run` JSON includes artifact paths, counts, input source metadata, failed platforms, and safe recommended next actions when work remains.

`sample run` shows controller tokens, Gem managed `vCPU·s` / `GiB·s`, pricing status, and the total in its human summary, and writes the full statement to `usage.json` in the run directory. `evaluate run` and `evaluate external` independently count Judge tokens; they stay stdout-only unless `--usage-output PATH` is supplied. Controller prices use `OPENAI_*`, Gem prices use `VOLCENGINE_GEM_*`, and Judge prices use `JUDGE_METRIC_LLM_*`; only the shared currency uses `GEO_USAGE_CURRENCY`. Missing prices preserve raw usage as unpriced rather than treating it as zero.

`auth init --browser remote --events ndjson` and `sample run --events ndjson` are the exceptions to the single-document stdout rule. They write live events to stdout for long-running human handoff. `evaluate external --format json` keeps its final single document on stdout and emits flushed `evaluation-events/v1` NDJSON progress on stderr, so process wrappers must drain both streams concurrently.

```bash
geo-crawler auth status kimi --format json
geo-crawler doctor auth kimi --format json
geo-crawler doctor browser --format json
geo-crawler config inspect --format json
geo-crawler runs inspect runs/run-20260614-143025-a1b2c3 --format json
geo-crawler version --format json
geo-crawler sample run --platform kimi --query "20万左右电动车推荐" --format json
geo-crawler sample run --platform kimi --query "20万左右电动车推荐" --events ndjson
geo-crawler evaluate run runs/run-20260614-143025-a1b2c3 --context examples/evaluation_context.json --format json
geo-crawler evaluate external examples/evaluation_external_request.json --format json
geo-crawler evaluate external examples/evaluation_external_request.json --format json --state-dir runs/evaluation-001
```

External evaluation keeps the complete request JSON intact. The evaluator owns batching, bounded concurrency, transient retries, per-output atomic checkpoints, and ordered result merging. Repeating the same command with the same `--state-dir` resumes missing work without reevaluating successes; add `--retry-failed` only to retry records that exhausted their earlier attempts.

Use `config inspect` before a run to see effective controller, browser, and sampling config with source provenance. Use `doctor browser` for local, remote CDP, and Gem Browser readiness checks; live browser attachment only runs when `--probe` is passed. Use `runs inspect` after a run to summarize artifact health, platform counts, empty answers, diagnostics, and next actions without re-running sampling.

In sample event mode, stdout is NDJSON only: each line is one complete event with `schema_version`, `sequence`, `timestamp`, and `event`. Progress logs, Rich output, and browser-use logs go to stderr. Treat `viewer_url` as an authorized-operator-only bearer capability: it may show an authenticated third-party page and should not be posted to broad chat channels, tickets, CI logs, or shared artifacts. Manual actions in the viewer do not resume the current run or automatically retry failed inputs.

Long-running sample items emit a platform-neutral `sample_item_heartbeat` every 30 seconds. The heartbeat contains only the platform, input ID, mode, round, stage, and elapsed time; it deliberately excludes the original query and browser URLs. Each item has a 900-second total budget by default; set `execution.item_timeout_seconds` in a sampling config or `GEO_SAMPLE_ITEM_TIMEOUT_SECONDS` in an env file to change it. A budget expiry is persisted as `sample_item_timeout`, while browser-use uses fixed internal bounds for one LLM call (180 seconds), one Agent step (240 seconds), and consecutive step failures (2).

When a Gem Browser managed sampling run fails and the remote session can be retained, `sample run --format json` returns a short-lived `live_browser.viewer_url` and an `open_live_browser` recommended action. This is for inspecting the failed browser state only; manual actions in the viewer do not resume the current run or automatically retry failed inputs.

JSON output redacts secrets such as API keys, AK/SK, session tokens, cookies, storage state, and full CDP WebSocket URLs. Short-lived viewer URLs may appear only in commands that explicitly hand them to a user for login or failure inspection. Long-lived artifacts and `runs inspect` keep only redacted live-browser summaries; ordinary page URLs may appear unless credential-like query parameters need redaction.

## Auth State Management

The recommended auth directory is `.geo-auth/`. The legacy `browser_profiles/` path is still read for compatibility, but new users and automation should use `auth init` or `auth import`.

Supported platforms:

- `kimi`
- `doubao`
- `yuanbao`
- `deepseek`
- `qianwen`
- `all` for `auth init`, `auth status`, and `doctor auth`

Directory example:

```text
.geo-auth/
├── kimi_storage_state.json
├── doubao_storage_state.json
├── yuanbao_storage_state.json
├── deepseek_storage_state.json
└── qianwen_storage_state.json
```

## Developer Workflow

Clone the repository and use `uv` only for development, testing, or source changes. For regular operation, prefer `pipx`.

```bash
git clone https://github.com/fengclient/geo-crawler.git
cd geo-crawler

uv sync --all-extras --dev
uv run python -m pytest
uv run python -m geo_crawler.cli --help
```

In a development checkout, the same CLI can be run through the source entry point:

```bash
uv run python -m geo_crawler.cli sample run \
  --platform kimi \
  --query "20万左右电动车推荐"
```

Legacy `python examples/geo_cli.py ...` wrappers have been removed. Use `geo-crawler` or `uv run python -m geo_crawler.cli`.

## Architecture

The CLI has four main layers: command entry point, sampling configuration, platform query execution, and result storage. Auth state is managed by `src/auth_state.py`, and browser storage state is checked before sampling starts.

```text
geo_crawler.cli

src/sampling/
├── config.py
├── runner.py
└── storage.py

src/<platform>/
├── query.py
├── schema.py
├── tools.py
├── prompt_quick.md
└── prompt_deep.md

src/orchestrator/
├── base_judge.py
├── base_metric.py
├── judges/
└── metrics/
```

## Troubleshooting

### `geo-crawler: command not found`

Make sure `pipx ensurepath` has been run, then reopen your terminal.

```bash
python3 -m pipx ensurepath
```

### `doctor auth` reports missing or expired auth

Reinitialize interactively, or import a fresh storage state.

```bash
geo-crawler auth init kimi --force
geo-crawler doctor auth kimi
```

### `--env-file` does not take effect

Confirm the command explicitly passes `--env-file`, and that variable names match supported CLI configuration. `.env` is not loaded automatically.

### Does a remote browser still need `browser_profiles/`?

No. Remote browser workflows should still use `.geo-auth/<platform>_storage_state.json`. `browser_profiles/` is only a legacy compatibility path, not a dependency for the new flow.

### Check installed package metadata

```bash
pipx list
pipx runpip geo-crawler show geo-crawler
```

## Security Notes

`.geo-auth/` and storage state JSON files are browser login credentials. Treat them as secrets. Do not commit them, paste them into chats or tickets, or expose them in logs. In CI, prefer private files or controlled secret injection.

## License

MIT
