Metadata-Version: 2.4
Name: geo-crawler
Version: 0.2.3
Summary: AI-powered geographic information crawler with multi-model support
Author: GEO Crawler Team
Requires-Python: <3.14,>=3.11
Description-Content-Type: text/markdown
Requires-Dist: beautifulsoup4>=4.0.0
Requires-Dist: browser-use-volcengine>=0.12.6
Requires-Dist: markdownify>=1.0.0
Requires-Dist: openai>=2.0.0
Requires-Dist: openpyxl>=3.0.0
Requires-Dist: pandas>=2.0.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: rich>=14.0.0
Requires-Dist: typer>=0.20.0
Requires-Dist: volcengine-python-sdk>=4.0.34
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.21.0; extra == "dev"
Requires-Dist: hypothesis>=6.0.0; extra == "dev"

# GEO Crawler

[中文文档](README.zh-CN.md)

GEO Crawler is an installable Python CLI for running standardized sampling on AI platforms such as Kimi, Doubao, Yuanbao, DeepSeek, and Qianwen. It saves auditable raw outputs and includes the data models, judges, and metrics used for brand evaluation.

## Recommended Runtime

For regular users, `pipx` is the recommended way to run the CLI. It installs the command into an isolated virtual environment, so you can run `geo-crawler` from any project folder without cloning this repository or managing dependencies manually.

```bash
python3 -m pip install --user pipx
python3 -m pipx ensurepath

pipx install geo-crawler
geo-crawler --help
```

Upgrade an installed version:

```bash
pipx upgrade geo-crawler
```

Install from Git before a version is published to PyPI:

```bash
pipx install git+https://github.com/fengclient/geo-crawler.git
```

## Quick Start

### 1. Prepare Controller Model Environment

Sampling needs an OpenAI-compatible model for the browser controller. The simplest setup is to place a `.env` file, or a dedicated env file such as `.env.remote-sandbox`, in your working directory.

```bash
OPENAI_MODEL=gpt-4.1-mini
OPENAI_API_KEY=sk-...
OPENAI_API_BASE=https://api.openai.com/v1
BROWSER_USE_LOGGING_LEVEL=warning
```

`--env-file` is loaded only when explicitly passed. It is not an authentication mechanism and does not import browser login state.

### 2. Initialize Platform Auth

Before the first run, prepare a browser-use/Playwright `storage_state` for each target platform. The default auth directory is `.geo-auth/` under the current working directory, so no extra flag is usually needed.

Interactive initialization:

```bash
geo-crawler auth init kimi
geo-crawler doctor auth kimi
```

Initialize every supported platform:

```bash
geo-crawler auth init all
geo-crawler doctor auth all
```

Import an existing storage state in CI, on a remote machine, or when you already have the file:

```bash
geo-crawler auth import kimi ./kimi_storage_state.json
geo-crawler doctor auth kimi
```

`auth import` overwrites the same platform file in the default auth directory. For example, `geo-crawler auth import kimi ./kimi_storage_state.json` writes `.geo-auth/kimi_storage_state.json`. It affects only `kimi`, not other platforms.

### 3. Run Sampling

Run commands from the project folder that owns your dataset, auth directory, and outputs. That keeps `.geo-auth/`, env files, and `runs/` in the same workspace.

Single query:

```bash
geo-crawler sample run \
  --platform kimi \
  --query "20万左右电动车推荐" \
  --mode quick \
  --repeat 1 \
  --env-file .env.remote-sandbox \
  --out runs
```

Dataset:

```bash
geo-crawler sample run \
  --platform kimi \
  --dataset dataset.json \
  --mode quick \
  --repeat 3 \
  --env-file .env.remote-sandbox \
  --out runs
```

Config file:

```bash
geo-crawler sample run --config sampling.json
```

After sampling, outputs are written to `runs/run-YYYYMMDD-HHMMSS-xxxxxx/`:

```text
runs/run-20260614-143025-a1b2c3/
├── manifest.json
├── inputs.json
├── outputs.json
├── summary.json
└── DONE
```

## Configuration Files

### Dataset

```json
[
  {
    "id": "q1",
    "query": "20万左右电动车推荐"
  },
  {
    "id": "q2",
    "query": "适合家庭使用的混动车有哪些"
  }
]
```

### Sampling Config

A config file is useful when platforms, inputs, and browser settings should be repeatable. The auth directory defaults to `.geo-auth/`; set `browser.profile_dir` or pass `--profile-dir` only when you need an override.

```json
{
  "platforms": ["kimi", "qianwen"],
  "inputs": [
    {
      "id": "q1",
      "query": "20万左右电动车推荐"
    }
  ],
  "mode": "quick",
  "repeat": 3,
  "out": "runs",
  "controller": {
    "model": "gpt-4.1-mini",
    "api_key_env": "OPENAI_API_KEY"
  },
  "browser": {
    "cdp_url": "wss://example.com/browser"
  }
}
```

### Env File

An env file is suitable for controller model settings and remote browser parameters. Do not commit `.geo-auth/`, storage state JSON files, or env files containing real tokens.

```bash
OPENAI_MODEL=gpt-4.1-mini
OPENAI_API_KEY=sk-...
OPENAI_API_BASE=https://api.openai.com/v1
BROWSER_USE_LOGGING_LEVEL=warning
BROWSER_USE_CDP_URL=wss://example.com/browser
```

## Common Commands

```bash
# Show CLI help
geo-crawler --help
geo-crawler sample run --help

# Auth status
geo-crawler auth status all
geo-crawler doctor auth all

# Force re-login
geo-crawler auth init kimi --force

# Import from stdin
geo-crawler auth import kimi -

# Print JSON summary
geo-crawler sample run \
  --platform kimi \
  --query "20万左右电动车推荐" \
  --format json
```

## Auth State Management

The recommended auth directory is `.geo-auth/`. The legacy `browser_profiles/` path is still read for compatibility, but new users and automation should use `auth init` or `auth import`.

Supported platforms:

- `kimi`
- `doubao`
- `yuanbao`
- `deepseek`
- `qianwen`
- `all` for `auth init`, `auth status`, and `doctor auth`

Directory example:

```text
.geo-auth/
├── kimi_storage_state.json
├── doubao_storage_state.json
├── yuanbao_storage_state.json
├── deepseek_storage_state.json
└── qianwen_storage_state.json
```

## Developer Workflow

Clone the repository and use `uv` only for development, testing, or source changes. For regular operation, prefer `pipx`.

```bash
git clone https://github.com/fengclient/geo-crawler.git
cd geo-crawler

uv sync --all-extras --dev
uv run python -m pytest
uv run python -m geo_crawler.cli --help
```

In a development checkout, the same CLI can be run through the source entry point:

```bash
uv run python -m geo_crawler.cli sample run \
  --platform kimi \
  --query "20万左右电动车推荐"
```

The compatibility entry point `python examples/geo_cli.py ...` remains for old scripts, but it is no longer the recommended README path.

## Architecture

The CLI has four main layers: command entry point, sampling configuration, platform query execution, and result storage. Auth state is managed by `src/auth_state.py`, and browser storage state is checked before sampling starts.

```text
geo_crawler.cli
└── examples/geo_cli.py compatibility wrapper

src/sampling/
├── config.py
├── runner.py
└── storage.py

src/<platform>/
├── query.py
├── schema.py
├── tools.py
├── prompt_quick.md
└── prompt_deep.md

src/orchestrator/
├── base_judge.py
├── base_metric.py
├── judges/
└── metrics/
```

## Troubleshooting

### `geo-crawler: command not found`

Make sure `pipx ensurepath` has been run, then reopen your terminal.

```bash
python3 -m pipx ensurepath
```

### `doctor auth` reports missing or expired auth

Reinitialize interactively, or import a fresh storage state.

```bash
geo-crawler auth init kimi --force
geo-crawler doctor auth kimi
```

### `--env-file` does not take effect

Confirm the command explicitly passes `--env-file`, and that variable names match supported CLI configuration. `.env` is not loaded automatically.

### Does a remote browser still need `browser_profiles/`?

No. Remote browser workflows should still use `.geo-auth/<platform>_storage_state.json`. `browser_profiles/` is only a legacy compatibility path, not a dependency for the new flow.

### Check installed package metadata

```bash
pipx list
pipx runpip geo-crawler show geo-crawler
```

## Security Notes

`.geo-auth/` and storage state JSON files are browser login credentials. Treat them as secrets. Do not commit them, paste them into chats or tickets, or expose them in logs. In CI, prefer private files or controlled secret injection.

## License

MIT
