Metadata-Version: 2.4
Name: advanced-keyword-scraper
Version: 1.9.4
Summary: Advanced keyword research, SERP analysis, clustering, trends, and SEO intelligence toolkit
License: MIT
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: aiohttp<4,>=3.9
Requires-Dist: rich<15,>=13.0
Requires-Dist: pandas<3,>=2.0
Requires-Dist: openpyxl<4,>=3.1
Requires-Dist: requests<3,>=2.31
Requires-Dist: beautifulsoup4<5,>=4.12

# Advanced Keyword Scraper v1.9.2

**Google Request Gateway + Reliability Upgrade.** Every Google request (SERP *and* suggestions) goes
through one gateway: cache-first, in-flight dedup, <=2 concurrent, global pacing, one total budget,
global 429 cooldown, circuit breaker, and a CAPTCHA hard stop. It reduces traffic and stops on abuse
signals; it never tries to solve or bypass a CAPTCHA.

```cmd
keyword-scraper "cotton kurti" --mode research --serp-limit 20 --google-max-requests 100 --debug
```

Exit codes: `0` ok, `1` error, `3` stopped by Google CAPTCHA/unusual-traffic page (partial results are saved when any exist).

Useful flags: `--google-min-interval`, `--google-jitter`, `--google-max-requests` (TOTAL budget),
`--google-concurrency` (hard-capped at 2), `--google-max-attempts`, `--google-initial-cooldown`,
`--google-max-cooldown`, `--google-breaker-threshold`, `--google-cache`, `--google-cache-ttl-hours`, `--debug`.

Run tests (no pytest needed):

```cmd
python -m unittest discover -s tests -t . -v
```

The end-to-end pipeline test needs the normal dependencies (`aiohttp`, `rich`); it is skipped otherwise.

---

# Advanced Keyword Scraper v1.8.0

Google SERP safety/persistence upgrade with conservative global pacing, persistent SQLite SERP cache, duplicate suppression, progressive 429 cooldown, CAPTCHA/unusual-traffic stop, Google circuit breaker, per-run request budget, and console telemetry.

This safety layer is designed to reduce request volume and stop on provider abuse signals; it is not an anti-bot bypass mechanism.

Example:
```bash
keyword-scraper "cotton kurti" --mode serp --threads 2 --google-min-interval 3 --google-jitter 1.5 --google-max-requests 100
```

SERP cache defaults to `<output>/google_serp_cache.sqlite3` and is valid for 72 hours.

# Advanced Keyword Scraper v1.7.0

A local-first keyword research CLI with multi-source discovery, heuristic intelligence, optional SERP analysis, resilient HTTP handling, and export workflows.

## Requirements

- Python 3.10 or newer
- Windows, macOS, or Linux

## Install

Recommended on Windows:

```cmd
py -m pip install --upgrade pip
py -m pip install -e .
```

If `py` is unavailable, use `python` instead.

## Verify the package

```cmd
python -m unittest discover -s tests -t . -v
```

The repository includes a `tests/__init__.py` bootstrap so tests also work directly from this source-layout checkout before an editable install.

## Run keyword research

```cmd
keyword-scraper "saree" "kurti" --depth 2 --format all
```

Or without installing the console script:

```cmd
python -m keyword_scraper.cli "saree" "kurti" --depth 2 --format all
```

## Run SERP research

```cmd
keyword-scraper "saree" --mode serp --format excel
```

## Run full keyword intelligence research

The `research` mode combines the existing multi-source discovery engine with
optional SERP/PAA enrichment, related searches, local-first keyword clustering,
transparent opportunity scoring, and optional provider metrics.

```cmd
keyword-scraper "cotton kurti" --mode research --depth 2 --serp-limit 20 --format excel
```

Optional competitor-gap analysis:

```cmd
keyword-scraper "cotton kurti" --mode research --serp-limit 30 ^
  --target-domain example.com --competitor-domain competitor.com
```

Optional metrics hydration (no metrics are invented):

```cmd
keyword-scraper "cotton kurti" --mode research --metrics-file metrics.csv
```

The metrics CSV can contain:

```text
keyword,search_volume,kd,cpc,competition,competition_level
cotton kurti,12000,28,0.42,0.31,LOW
```

### Research report sheets

The Excel report can include:

- All Keywords
- Opportunity Shortlist
- KPI Summary
- Keyword Clusters
- PAA Questions
- Related Searches
- SERP Analysis
- Competitor Gap
- Intent Summary / n-grams / topic summaries

Opportunity Score is a transparent heuristic. Search volume, KD and CPC are
only used when supplied by a provider metrics CSV; the package never fabricates
those values.

## Workflow modes

These modes create local action-oriented CSV plans from the supplied keywords; they do not claim exact search volume or guaranteed rankings.

```cmd
keyword-scraper "budget smartphone" --mode youtube --mode-output youtube.csv
keyword-scraper "wireless earbuds" --mode ecommerce --mode-output ecommerce.csv
keyword-scraper "best accounting software" --mode seo --mode-output seo.csv
keyword-scraper "project management software" --mode copywriting --mode-output copywriting.csv
```

## Notes

- SERP providers may change markup, throttle requests, or restrict automated access.
- Opportunity scores are heuristics, not search-engine ranking scores.
- SERP/PAA extraction depends on changing Google markup and may return partial results.
- Clustering is deterministic lexical/intent clustering, not a claim of semantic ML understanding.
- The project does not attempt to bypass provider controls.
- The project does not attempt to bypass provider controls.


### Google gateway hardening (v1.9.2)
- Exactly one half-open circuit probe is admitted after breaker cooldown; concurrent probes are blocked.
- Google transport uses a stable per-run User-Agent and does not rotate proxies.
- Proxy rotation, if configured for other providers, is not passed into the Google gateway.
- CAPTCHA/429 remains a hard-stop/cooldown signal; the gateway never changes proxies or User-Agent to retry around it.
