Metadata-Version: 2.4
Name: advanced-keyword-scraper
Version: 1.9.6
Summary: Advanced keyword research, SERP analysis, clustering, trends, and SEO intelligence toolkit
License: MIT
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: aiohttp<4,>=3.9
Requires-Dist: rich<15,>=13.0
Requires-Dist: pandas<3,>=2.0
Requires-Dist: openpyxl<4,>=3.1
Requires-Dist: requests<3,>=2.31
Requires-Dist: beautifulsoup4<5,>=4.12

# Advanced Keyword Scraper v1.9.6

## Multi-source SEO / Keyword Intelligence

Version 1.9.6 restores, hardens, and strengthens the multi-source workflow: **Google, YouTube Creator suggestions, Bing and Amazon autocomplete** can be queried individually or together. Google/YouTube traffic remains behind the safety gateway; Bing/Amazon use their public suggestion endpoints and do not bypass access controls.

### New in 1.9.6
- `--engine all` for Google + YouTube + Bing + Amazon in one keyword run.
- `--engines google,youtube,bing,amazon` for explicit source selection.
- Source-aware deduplication: the report keeps a combined `Source` field when the same keyword appears across providers.
- Expanded modifier discovery: geo/local, comparison, content and buyer-intent patterns with `--include-modifiers`.
- `--min-words` / `--max-words` keyword-length filtering.
- `--content-brief` adds recommended format, title angle and content angle.
- Research workbook now includes `Source Coverage`.
- More tolerant parsing for Bing OSJSON and Amazon completion payloads.
- Version metadata, CLI output, changelog, and package documentation are synchronized to 1.9.6.

Example:
```bash
keyword-scraper "cotton kurti" --engine all --depth 2 --include-modifiers --content-brief --format excel
```

For only the YouTube creator workflow:
```bash
keyword-scraper "cotton kurti" --engine youtube --depth 2
```



Google SERP safety/persistence upgrade with conservative global pacing, persistent SQLite SERP cache, duplicate suppression, progressive 429 cooldown, CAPTCHA/unusual-traffic stop, Google circuit breaker, per-run request budget, and console telemetry.

This safety layer is designed to reduce request volume and stop on provider abuse signals; it is not an anti-bot bypass mechanism.

Example:
```bash
keyword-scraper "cotton kurti" --mode serp --threads 2 --google-min-interval 3 --google-jitter 1.5 --google-max-requests 100
```

SERP cache defaults to `<output>/google_serp_cache.sqlite3` and is valid for 72 hours.

## v1.9.6 hardening

- Multi-source aggregation preserves Google, YouTube, Bing, and Amazon provenance for duplicate keywords.
- Source Count and per-provider Found flags are included in keyword outputs.
- Research Opportunity Score uses the observed source count instead of a fixed value.
- Depth-2 expansion preserves provider lineage.
- YouTube Creator and Amazon buyer-intent intelligence are included when their modes are selected.
- Non-Google retry routing is provider-aware: Bing/Amazon requests use their own provider identity when selecting retry/throttling delays. Failed Bing/Amazon responses are not cached as empty results.
- Regression coverage includes multi-provider integration, depth-2 lineage, provider-specific retry routing, export schema, and version consistency.

## Historical baseline: v1.7.0

A local-first keyword research CLI with multi-source discovery, heuristic intelligence, optional SERP analysis, resilient HTTP handling, and export workflows.

## Requirements

- Python 3.10 or newer
- Windows, macOS, or Linux

## Install

Recommended on Windows:

```cmd
py -m pip install --upgrade pip
py -m pip install -e .
```

If `py` is unavailable, use `python` instead.

## Verify the package

```cmd
python -m unittest discover -s tests -t . -v
```

The repository includes a `tests/__init__.py` bootstrap so tests also work directly from this source-layout checkout before an editable install.

## Run keyword research

```cmd
keyword-scraper "saree" "kurti" --depth 2 --format all
```

Or without installing the console script:

```cmd
python -m keyword_scraper.cli "saree" "kurti" --depth 2 --format all
```

## Run SERP research

```cmd
keyword-scraper "saree" --mode serp --format excel
```

## Run full keyword intelligence research

The `research` mode combines the existing multi-source discovery engine with
optional SERP/PAA enrichment, related searches, local-first keyword clustering,
transparent opportunity scoring, and optional provider metrics.

```cmd
keyword-scraper "cotton kurti" --mode research --depth 2 --serp-limit 20 --format excel
```

Optional competitor-gap analysis:

```cmd
keyword-scraper "cotton kurti" --mode research --serp-limit 30 ^
  --target-domain example.com --competitor-domain competitor.com
```

Optional metrics hydration (no metrics are invented):

```cmd
keyword-scraper "cotton kurti" --mode research --metrics-file metrics.csv
```

The metrics CSV can contain:

```text
keyword,search_volume,kd,cpc,competition,competition_level
cotton kurti,12000,28,0.42,0.31,LOW
```

### Research report sheets

The Excel report can include:

- All Keywords
- Opportunity Shortlist
- KPI Summary
- Keyword Clusters
- PAA Questions
- Related Searches
- SERP Analysis
- Competitor Gap
- Intent Summary / n-grams / topic summaries

Opportunity Score is a transparent heuristic. Search volume, KD and CPC are
only used when supplied by a provider metrics CSV; the package never fabricates
those values.

## Workflow modes

These modes create local action-oriented CSV plans from the supplied keywords; they do not claim exact search volume or guaranteed rankings.

```cmd
keyword-scraper "budget smartphone" --mode youtube --mode-output youtube.csv
keyword-scraper "wireless earbuds" --mode ecommerce --mode-output ecommerce.csv
keyword-scraper "best accounting software" --mode seo --mode-output seo.csv
keyword-scraper "project management software" --mode copywriting --mode-output copywriting.csv
```

## Notes

- SERP providers may change markup, throttle requests, or restrict automated access.
- Opportunity scores are heuristics, not search-engine ranking scores.
- SERP/PAA extraction depends on changing Google markup and may return partial results.
- Clustering is deterministic lexical/intent clustering, not a claim of semantic ML understanding.
- The project does not attempt to bypass provider controls.
- The project does not attempt to bypass provider controls.


### Google gateway hardening (v1.9.2)
- Exactly one half-open circuit probe is admitted after breaker cooldown; concurrent probes are blocked.
- Google transport uses a stable per-run User-Agent and does not rotate proxies.
- Proxy rotation, if configured for other providers, is not passed into the Google gateway.
- CAPTCHA/429 remains a hard-stop/cooldown signal; the gateway never changes proxies or User-Agent to retry around it.

### Multi-source intelligence (v1.9.6)
Keyword output now preserves provider provenance. Duplicate keywords discovered by multiple sources are merged into one row with `Source Count`, `Google Found`, `YouTube Found`, `Bing Found`, `Amazon Found`, and `Cross-Source Signal` fields. Research opportunity scoring uses the actual source count when available.

YouTube workflow output includes creator opportunity and search-intent fields. Ecommerce workflow output includes buyer intent and product-attribute angles.
