Metadata-Version: 2.4
Name: advanced-keyword-scraper
Version: 1.9.11
Summary: Advanced keyword research, SERP analysis, clustering, trends, and SEO intelligence toolkit
License: MIT
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: aiohttp<4,>=3.9
Requires-Dist: rich<15,>=13.0
Requires-Dist: pandas<3,>=2.0
Requires-Dist: openpyxl<4,>=3.1
Requires-Dist: beautifulsoup4<5,>=4.12
Requires-Dist: scikit-learn<2,>=1.5
Requires-Dist: rapidfuzz<4,>=3.9
Requires-Dist: httpx<1,>=0.27
Requires-Dist: lxml<7,>=5
Requires-Dist: fastapi<1,>=0.115
Requires-Dist: pydantic<3,>=2.7
Requires-Dist: sqlalchemy<3,>=2.0
Requires-Dist: diskcache<6,>=5.6
Requires-Dist: uvicorn[standard]<1,>=0.30

# Advanced Keyword Scraper v1.9.11

## v1.9.11 fixes

- Research mode now reserves Google request budget for SERP/PAA enrichment instead of allowing keyword discovery to consume the full budget first.
- Topic clustering now uses TF-IDF + cosine similarity with representative phrase labels instead of first-token labels.
- Opportunity scoring now has better differentiation from lexical specificity, intent, SERP exact-title prevalence and optional provider metrics.
- Flipkart/Meesho generic product queries are classified as **Product Discovery** rather than automatically as Commercial Research.
- CSV workflow exports use standard UTF-8 without a BOM-prefixed header artifact.

## Security & stability hardening (introduced in v1.9.8)

- SSRF protection and redirect validation for SEO audits.
- 2 MB HTML response cap and HTML content-type validation.
- API key authentication, local-only fallback, and in-process rate limiting.
- Bounded TF-IDF similarity/clustering to prevent O(n²) resource exhaustion.
- Project-aware keyword persistence with duplicate protection on fresh databases.
- Improved intent fallback (`unknown` instead of assuming navigational).
- robots.txt and sitemap.xml checks in SEO audits.

## Multi-source SEO / Keyword Intelligence

Version 1.9.11 includes the hardened multi-source workflow: **Google, YouTube Creator suggestions, Bing and Amazon autocomplete** can be queried individually or together. Google/YouTube traffic remains behind the safety gateway; Bing/Amazon use their public suggestion endpoints and do not bypass access controls.

### Multi-source features
- `--engine all` for Google + YouTube + Bing + Amazon in one keyword run.
- `--engines google,youtube,bing,amazon` for explicit source selection.
- Source-aware deduplication: the report keeps a combined `Source` field when the same keyword appears across providers.
- Expanded modifier discovery: geo/local, comparison, content and buyer-intent patterns with `--include-modifiers`.
- `--min-words` / `--max-words` keyword-length filtering.
- `--content-brief` adds recommended format, title angle and content angle.
- Research workbook now includes `Source Coverage`.
- More tolerant parsing for Bing OSJSON and Amazon completion payloads.
- Version metadata, CLI output, changelog, and package documentation are synchronized to 1.9.11.

Example:
```bash
keyword-scraper "cotton kurti" --engine all --depth 2 --include-modifiers --content-brief --format excel
```

For only the YouTube creator workflow:
```bash
keyword-scraper "cotton kurti" --engine youtube --depth 2
```



Google SERP safety/persistence upgrade with conservative global pacing, persistent SQLite SERP cache, duplicate suppression, progressive 429 cooldown, CAPTCHA/unusual-traffic stop, Google circuit breaker, per-run request budget, and console telemetry.

This safety layer is designed to reduce request volume and stop on provider abuse signals; it is not an anti-bot bypass mechanism.

Example:
```bash
keyword-scraper "cotton kurti" --mode serp --threads 2 --google-min-interval 3 --google-jitter 1.5 --google-max-requests 100
```

SERP cache defaults to `<output>/google_serp_cache.sqlite3` and is valid for 72 hours.

## Historical multi-source hardening (v1.9.6)

- Multi-source aggregation preserves Google, YouTube, Bing, and Amazon provenance for duplicate keywords.
- Source Count and per-provider Found flags are included in keyword outputs.
- Research Opportunity Score uses the observed source count instead of a fixed value.
- Depth-2 expansion preserves provider lineage.
- YouTube Creator and Amazon buyer-intent intelligence are included when their modes are selected.
- Non-Google retry routing is provider-aware: Bing/Amazon requests use their own provider identity when selecting retry/throttling delays. Failed Bing/Amazon responses are not cached as empty results.
- Regression coverage includes multi-provider integration, depth-2 lineage, provider-specific retry routing, export schema, and version consistency.

## Historical baseline: v1.7.0

A local-first keyword research CLI with multi-source discovery, heuristic intelligence, optional SERP analysis, resilient HTTP handling, and export workflows.

## Requirements

- Python 3.10 or newer
- Windows, macOS, or Linux

## Install

Recommended on Windows:

```cmd
py -m pip install --upgrade pip
py -m pip install -e .
```

If `py` is unavailable, use `python` instead.

## Verify the package

```cmd
python -m unittest discover -s tests -t . -v
```

The repository includes a `tests/__init__.py` bootstrap so tests also work directly from this source-layout checkout before an editable install.

## Run keyword research

```cmd
keyword-scraper "saree" "kurti" --depth 2 --format all
```

Or without installing the console script:

```cmd
python -m keyword_scraper.cli "saree" "kurti" --depth 2 --format all
```

## Run SERP research

```cmd
keyword-scraper "saree" --mode serp --format excel
```

## Run full keyword intelligence research

The `research` mode combines the existing multi-source discovery engine with
optional SERP/PAA enrichment, related searches, local-first keyword clustering,
transparent opportunity scoring, and optional provider metrics.

```cmd
keyword-scraper "cotton kurti" --mode research --depth 2 --serp-limit 20 --format excel
```

Optional competitor-gap analysis:

```cmd
keyword-scraper "cotton kurti" --mode research --serp-limit 30 ^
  --target-domain example.com --competitor-domain competitor.com
```

Optional metrics hydration (no metrics are invented):

```cmd
keyword-scraper "cotton kurti" --mode research --metrics-file metrics.csv
```

The metrics CSV can contain:

```text
keyword,search_volume,kd,cpc,competition,competition_level
cotton kurti,12000,28,0.42,0.31,LOW
```

### Research report sheets

The Excel report can include:

- All Keywords
- Opportunity Shortlist
- KPI Summary
- Keyword Clusters
- PAA Questions
- Related Searches
- SERP Analysis
- Competitor Gap
- Intent Summary / n-grams / topic summaries

Opportunity Score is a transparent heuristic. Search volume, KD and CPC are
only used when supplied by a provider metrics CSV; the package never fabricates
those values.

## Workflow modes

These modes create local action-oriented CSV plans from the supplied keywords; they do not claim exact search volume or guaranteed rankings.

```cmd
keyword-scraper "budget smartphone" --mode youtube --mode-output youtube.csv
keyword-scraper "wireless earbuds" --mode ecommerce --mode-output ecommerce.csv
keyword-scraper "best accounting software" --mode seo --mode-output seo.csv
keyword-scraper "project management software" --mode copywriting --mode-output copywriting.csv
```

## Notes

- SERP providers may change markup, throttle requests, or restrict automated access.
- Opportunity scores are heuristics, not search-engine ranking scores.
- SERP/PAA extraction depends on changing Google markup and may return partial results.
- Clustering is deterministic lexical/intent clustering, not a claim of semantic ML understanding.
- The project does not attempt to bypass provider controls.
- The project does not attempt to bypass provider controls.


### Google gateway hardening (v1.9.2)
- Exactly one half-open circuit probe is admitted after breaker cooldown; concurrent probes are blocked.
- Google transport uses a stable per-run User-Agent and does not rotate proxies.
- Proxy rotation, if configured for other providers, is not passed into the Google gateway.
- CAPTCHA/429 remains a hard-stop/cooldown signal; the gateway never changes proxies or User-Agent to retry around it.

### Multi-source intelligence
Keyword output now preserves provider provenance. Duplicate keywords discovered by multiple sources are merged into one row with `Source Count`, `Google Found`, `YouTube Found`, `Bing Found`, `Amazon Found`, and `Cross-Source Signal` fields. Research opportunity scoring uses the actual source count when available.

YouTube workflow output includes creator opportunity and search-intent fields. Ecommerce workflow output includes buyer intent and product-attribute angles.


## AI-free SEO Intelligence Stack

v1.9.11 includes an optional-but-packaged intelligence/API layer with no external AI provider:

- **scikit-learn** — TF-IDF, cosine similarity and deterministic clustering
- **RapidFuzz** — fuzzy/near-duplicate keyword matching
- **httpx** — modern HTTP client for SEO audits and integrations
- **BeautifulSoup4 + lxml** — fast HTML parsing and on-page SEO analysis
- **pandas** — keyword analytics and report dataframes
- **FastAPI + Pydantic** — typed REST API
- **SQLAlchemy** — persistent projects/keyword storage
- **diskcache** — local result caching
- **Uvicorn** — local API server

### Start the API

```bash
keyword-scraper-api
```

or:

```bash
python -m keyword_scraper.api
```

Default: `http://127.0.0.1:8000`

Useful endpoints:

- `GET /health`
- `POST /keywords/analyze`
- `POST /keywords/clusters`
- `POST /keywords/similarity`
- `POST /keywords/matches`
- `POST /keywords/score`
- `POST /seo/audit`
- `POST /projects`
- `GET /keywords/recent`

Swagger documentation is available at `/docs` when the API server is running.

The intelligence layer is deliberately algorithm-first: no OpenAI/Gemini/Claude API key is required.

### Marketplace Seller Layers (v1.9.11)

Two local-first marketplace planning layers are included:

```bash
keyword-scraper "cotton nighty" --mode flipkart --mode-output flipkart.csv
keyword-scraper "cotton kurti" --mode meesho --mode-output meesho.csv
```

These layers generate buyer intent, listing priority, title angle, product attributes, bullet angle, catalog action and platform-specific research notes. They are **research/planning layers**, not official Flipkart/Meesho search-volume APIs and do not bypass authentication, CAPTCHAs or access controls.
