Metadata-Version: 2.4
Name: stealth-google-search
Version: 0.1.0
Summary: Stealthy Google Search and AI Overview scraper with anti-bot evasion and automatic redirect resolution.
Author-email: Your Name <your.email@example.com>
License: MIT
Project-URL: Homepage, https://github.com/SeMoazamAli/google-search
Project-URL: Bug Tracker, https://github.com/SeMoazamAli/google-search/issues
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: undetected-chromedriver>=3.5.5
Requires-Dist: selenium>=4.20.0
Requires-Dist: parsel>=1.9.0
Requires-Dist: requests>=2.31.0
Requires-Dist: setuptools>=65.0.0
Dynamic: license-file

# Google Search & AI Overview Scraper

A robust, stealthy Python scraper for Google Search results. Built using `undetected-chromedriver` and `parsel`, this tool automatically bypasses Google's bot detection, extracts **Google AI Overviews**, scrapes clean organic search results (titles, genuine snippets, provider domains, dates), and automatically resolves Google's `/goto?url=...` encrypted redirect links into their **original destination URLs**.

---

## Table of Contents

- [Features](#features)
- [Prerequisites](#prerequisites)
- [Installation](#installation)
- [Quick Start](#quick-start)
- [Detailed Parameter Reference](#detailed-parameter-reference)
- [Response Structure & Schema](#response-structure--schema)
- [Advanced Usage Examples](#advanced-usage-examples)
  - [1. Scraping Multiple Pages](#1-scraping-multiple-pages)
  - [2. Extracting AI Overview Answers](#2-extracting-ai-overview-answers)
  - [3. Running in Visible (Non-Headless) Mode](#3-running-in-visible-non-headless-mode)
  - [4. Ultra-Fast Scraping (Disabling URL Resolution)](#4-ultra-fast-scraping-disabling-url-resolution)
  - [5. Pinning a Specific Chrome Version](#5-pinning-a-specific-chrome-version)
- [Troubleshooting & FAQs](#troubleshooting--faqs)

---

## Features

- 🛡️ **Anti-Bot Evasion**: Employs `undetected-chromedriver` with native browser fingerprinting to evade Google CAPTCHA blocks.
- 🤖 **Google AI Overview Extraction**: Captures the generated AI summary answer and cited sources whenever present.
- 🔗 **Automatic Redirect Unmasking**: Uses concurrent thread pooling to automatically resolve Google's `/goto?url=CAES...` redirects into full, direct landing page URLs (e.g. `https://en.wikipedia.org/...`).
- 🧹 **Clean Snippet Parsing**: Specifically targets Google's snippet elements while stripping inline scripts, image loaders, and feedback modals.
- 🪟 **Stealth Headless Mode**: Rather than using detectable `--headless=new` flags that trigger bot detection, runs off-screen (`--window-position=-52000,-52000`) so it doesn't disturb your desktop.
- 📄 **Multi-Page Pagination**: Supports scraping across multiple pages (`start=0`, `start=10`, etc.) with URL deduplication.

---

## Prerequisites

1. **Python 3.10+** (Python 3.10, 3.11, or 3.12+).
2. **Google Chrome Browser**: A standard desktop installation of Google Chrome must be installed on your machine.

---

## Installation

### 1. Clone the Repository

```bash
git clone https://github.com/your-username/google-search.git
cd google-search
```

### 2. Create and Activate a Virtual Environment

#### On Windows (PowerShell):
```powershell
python -m venv .venv
.venv\Scripts\Activate.ps1
```

#### On Linux / macOS:
```bash
python3 -m venv .venv
source .venv/bin/activate
```

### 3. Install Dependencies

```bash
pip install -r requirements.txt
```

> **Note for Python 3.12+ users:** `setuptools` is included in `requirements.txt`. It is required because Python 3.12 removed `distutils`, which `undetected-chromedriver` relies on for version resolution.

---

## Quick Start

Create a script (or run directly from `google_search/google_search.py`):

```python
from google_search.google_search import get_search

# Run a simple search
response = get_search("Who won the 2022 FIFA World Cup?", pages=1)

# Check if Google provided an AI Overview
if response.get("ai_response"):
    print("🤖 AI Overview:")
    print(response["ai_response"]["text"])
    print("-" * 50)

# Print organic search results
print(f"Found {response['meta']['result_count']} results:\n")
for item in response["results"]:
    print(f"Title:       {item['title']}")
    print(f"URL:         {item['url']}")
    print(f"Provider:    {item.get('provider')}")
    print(f"Description: {item.get('description')}")
    print(f"Date:        {item.get('date')}")
    print("-" * 50)
```

---

## Detailed Parameter Reference

The primary entry point is the function:

```python
def get_search(
    query: str,
    pages: int = 1,
    version: int | None = None,
    headless: bool = True,
    resolve_urls: bool = True
) -> dict:
```

### Parameter Details

| Parameter | Type | Default | Description |
| :--- | :--- | :--- | :--- |
| `query` | `str` | *(Required)* | The search query string (e.g., `"Python web scraping"`, `"World Cup 2022 winner"`). Special characters and spaces are safely encoded. |
| `pages` | `int` | `1` | Number of search result pages to scrape. <br>• Set to `1` for the first 10 results.<br>• Set to `2` or higher to paginate (`&start=10`, `&start=20`, etc.).<br>• If set to `0` or `None`, it scrapes until no more results are found. |
| `version` | `int \| None` | `None` | The major version of Google Chrome installed on your machine (e.g., `128`, `130`, `155`). <br>• If `None` (recommended), `undetected-chromedriver` automatically detects your installed Chrome version and downloads the matching driver.<br>• Pin this only if auto-detection fails or you have multiple browser versions. |
| `headless` | `bool` | `True` | Browser visibility mode:<br>• `True` *(default)*: Spawns the browser pushed off-screen (`--window-position=-52000,-52000`). It operates invisibly without popping up windows or triggering Google's headless detection CAPTCHAs.<br>• `False`: Launches Chrome in normal visible mode so you can see navigation, scrolling, and debugging in real time. |
| `resolve_urls` | `bool` | `True` | Redirect resolution setting:<br>• `True` *(default)*: Concurrent background threads send HTTP requests to Google's `/goto?url=...` redirect tokens to extract and return the real landing page URL (e.g., `https://en.wikipedia.org/...`).<br>• `False`: Skips redirect resolution and leaves the raw Google redirect URLs. Useful if you want the absolute fastest scraping speed and do not need direct external URLs. |

---

## Response Structure & Schema

The function returns a Python dictionary with the following schema:

```python
{
    "query": "Who won the 2022 FIFA World Cup?",
    "ai_response": {
        "text": "The Argentina national football team won the 2022 FIFA World Cup...",
        "sources": [
            {
                "title": "Learn more",
                "url": "https://support.google.com/websearch?p=ai_overviews",
                "provider": "support.google.com",
                "description": "AI responses may include mistakes. Learn more",
                "type": "ai_source"
            }
        ]
    },
    "results": [
        {
            "title": "2022 FIFA World Cup final",
            "url": "https://en.wikipedia.org/wiki/2022_FIFA_World_Cup_final",
            "provider": "en.wikipedia.org",
            "description": "They defeated Poland 3–1 in the round of 16, England 2–1 in the quarter-final, and Morocco 2–0 in the semi-final. The final took place in front of 88,966 ...",
            "date": "18 December 2022",
            "type": "organic"
        },
        {
            "title": "FIFA World Cup Qatar 2022™",
            "url": "https://www.fifa.com/en/tournaments/mens/worldcup/qatar2022",
            "provider": "fifa.com",
            "description": "The FIFA World Cup Qatar 2022 was played from 20 November to 18 December 2022. 32 teams competed across 64 matches in the 22nd edition of the tournament.",
            "date": "18 December 2022",
            "type": "organic"
        }
    ],
    "urls": [
        "https://en.wikipedia.org/wiki/2022_FIFA_World_Cup_final",
        "https://www.fifa.com/en/tournaments/mens/worldcup/qatar2022"
    ],
    "meta": {
        "result_count": 7,
        "url_count": 7,
        "has_ai_response": True,
        "pages_scraped": 1
    }
}
```

### Field Descriptions

- **`query`** (`str`): The query string executed.
- **`ai_response`** (`dict | None`): Contains Google's AI Overview answer. If Google did not generate an AI Overview for the query, this is `None`.
  - **`text`** (`str`): Clean text of the AI Overview answer.
  - **`sources`** (`list[dict]`): List of sources referenced by the AI answer.
- **`results`** (`list[dict]`): Deduplicated list of organic search results:
  - **`title`** (`str`): Title of the web page.
  - **`url`** (`str`): Original destination URL (unmasked from Google redirect).
  - **`provider`** (`str`): Domain or provider name (e.g., `en.wikipedia.org`, `fifa.com`).
  - **`description`** (`str`): Clean search snippet/summary from Google (with scripts and UI noise filtered).
  - **`date`** (`str | None`): Publish or update date if Google displayed one.
  - **`type`** (`str`): Type of result (`"organic"`).
- **`urls`** (`list[str]`): Deduplicated list of all external URLs found across the scraped pages.
- **`meta`** (`dict`):
  - **`result_count`** (`int`): Total number of organic results returned.
  - **`url_count`** (`int`): Total count of unique links.
  - **`has_ai_response`** (`bool`): `True` if an AI overview was detected and parsed.
  - **`pages_scraped`** (`int`): Number of Google result pages successfully processed.

---

## Advanced Usage Examples

### 1. Scraping Multiple Pages

To scrape 3 pages of results (up to ~30 results):

```python
from google_search.google_search import get_search

results = get_search("machine learning tutorials", pages=3)
print(f"Scraped {results['meta']['pages_scraped']} pages.")
print(f"Total results: {results['meta']['result_count']}")
```

### 2. Extracting AI Overview Answers

Google often provides generative AI summaries for informative or question queries:

```python
from google_search.google_search import get_search

data = get_search("How does photosynthesis work?", pages=1)

if data["meta"]["has_ai_response"]:
    ai = data["ai_response"]
    print("Answer:\n", ai["text"])
    print("\nCited Sources:")
    for s in ai["sources"]:
        print(f"- {s['title']} ({s['url']})")
else:
    print("No AI Overview generated for this query.")
```

### 3. Running in Visible (Non-Headless) Mode

If you are debugging or want to observe Chrome typing and scrolling:

```python
from google_search.google_search import get_search

data = get_search("latest tech news", pages=1, headless=False)
```

### 4. Ultra-Fast Scraping (Disabling URL Resolution)

If you only need titles and snippet text and do not require resolved destination URLs, pass `resolve_urls=False`:

```python
from google_search.google_search import get_search

# Saves ~1 second of network time by skipping HTTP 302 redirect checks
data = get_search("fast query", pages=1, resolve_urls=False)
```

### 5. Pinning a Specific Chrome Version

If auto-detection fails or you have a standalone Chrome version installed:

```python
from google_search.google_search import get_search

# Forces undetected-chromedriver to download and match Chrome 130 driver
data = get_search("Python tutorials", version=130)
```

---

## Troubleshooting & FAQs

### Q1: `ModuleNotFoundError: No module named 'distutils'`
* **Cause**: You are running Python 3.12+ where `distutils` was removed.
* **Fix**: Ensure `setuptools` is installed in your virtual environment:
  ```bash
  pip install setuptools
  ```

### Q2: `SessionNotCreatedException: This version of ChromeDriver only supports Chrome version X`
* **Cause**: Your installed Chrome version was updated, but the driver cache is stale or a version was hardcoded.
* **Fix**: Leave `version=None` in `get_search(...)` so it matches automatically. You can also delete the driver cache directory located in `%APPDATA%\undetected_chromedriver` (Windows) or `~/.local/share/undetected_chromedriver` (Linux/Mac).

### Q3: `OSError: [WinError 6] The handle is invalid` on script exit
* **Cause**: On Windows, when Chrome closes its background communication pipes, Python's garbage collection occasionally warns that the process handle closed early.
* **Status**: This is harmless and does not affect the scraped data. The script includes a safe deletion hook (`_safe_del`) to suppress this warning.

### Q4: Does this work on cloud servers (AWS / DigitalOcean / Heroku)?
* **Note**: Google flags datacenter IP ranges very aggressively. If running on a cloud VM, install standard Chrome (`google-chrome-stable`), `xvfb` for display emulation, and route traffic through residential proxies to prevent CAPTCHAs.

---

## License

This project is licensed under the [MIT License](LICENSE).
