Metadata-Version: 2.4
Name: kazparserbot
Version: 0.1.1
Summary: Keyword-driven web scraping pipeline (Serper + OpenAI).
Author: Kirill Yakunin
Author-email: yakunin.k@mail.ru
Requires-Python: >=3.11,<4.0
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Dist: aiohttp (>=3.11.11,<4.0.0)
Requires-Dist: beautifulsoup4 (>=4.13.0,<5.0.0)
Requires-Dist: fire (>=0.7.0,<0.8.0)
Requires-Dist: openai (>=1.58.1,<2.0.0)
Requires-Dist: pillow (>=11.1.0,<12.0.0)
Requires-Dist: python-dotenv (>=1.0.1,<2.0.0)
Requires-Dist: tenacity (>=9.0.0,<10.0.0)
Description-Content-Type: text/markdown

# Keyword Processor with Serper.dev and GPT

## Description
This script:
1. Expands each keyword into 2 * `queries_to_generate_per_keyword` queries (the first half in Russian, the second half in Kazakh).
2. Uses serper.dev to retrieve search results for these queries.
3. De-duplicates and merges those results.
4. Asks GPT to pick the top `top_results_to_get` results for each keyword.
5. Scrapes those pages via serper.dev to get full text.
6. Outputs everything to a JSON file.

## Setup
- Python 3.11+
- `poetry install`
- `poetry shell`
- Copy `.env.template` and fill in the OpenAI and Serper keys

## Usage
1. Create a JSON file with a list of keywords, for example:
 ```json
 ["Dimash Qudaibergen", "Apple Inc"]
```
2. Run:
```bash
python scrap_by_keywords.py keywords.json output.json \
    --google_results_to_get_per_query=10 \
    --top_results_to_get=5 \
    --queries_to_generate_per_keyword=3 \
    --collect_imgs_and_context=True
```
- `google_results_to_get_per_query` controls how many results to retrieve for each query.
- `top_results_to_get` is how many of those results to parse further.
- `queries_to_generate_per_keyword` is how many queries to generate in Russian (and then again in Kazakh), so total is double that.
- `collect_imgs_and_context` controls whether to collect images and context for each result after the main results are collected. Will output `<output_json>_imgs.json` and imgs folder with images. Json will contain list of dicts with 'url', 'img_url', 'context_text_before', 'context_text_after', 'file_path' fields. Only keep images > N pixels to prevent getting trash, technical imgs, headers, etc.

3. The output will be in path/to/output.json, for instance:
```json
{
  "Dimash Qudaibergen": [
    {
      "url": "...",
      "google_snippet": "...",
      "full_text": "..."
    },
    ...
  ],
  ...
}
```

