Metadata-Version: 2.4
Name: gem-scrapscrap
Version: 1.6.1
Summary: CLI tool to scrape and download contract documents from GeM (Government e-Marketplace)
Author: Wickcore
License: MIT
Project-URL: Homepage, https://github.com/Wickcore/gem-scrapscrap
Project-URL: Repository, https://github.com/Wickcore/gem-scrapscrap
Project-URL: Issues, https://github.com/Wickcore/gem-scrapscrap/issues
Keywords: gem,government,scraping,contracts,tenders,schedule,cron
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: requests>=2.28.0
Requires-Dist: beautifulsoup4>=4.12.0
Requires-Dist: rich>=13.0.0
Requires-Dist: questionary>=2.0.0
Requires-Dist: schedule>=1.2.0

# GeM Document Scraper

A Python web app that scrapes and downloads contract documents from [GeM (Government e-Marketplace)](https://gem.gov.in). Enter a category, set a date range, and download all matching contract PDFs — with automatic CAPTCHA solving.

## Features

- **Web UI** — Clean dark-themed interface, no terminal needed
- **Category Search** — Autocomplete from GeM's 13,000+ categories
- **Auto CAPTCHA** — Solves GeM's image CAPTCHAs via Tesseract OCR
- **PDF Downloads** — Downloads individual files or all as ZIP
- **Real-time Progress** — Live logs and progress bar while scraping
- **Background Processing** — Scraper runs in a thread, UI stays responsive

## Demo

1. Enter a category (e.g., "Ball Point Pens (V2) as per IS 3705")
2. Set date range (DD-MM-YYYY)
3. Click "Start Scraping"
4. Watch live progress, download PDFs when done

## Quick Start

### Prerequisites

- Python 3.9+

### Install

```bash
pip3 install -r requirements.txt
```

### Run CLI (Interactive Terminal)

```bash
python3 gem_cli.py
```

### Run Web UI

```bash
python3 app.py
```

Open **http://localhost:5000** in your browser.

## How It Works

1. **Search** — User enters category + date range in the web UI
2. **CAPTCHA** — Opens GeM's contract page, extracts the CAPTCHA image, reads it via Tesseract OCR, submits automatically (retries up to 15 times)
3. **Scrape** — Collects all matching contract documents from the results
4. **Download** — For each document, calls GeM's AJAX endpoint to get the download URL, then fetches the PDF via HTTP
5. **Serve** — Downloaded PDFs are served back through the web UI for browsing/downloading

## Project Structure

```
GeM-Document-Scraper/
├── app.py              # Flask web server (routes, job management, threading)
├── scraper.py          # Background scraping engine (Selenium + AJAX downloads)
├── actions.py          # Selenium helper functions (category, dates, CAPTCHA)
├── main.py             # Legacy CLI entry point (still works standalone)
├── templates/
│   └── index.html      # Web UI (HTML + CSS + JS, no frameworks)
├── requirements.txt    # Python dependencies
├── downloads/          # Downloaded PDFs (per job)
└── captcha/            # CAPTCHA images (temporary)
```

## API Endpoints

| Method | Route | Description |
|--------|-------|-------------|
| GET | `/` | Web UI |
| POST | `/scrape` | Start a new scraping job |
| GET | `/status/<job_id>` | Job status + logs + file list |
| GET | `/download/<job_id>/<file>` | Download a single PDF |
| GET | `/download_all/<job_id>` | Download all PDFs as ZIP |

## Tested Categories

These categories are known to have contracts on GeM:

| Category | Notes |
|----------|-------|
| `Ball Point Pens (V2) as per IS 3705` | Office supplies, high volume |
| `Note Sorting Machines (V2)` | Banking equipment |
| `LED Bulb with Battery as per IS 16102` | Govt LED scheme |
| `Split Air Conditioner, Wall Mount Type (V3) ISI Marked to IS 1391 (Part 2)` | HVAC |
| `Nitrogen Tyre Inflators` | Automotive |

Use a wide date range (e.g., `01-01-2025` to `07-07-2025`) for more results.

## Date Format

The web UI accepts multiple formats and auto-converts:
- `DD-MM-YYYY` (e.g., 01-07-2025)
- `DD/MM/YYYY` (e.g., 01/07/2025)
- `YYYY-MM-DD` (e.g., 2025-07-01)

## Dependencies

| Package | Purpose |
|---------|---------|
| Flask | Web server |
| Selenium | Browser automation |
| Tesseract OCR (`pytesseract`) | CAPTCHA reading |
| Pillow | Image processing |
| Requests | HTTP file downloads |
| Gunicorn | Production WSGI server |

## Deploy to Railway

1. Go to [railway.app](https://railway.app) → Sign up with GitHub
2. Click **New Project** → **Deploy from GitHub repo**
3. Select `Wickcore/gem-scrapscrap`
4. Railway auto-detects the Dockerfile
5. Click **Deploy**

Your app will be live at `https://your-app.up.railway.app`

**Note:** Railway gives $5 free credits/month — enough for this app.

## Notes

- CAPTCHA accuracy depends on image quality; the scraper retries up to 15 times
- Headless Chrome is used — no browser window pops up
- Each scraping job runs in its own thread with its own download directory
- GeM may change their page structure, which could break selectors

## License

MIT
