Metadata-Version: 2.4
Name: scrapedesk
Version: 1.0.1
Summary: A friendly desktop web scraper: grab images, tables, links and text from any website and export to Excel, CSV, JSON or SQLite.
Author-email: "Engr. Mubashir Hussain" <mubashir6u@gmail.com>, Atiya Batool <attiyabatool731@gmail.com>, Zainab Wasim <zainabwasim584@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/mubashir6u/scrapedesk
Project-URL: Issues, https://github.com/mubashir6u/scrapedesk/issues
Project-URL: Changelog, https://github.com/mubashir6u/scrapedesk/blob/main/CHANGELOG.md
Keywords: web scraping,scraper,gui,tkinter,images,tables,excel,csv,crawler,beautifulsoup,data extraction,desktop
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: MacOS X
Classifier: Environment :: Win32 (MS Windows)
Classifier: Environment :: X11 Applications
Classifier: Intended Audience :: End Users/Desktop
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Topic :: Office/Business
Classifier: Topic :: Utilities
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.31
Requires-Dist: beautifulsoup4>=4.12
Requires-Dist: lxml>=5.0
Requires-Dist: pandas>=2.0
Requires-Dist: openpyxl>=3.1
Requires-Dist: pillow>=10.0
Provides-Extra: js
Requires-Dist: playwright>=1.40; extra == "js"
Provides-Extra: selenium
Requires-Dist: selenium>=4.15; extra == "selenium"
Provides-Extra: all
Requires-Dist: playwright>=1.40; extra == "all"
Requires-Dist: selenium>=4.15; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Requires-Dist: build>=1.0; extra == "dev"
Requires-Dist: twine>=5; extra == "dev"
Dynamic: license-file

# ScrapeDesk

**A friendly desktop web scraper.** Paste a link, choose what you want (images, tables, links, text, emails or custom fields) and export it to Excel, CSV, JSON, SQLite and more. You don't need to write any code.

```bash
pip install scrapedesk
scrapedesk
```

Works on Windows, macOS and Linux with Python 3.9+.

![ScrapeDesk home screen](https://raw.githubusercontent.com/mubashir6u/scrapedesk/main/docs/screenshots/home.png)

| Image gallery | Tables (dark mode) |
|---|---|
| ![Image gallery](https://raw.githubusercontent.com/mubashir6u/scrapedesk/main/docs/screenshots/images.png) | ![Tables in dark mode](https://raw.githubusercontent.com/mubashir6u/scrapedesk/main/docs/screenshots/tables-dark.png) |

---

## Highlights

| | |
|---|---|
| **Easy to use** | Sidebar navigation, big toggle cards, live result counters, tooltips, notifications, light and dark mode, and keyboard shortcuts |
| **Images** | Thumbnail gallery with click-to-select and a full-size viewer. Finds `img`, `srcset`, lazy-loaded (`data-src` and more), `<picture>`, CSS backgrounds, `og:image`, icons and linked image files |
| **Smart downloads** | Filter by type, width/height and file size. Removes duplicates, downloads in parallel, renames, and converts to PNG/JPG/WEBP |
| **Tables** | Every `<table>`, including merged cells (`colspan`/`rowspan`) and multi-row headers. Can also turn product grids and lists into tables. Sort, filter, merge, and copy to Excel |
| **Exports** | Excel (one sheet per table), CSV, TSV, JSON, HTML, SQLite and Markdown, or *everything* in one workbook |
| **Custom fields** | Pick exact values (title, price, rating, link…) with CSS or XPath and get one row per item |
| **More pages** | Follows "Next page" pagination and crawls links with depth, page limits and include/exclude rules |
| **Tough sites** | Browser identities, headers, cookies, proxy, retries, request delay, robots.txt, and optional JavaScript rendering |

## Install

```bash
pip install scrapedesk
```

Start the app:

```bash
scrapedesk               # or:  python -m scrapedesk
scrapedesk https://example.com   # open with a URL already filled in
```

> **Linux:** tkinter is sometimes packaged separately: `sudo apt install python3-tk`.

### JavaScript-heavy websites (optional)

If a site builds its content with JavaScript (React or Vue apps, infinite scroll), install the browser extra and turn on **Settings → JavaScript websites → Playwright**:

```bash
pip install "scrapedesk[js]"
python -m playwright install chromium
```

If you skip the last step, ScrapeDesk falls back to an installed Google Chrome or Microsoft Edge.

## How to use

1. **Home:** paste a URL (or several), then click the cards for what you want to collect and press **Start**.
2. Watch the counters update. When the job finishes, a notification takes you to the results.
3. **Images:** click thumbnails to select them, double-click to see full size, then **Download selected**.
4. **Tables:** pick a table, then **Export** it (Excel, CSV…) or **Copy** it straight into Excel.
5. **Custom Fields:** press **Load example** to see how product names and prices are extracted.
6. Press **Ctrl+E** to export everything in one Excel workbook.

Keyboard shortcuts:

| Keys | Action |
|---|---|
| Enter / Ctrl+Enter | Start |
| Esc | Stop |
| Ctrl+1…9 | Switch section |
| Ctrl+E | Export all |
| Ctrl+D | Dark mode |
| F1 | Help |

## Command line

Everything also works without the window:

```bash
# all images wider than 300 px
scrapedesk-cli https://example.com --images --download --min-width 300 --types jpg,png

# every table to Excel
scrapedesk-cli https://example.com/stats --tables -o stats.xlsx

# one row per product across 50 pages
scrapedesk-cli https://books.toscrape.com --paginate --max-pages 50 \
    --row article.product_pod --css "title=h3 a@title" --css "price=.price_color" -o books.csv

# crawl a site and export everything
scrapedesk-cli https://example.com --crawl --depth 2 --max-pages 100 --report everything.xlsx
```

Run `scrapedesk-cli --help` for all options.

## Use it from Python

```python
from scrapedesk import ScrapeSettings, WebScraper
from scrapedesk.exporters import export_dataframes, tables_to_frames

settings = ScrapeSettings(extract_images=True, extract_tables=True, min_width=200)
scraper = WebScraper(settings)
pages = scraper.run(["https://example.com"])

tables = [t for p in pages for t in p.tables]
export_dataframes(tables_to_frames(tables), "tables.xlsx")
scraper.download_images([img for p in pages for img in p.images])
```

## Responsible use

Only scrape content you are allowed to access. Respect each website's terms of service, robots.txt (there is a setting for it) and rate limits (use the delay setting), and follow privacy laws such as GDPR when you collect personal data. ScrapeDesk does not bypass logins, CAPTCHAs or anti-bot protection.

## Contributing

Bug reports and ideas are welcome. Please open an issue at https://github.com/mubashir6u/scrapedesk/issues

To work on the code:

```bash
git clone https://github.com/mubashir6u/scrapedesk.git
cd scrapedesk
pip install -e ".[dev]"
pytest
```

## License

MIT © Engr. Mubashir Hussain, Atiya Batool and Zainab Wasim
