Metadata-Version: 2.1
Name: globy_webcrawlers
Version: 0.0.1
Summary: Globy Webcrawlers
Author-email: Björn L <globy@globy.ai>
Project-URL: Homepage, https://github.com/globyai
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: aiohttp
Requires-Dist: beautifulsoup4
Requires-Dist: colorlog
Requires-Dist: pandas

# Globy webcrawlers
### :star: Scrape the world with Globy! :star:
## Usage
```bash
$ globy-scraper.py -h
usage: globy-scraper.py [-h] -f URLS_FILE [-l LOGLEVEL] [-o OUTPUT_FILE] [-d] [-s] [-b BACKEND]

options:
  -h, --help            show this help message and exit
  -f URLS_FILE, --urls_file URLS_FILE
                        Newline separated file of URLs to scrape
  -l LOGLEVEL, --loglevel LOGLEVEL
                        Set log level (1-3)
  -o OUTPUT_FILE, --output_file OUTPUT_FILE
                        Output CSV file path
  -d, --debug           Debug/inspect responses (Will set PYTHONINSPECT to True)
  -s, --store_html_to_file
                        Store HTML all content to file per domain in the "html_output" folder
  -b BACKEND, --backend BACKEND
      Select backend to use for scraping: "globy" (default) or "scrapy". 
      (!) The scrapy backend will only dump data to the "html_output" folder for now. No website analysis or other functionallity is supported yet.
```
## Quickstart
You can just run globy-scraper as a script (no installation needed):
```bash
pip3 install -r requirements.txt  # Install dependencies if not already installed
./globy-scraper.py -f wordpress-top50.txt
```
----
## Install the globy-webcrawlers package
You can install the package for development and use with other Globy projects.
It's **recommended** to set up a python virtual environment before installing the package.
```bash
pip3 install -e .
```
Now you can use the package in python:
```python
from globy_webcrawlers.crawler import WebSiteDataCrawler
>>> c = WebSiteDataCrawler()
>>> c.load_urls_from_file("urls.txt")
>>> c.run()
```
Also, after installing globy-scraper, you can just run it: `globy-scraper.py -h`

## Debugging/inspecting website content
Since the crawler is asynchronous, it can be a bit tricky to debug the responses. To make it easier, you can use the `-d` flag. This will allow you to inspect the HTML content and any internal objects from the most recent website.
Here's an example:
```bash
$ ./globy-scraper.py -f wordpress-top50.txt -d
```
```python
>>> w = crawler.debug_latest_response()
>>> w.url
'https://www.thyroid.org/'
>>> len(w.html_content)
249167
>>> w.get_website_info()
```
Change the crawler.urls to get the response you want to debug, or the input url list.


## Development only
Build and & create python package:
```bash
python3 -m build
```
