Metadata-Version: 2.1
Name: dig-spider
Version: 0.0.8
Summary: NO CODE!!! Base on Scrapy, crawl websites with simple configuration.
Author-email: "viking.liu" <viking.liu@qq.com>
Project-URL: Homepage, https://github.com/vikingliu/dig
Project-URL: Issues, https://github.com/vikingliu/dig/issues
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: scrapy >=2.11.2
Requires-Dist: jsonpath >=0.82.2

======
dig-spider
======


Overview
========

Dig-spider is a BSD-licensed fast high-level web crawling and web scraping framework, used to
crawl websites and extract structured data from their pages. It can be used for
a wide range of purposes, from data mining to monitoring and automated testing.

Dig-spider is a code-free crawler. It is based on scrapy, and support the same command line with scrapy.



Requirements
============

* Python 3.9+
* Scrapy 2.11+
* Works on Linux, Windows, macOS, BSD

Install
=======

The quick way:

    pip install dig-spider

Usage
=======
    dig-spider gentemplate dst

generate template to target directory (dst)
modify config_template.yaml and code_template.py

    dig-spider crawl website -a config=dst/config_template.yaml

use dig-spider to replace scrapy, website is the default spider, dst/config_template.yaml is the webpage parse rule.
