Metadata-Version: 2.1
Name: dig-spider
Version: 0.0.6
Summary: NO CODE!!! Base on Scrapy, crawl websites with simple configuration.
Author-email: "viking.liu" <viking.liu@qq.com>
Project-URL: Homepage, https://github.com/vikingliu/dig
Project-URL: Issues, https://github.com/vikingliu/dig/issues
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: scrapy >=2.11.2
Requires-Dist: jsonpath >=0.82.2

======
dig-spider
======


Overview
========

Dig-spider is a BSD-licensed fast high-level web crawling and web scraping framework, used to
crawl websites and extract structured data from their pages. It can be used for
a wide range of purposes, from data mining to monitoring and automated testing.

Dig-spider is a code-free crawler. It is based on scrapy, and support the same command line with scrapy.



Requirements
============

* Python 3.9+
* Scrapy 2.11+
* Works on Linux, Windows, macOS, BSD

Install
=======

The quick way:

.. code:: bash

    pip install dig-spider

Usage
=======

.. code:: bash
    
    dig-spider crawl website -a config=xxx.yaml

use dig-spider to replace scrapy, website is the default spider, xxx.yaml is the webpage parse rule.
