Metadata-Version: 2.4
Name: scrapy-poet
Version: 0.27.2
Summary: Page Object pattern for Scrapy
Project-URL: Homepage, https://github.com/scrapinghub/scrapy-poet
Project-URL: Tracker, https://github.com/scrapinghub/scrapy-poet/issues
Project-URL: Documentation, https://scrapy-poet.readthedocs.io/
Project-URL: Release notes, https://scrapy-poet.readthedocs.io/en/stable/changelog.html
Author-email: Mikhail Korobov <kmike84@gmail.com>
License-Expression: BSD-3-Clause
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Framework :: Scrapy
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: BSD License
Classifier: Natural Language :: English
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.10
Requires-Dist: andi>=0.6.0
Requires-Dist: attrs>=21.3.0
Requires-Dist: parsel>=1.5.0
Requires-Dist: scrapy>=2.6.0
Requires-Dist: sqlitedict>=1.5.0
Requires-Dist: time-machine>=2.7.1
Requires-Dist: twisted>=18.9.0
Requires-Dist: url-matcher>=0.2.0
Requires-Dist: web-poet>=0.17.0
Description-Content-Type: text/x-rst

===========
scrapy-poet
===========

.. image:: https://img.shields.io/pypi/v/scrapy-poet.svg
   :target: https://pypi.python.org/pypi/scrapy-poet
   :alt: PyPI Version

.. image:: https://img.shields.io/pypi/pyversions/scrapy-poet.svg
   :target: https://pypi.python.org/pypi/scrapy-poet
   :alt: Supported Python Versions

.. image:: https://github.com/scrapinghub/scrapy-poet/workflows/tox/badge.svg
   :target: https://github.com/scrapinghub/scrapy-poet/actions
   :alt: Build Status

.. image:: https://codecov.io/github/scrapinghub/scrapy-poet/coverage.svg?branch=master
   :target: https://codecov.io/gh/scrapinghub/scrapy-poet
   :alt: Coverage report

.. image:: https://readthedocs.org/projects/scrapy-poet/badge/?version=stable
   :target: https://scrapy-poet.readthedocs.io/en/stable/?badge=stable
   :alt: Documentation Status

``scrapy-poet`` is the `web-poet`_ Page Object pattern implementation for Scrapy.
``scrapy-poet`` allows to write spiders where extraction logic is separated from the crawling one.
With ``scrapy-poet`` is possible to make a single spider that supports many sites with
different layouts.

Requires **Python 3.10+** and **Scrapy >= 2.6.0**.

Read the `documentation <https://scrapy-poet.readthedocs.io>`_  for more information.

License is BSD 3-clause.

* Documentation: https://scrapy-poet.readthedocs.io
* Source code: https://github.com/scrapinghub/scrapy-poet
* Issue tracker: https://github.com/scrapinghub/scrapy-poet/issues

.. _`web-poet`: https://github.com/scrapinghub/web-poet


Quick Start
***********

Installation
============

.. code-block::

    pip install scrapy-poet

Usage in a Scrapy Project
=========================

Add the following inside Scrapy's ``settings.py`` file:

-   Scrapy ≥ 2.10:

    .. code-block:: python

        ADDONS = {
            "scrapy_poet.Addon": 300,
        }

-   Scrapy < 2.10:

    .. code-block:: python

        DOWNLOADER_MIDDLEWARES = {
            "scrapy_poet.InjectionMiddleware": 543,
            "scrapy.downloadermiddlewares.stats.DownloaderStats": None,
            "scrapy_poet.DownloaderStatsMiddleware": 850,
        }
        REQUEST_FINGERPRINTER_CLASS = "scrapy_poet.ScrapyPoetRequestFingerprinter"
        SPIDER_MIDDLEWARES = {
            "scrapy_poet.RetryMiddleware": 275,
        }

Developing
==========

Setup your local Python environment via:

1. `pip install -r requirements-dev.txt`
2. `pre-commit install`

Now everytime you perform a `git commit`, these tools will run against the
staged files:

* `black`
* `isort`
* `flake8`

You can also directly invoke `pre-commit run --all-files` or `tox -e linters`
to run them without performing a commit.
