Metadata-Version: 2.4
Name: metadata_parser
Version: 2.1.0
Summary: A module to parse metadata out of urls and html documents
Home-page: https://github.com/jvanasco/metadata_parser
Author: Jonathan Vanasco
Author-email: jvanasco@2xlp.com
License: MIT
Keywords: metadata,opengraph protocol,facebook
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: 3.15
Classifier: Topic :: Text Processing :: Markup :: HTML
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.10
Description-Content-Type: text/x-rst
License-File: LICENSE.txt
Requires-Dist: BeautifulSoup4<4.14.0,>4.13.0
Requires-Dist: requests>=2.19.1
Requires-Dist: requests-toolbelt>=0.8.0
Requires-Dist: setuptools
Requires-Dist: tldextract<6.0.0
Requires-Dist: legacy-cgi; python_version >= "3.13"
Provides-Extra: testing
Requires-Dist: httpbin; extra == "testing"
Requires-Dist: flake8; extra == "testing"
Requires-Dist: flake8-import-order>=0.19.2; extra == "testing"
Requires-Dist: flask<3.0; extra == "testing"
Requires-Dist: lxml; extra == "testing"
Requires-Dist: pytest; extra == "testing"
Requires-Dist: pytest-cov; extra == "testing"
Requires-Dist: pytest-httpbin; extra == "testing"
Requires-Dist: responses; extra == "testing"
Requires-Dist: types-beautifulsoup4; extra == "testing"
Requires-Dist: types-requests; extra == "testing"
Requires-Dist: werkzeug<3.0; extra == "testing"
Provides-Extra: testing-backends
Requires-Dist: httpbin; extra == "testing-backends"
Requires-Dist: flake8; extra == "testing-backends"
Requires-Dist: flake8-import-order>=0.19.2; extra == "testing-backends"
Requires-Dist: flask<3.0; extra == "testing-backends"
Requires-Dist: lxml; extra == "testing-backends"
Requires-Dist: pytest; extra == "testing-backends"
Requires-Dist: pytest-cov; extra == "testing-backends"
Requires-Dist: pytest-httpbin; extra == "testing-backends"
Requires-Dist: responses; extra == "testing-backends"
Requires-Dist: types-beautifulsoup4; extra == "testing-backends"
Requires-Dist: types-requests; extra == "testing-backends"
Requires-Dist: werkzeug<3.0; extra == "testing-backends"
Requires-Dist: html5lib; extra == "testing-backends"
Dynamic: license-file

MetadataParser
==============

.. |build_status| image:: https://github.com/jvanasco/metadata_parser/workflows/Python%20package/badge.svg

Build Status: |build_status|

MetadataParser is a Python module for pulling metadata out of web documents.

`BeautifulSoup` is required for parsing.
`Requests` is required for fetching remote documents.
`tldextract` is utilized to parse domains, but can be disabled by setting an
environment variable.

This project has been used in production for many years, and has successfully
parsed billions of documents.


Versioning, Pinning, and Support
================================

This project is using a Semantic Versioning release schedule,
with a {MAJOR}.{MINOR}.{PATCH} format.

Users are advised to pin their installations to "metadata_parser<{MINOR +1}"

For example:

* if the current release is: `1.0.0`
* the advised pin is:  `metadata_parser<1.1.0`

PATCH releases will usually be bug fixes and new features that support backwards
compatibility with Public Methods.  Private Methods are not guaranteed to be
backwards compatible.

MINOR releases are triggered when there is a breaking change to Public Methods.
Once a new MINOR release is triggered, first-party support for the previous MINOR
release is EOL (end of life). PRs for previous releases are welcome, but giving
them proper attention is not guaranteed.

Future deprecations will raise warnings.

By populating the following environment variable, future deprecations will raise exceptions:

    export METADATA_PARSER_FUTURE=1

Installation
=============

pip install metadata_parser


Installation Recommendation
===========================

The ``requests`` library version 2.4.3 or newer is strongly recommended.

This is not required, but it is better.  On earlier versions it is possible to
have an uncaught DecodeError exception when there is an underlying redirect/404.
Recent fixes to ``requests`` improve redirect handling, urllib3 and urllib3
errors.


Features
========

* ``metadata_parser`` pulls as much metadata out of a document as possible
* Developers can set a 'strategy' for finding metadata (i.e. only accept
  opengraph or page attributes)
* Lightweight but functional(!) url validation
* Verbose logging

Logging
=======

This file utilizes extensive logging to help developers pinpoint problems.

* ``log.debug`` (10)
  This log level is mostly used to handle library maintenance and
  troubleshooting, aka "Library Debugging".  Library Debugging is verbose, but
  is nested under ``if __debug__:`` statements, so it is compiled away when
  PYTHONOPTIMIZE is set.
  Several sections of logic useful to developers will also emit logging
  statements at the ``debug`` level, regardless of PYTHONOPTIMIZE.

* ``log.info`` (20)
    This log level is only used during package initialization to notify if
    the ``tldextract`` package is being utilized or not.

* ``log.warning`` (30)
  Currently unused

* ``log.error`` (40)
  This log level will record each URL that a parse is attempted for.

  This log level is mostly used to alert users of errors that were
  encountered during url fetching and document parsing, and often emits a log
  statement just before an Exception is raised. The log statements will contain
  at least the exception type, and may contain the active URL and additional
  debugging information, if any of that information is available.
  
  URLs that trigger error logging should be collected and run on a secondary
  system that utilizes `log.debug` without PYTHONOPTIMIZE.


* ``log.critical`` (50)
  Currently unused


It is STRONGLY recommended to keep Python's logging at ``debug`` and not run
PYTHONOPTIMIZE if you are new to this package.

For experienced users, running under PYTHONOPTIMIZE to not emit debug logging is
designed to make the system run as fast as possible.  The intent of
``log.error`` is to present you with a feed of URLs as they are processed, and
show any errors that arise.  Any issues that arise should then be run on a
second system that enables debug logging to pinpoint the error.  This allows one
to split a deployment into production and R&D/troubleshooting, to maximize
the throughput of the production system.


Optional Integrations
=====================

* ``tldextract``
  This package will attempt to use the package ``tldextract`` for advanced domain
  and hostname analysis. If ``tldextract`` is not wanted, it can be disabled
  with an environment variable.


Environment Variables
=====================

* ``METADATA_PARSER__DISABLE_TLDEXTRACT``
  Default: "0".
  If set to "1", the package will not attempt to load ``tldextract``.

* ``METADATA_PARSER__ENCODING_FALLBACK``
  Default: "ISO-8859-1"
  Used as the fallback when trying to decode a response.

*  ``METADATA_PARSER__DUMMY_URL``
   Used as the fallback URL when calculating url data.


Testing and coverage
====================

Install the testing dependencies and run pytest from the checkout::

    python -m pip install -e '.[testing]'
    python -m pytest

The testing extra includes ``pytest-cov``. Each test run reports package line
coverage and the uncovered line numbers. There is no minimum coverage threshold.
The same reporting applies to the existing tox and CI test runs.

Package metadata, dependencies, and pytest, mypy, tox, and flake8 configuration
are consolidated in ``setup.cfg``. ``pyproject.toml`` declares the setuptools
build backend and contains Black's configuration. Package versions continue to
come from ``metadata_parser.__VERSION__``. Install with pip as shown above;
to build source and wheel distributions, install ``build`` and run
``python -m build`` instead of invoking ``setup.py``.

The testing dependencies include ``lxml``, and a test verifies that the default
parser selects it rather than silently using the fallback. Named-entity tests
run against both ``lxml`` and Python's ``html.parser``. To also run the optional
``html5lib`` cases, install ``'.[testing-backends]'`` instead of ``'.[testing]'``.
These are separate BeautifulSoup backends; lxml does not use html5lib internally.
The entity expectations record backend differences for title text and quoted
metadata attributes, including names with and without trailing semicolons.
The default suite covers 53 common names. An advanced suite checks every name
in Python's HTML5 entity table in both semicolon forms::

    python -m pytest tests/test_html_entities.py --html5-entities

To select only the exhaustive cases, add ``-m html5_entities``. The backend
differences, verified versions, and commands are documented in
`HTML entity regression coverage <tests/HTML_ENTITIES.rst>`_. Differences have
explicit expected results; unexpected changes fail rather than being ignored.

For a browsable report, add ``--cov-report=html``. The report is written to
``htmlcov/index.html``. Use ``--no-cov`` when coverage collection is not needed.

While the fallback design remains under discussion, ``tests/test_url_fallbacks.py``
contains candidate expectations that intentionally fail against current behavior.
To measure coverage of the passing regression suite separately, run::

    python -m pytest --ignore=tests/test_url_fallbacks.py


Notes
=====

1. This package requires BeautifulSoup 4.
2. For speed, it will instantiate a BeautifulSoup parser with lxml, and
   fallback to 'None' (the internal pure Python) if it can not load lxml.
3. URL Validation is not RFC compliant, but tries to be "Real World" compliant.

It is HIGHLY recommended that you install lxml for usage.
lxml is considerably faster.
Considerably faster.

Developers should also use a very recent version of lxml.
segfaults have been reported on lxml versions < 2.3.x;
Using at least the most recent 3.x versions is strongly recommended

The default 'strategy' is to look in this order::

    meta,page,og,dc,

Which stands for the following::

    og = OpenGraph
    dc = DublinCore
    meta = metadata
    page = page elements

Developers can specify a strategy as a comma-separated list of the above.

The only 2 page elements currently supported are::

    <title>VALUE</title> -> metadata['page']['title']
    <link rel="canonical" href="VALUE"> -> metadata['page']['link']

'metadata' elements are supported by ``name`` and ``property``.

The MetadataParser object also wraps some convenience functions, which can be
used otherwise , that are designed to turn alleged urls into well formed urls.

For example, you may pull a page::

    http://www.example.com/path/to/file.html

and that file indicates a canonical url which is simple "/file.html".

This package will try to 'remount' the canonical url to the absolute url of
"http://www.example.com/file.html".
Tt will return None if the end result is not a valid url.

This all happens under-the-hood, and is honestly really useful when dealing
with indexers and spiders.


URL Validation
==============

"Real World" URL validation is enabled by default.  This is not RFC compliant.

There are a few gaps in the RFCs that allow for "odd behavior".
Just about any use-case for this package will desire/expect rules that parse
URLs "in the wild", not theoretical.

The differences:

* If an entirely numeric ip address is encountered, it is assumed to be a
  dot-notation IPV4 and it is checked to have the right amount of valid octets.
  
  The default behavior is to invalidate these hosts::

        http://256.256.256.256
        http://999.999.999.999.999

  According to RFCs those are valid hostnames that would fail as "IP Addresses"
  but pass as "Domain Names".  However in the real world, one would never
  encounter domain names like those.

* The only non-domain hostname that is allowed, is "localhost"

  The default behavior is to invalidate these hosts ::

        http://example
        http://examplecom

  Those are considered to be valid hosts, and might exist on a local network or
  custom hosts file.  However, they are not part of the public internet.

Although this behavior breaks RFCs, it greatly reduces the number of
"False Positives" generated when analyzing internet pages. If you want to
include bad data, you can submit a kwarg to ``MetadataParser.__init__``


Security Concerns
=================

# Security Concerns

`metadata_parser` fetches URLs supplied by the calling application. As with any
general-purpose HTTP client, applications which process untrusted or
attacker-controlled URLs should consider SSRF and establish an appropriate
security policy for the URLs they permit the library to fetch.

As with Requests and the Python standard library, developers should ALWAYS
inspect the initial URL and the resulting redirect chain to ensure that the
requests made by the application comply with its security policies.

For convenience, `MetadataParser.__init__` and `MetadataParser.fetch_url`
accept a `func_hook_security_policy_url` kwarg which provides an explicit
integration point for implementing such a policy.

`func_hook_security_policy_url` is a Callable which accepts a URL string and,
when inspecting a fetched response, the active `requests.Response` object.
It should return `True` to permit the operation or `False` to reject it.

The hook is invoked against the requested URL **BEFORE** attempting the initial
fetch. After Requests has completed its normal redirect processing, the hook
is also invoked with the requested URL and the active response, allowing the
application to inspect the resulting response and redirect chain. A failed
check raises an exception of `metadata_parser.exceptions.SecurityPolicyViolation`
or one of it's subclasses.

`metadata_parser` is a library, not an HTTP proxy, webhook, browser, or
service endpoint. It does not communicate with the party who supplied the URL.
The response returned by Requests remains within the process which invoked
`metadata_parser`.

If an application accepts an attacker-controlled URL and passes it to
`MetadataParser`, the application may cause its own process to make an HTTP
request to that destination. If the destination is an internal service, the
response is returned to the application's own process; it is **not** returned
to the attacker merely because the attacker supplied the initial URL.

An attacker may use redirects to cause `metadata_parser` / Requests to make
requests to destinations that the application did not intend to access.
Applications which accept untrusted URLs should therefore apply their own
security policy to both the initial URL and the resulting redirect chain.

The `func_hook_security_policy_url` hook is provided as a convenience for
implementing that application-specific policy. It does not impose a
"public Internet only" policy, since private, loopback, or internal URLs may
be legitimate for some applications.


Handling Bad URLs and Encoded URIs
==================================

This library corrects several common metadata and URL formatting problems.

Encoded URIs and relative urls
------------------------------

Most website publishers will define an image as a URL::

    <meta property="og:image" content="http://example.com/image.jpg" />

Some will define an image as an encoded URI::

    <meta property="og:image" content="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNM+Q8AAc0BZX6f84gAAAAASUVORK5CYII=" />

By default, the ``get_metadata_link()`` method can be used to ensure a valid link
is extracted from the metadata payload::

    >>> import metadata_parser
    >>> page = metadata_parser.MetadataParser(url="http://www.example.com")
    >>> print(page.get_metadata_link('image'))

This method accepts a kwarg ``allow_encoded_uri`` (default False) which will
return the image without further processing::

    >>> print(page.get_metadata_link('image', allow_encoded_uri=True))

Similarly, if a url is local::

    <meta property="og:image" content="/image.jpg" />

The ``get_metadata_link`` method will automatically upgrade it onto the domain::

    >>> print(page.get_metadata_link('image'))
    http://example.com/image.jpg

Poorly Constructed Canonical URLs
---------------------------------

Many website publishers implement canonical URLs incorrectly.
This package tries to fix that.

By default ``MetadataParser`` is constructed with ``require_valid_netloc=True``
and ``allow_localhosts=True``.

``require_valid_netloc`` checks hostname and IP-address formatting when
interpreting or reconstructing URLs. It does not determine whether a
destination is safe to contact.

``allow_localhosts`` controls whether known localhost values are accepted
during URL reconstruction. It originated as a routing compatibility option
and does not classify private networks or enforce request security. Both flags
are retained for compatibility.

Applications can integrate their own request security policy through
``func_hook_security_policy_url``. See Security Concerns above for hook timing
and redirect handling.

For example, these will all be valid URLs::

    http://example.com
    http://1.2.3.4
    http://localhost
    http://127.0.0.1
    http://0.0.0.0

If these known 'localhost' urls are not wanted, they can be filtered out with
``allow_localhosts=False``::

    http://localhost
    http://127.0.0.1
    http://0.0.0.0

There are two convenience methods that can be used to get a canonical url or
calculate the effective url::

* MetadataParser.get_discrete_url
* MetadataParser.get_metadata_link

These both accept an argument ``require_public_global``, which defaults to ``True``.
Despite its name, this option controls metadata URL normalization; it does not
establish that an address is globally routable or enforce a security policy.

Assuming we have the following content on the url ``http://example.com/path/to/foo``::

    <link rel="canonical" href="http://localhost:8000/alt-path/to/foo">

By default, versions 0.9.0 and later will detect 'localhost:8000' as an
improper canonical url, and remount the local part "/alt-path/to/foo" onto the
domain that served the file.  The vast majority of times this 'behavior'
has been encountered, this is the intended canonical::

    print(page.get_discrete_url())
    >>> http://example.com/alt-path/to/foo

In contrast, versions 0.8.3 and earlier will not catch this situation::

    print(page.get_discrete_url())
    >>> http://localhost:8000/alt-path/to/foo

In order to preserve the earlier behavior, just submit ``require_public_global=False``::

    print(page.get_discrete_url(require_public_global=False))
    >>> http://localhost:8000/alt-path/to/foo


Replaying saved HTML and URL context
------------------------------------

A fallback URL also supplies context when reconstructing a parser from saved
data. An application may save a fetched HTML document and its parsed metadata,
then parse the HTML again with a different configuration or a newer library.
Supplying the saved document URL lets relative metadata links resolve in the
same context as the original document::

    page = MetadataParser(
        url=saved_requested_url,
        url_actual=saved_response_url,
        html=saved_html,
        html_encoding=saved_encoding,
    )

When ``html`` is supplied, this constructor does not fetch ``url``. The URL is
document context, not a request to revisit the original server. Replaying HTML
creates a new parsed result; it does not restore previously extracted metadata.
For documents fetched through redirects, applications should retain both the
requested URL and the final response URL. Pass these as ``url`` and
``url_actual``, respectively. The final URL provides the base for relative links
when replaying the saved document, and is also used by the synthetic response.
Both values are retained in the parsed result's ``_internal`` metadata.
If ``url_actual`` is omitted, it defaults to ``url`` for compatibility.
Supplying ``url_actual`` requires ``html``; when fetching a live URL, the library
sets ``url_actual`` from ``Response.url`` instead.

URL reconstruction may be useful for public sites, local development servers,
or internal systems. ``require_public_global=False`` preserves local metadata
URLs where the corresponding normalization methods permit them.

There is currently an inconsistency in ``get_discrete_url``: its metadata
processing filters fallback candidates when ``require_public_global=True``,
but if no usable metadata is found, its final fallback can still return a local
source URL. The flag must not be interpreted as a guarantee that every returned
URL excludes localhost. The desired fallback behavior is being evaluated
separately from request security.

Applications enforce request security through their own policy, using
``func_hook_security_policy_url`` as an integration point. As described above,
the initial check runs before fetching; the response check runs after Requests
has followed redirects. URL normalization and replay options are independent
of that policy.


Handling Bad Data
=================

Many CMS systems (and developers) create malformed content or incorrect
document identifiers.  When this happens, the BeautifulSoup parser will lose
data or move it into an unexpected place.

There are two arguments that can help you analyze this data:

* force_doctype::

    ``MetadataParser(..., force_doctype=True, ...)``

``force_doctype=True`` will try to replace the identified doctype with "html"
via regex.  This will often make the input data usable by BS4.

* search_head_only::

    ``MetadataParser(..., search_head_only=False, ...)``

``search_head_only=False`` will not limit the search path to the "<head>" element.
This will have a slight performance hit and will incorporate data from CMS/User
content, not just templates/Site-Operators.


WARNING
=============

Please pin your releases.


Usage
=====

Until version ``0.9.19``, the recommended way to get metadata was to use
``get_metadata`` which will return a string (or None):

**From an URL**::

    >>> import metadata_parser
    >>> page = metadata_parser.MetadataParser(url="http://www.example.com")
    >>> print(page.metadata)
    >>> print(page.get_metadatas('title'))
    >>> print(page.get_metadatas('title', strategy=['og',]))
    >>> print(page.get_metadatas('title', strategy=['page', 'og', 'dc',]))

**From HTML**::

    >>> HTML = """<here>"""
    >>> page = metadata_parser.MetadataParser(html=HTML)
    >>> print(page.metadata)
    >>> print(page.get_metadatas('title'))
    >>> print(page.get_metadatas('title', strategy=['og',]))
    >>> print(page.get_metadatas('title', strategy=['page', 'og', 'dc',]))


Malformed Data
==============

It is very common to find malformed data. As of version ``0.9.20`` the following
methods should be used to allow malformed presentation::

    >>> page = metadata_parser.MetadataParser(html=HTML, support_malformed=True)

or::

    >>> parsed = page.parse(html=html, support_malformed=True)
    >>> parsed = page.parse(html=html, support_malformed=False)

The above options will support parsing common malformed options.  Currently
this only looks at alternate (improper) ways of producing twitter tags, but may
be expanded.

Notes
=====

when building on Python3, a ``static`` toplevel directory may be needed

This library was originally based on Erik River's
`opengraph module <https://github.com/erikriver/opengraph>`_. Something more
aggressive than Erik's module was needed, so this project was started.
