Metadata-Version: 2.4
Name: html-similarity
Version: 0.5.0
Summary: A set of similarity metricts to compare html files.
Author-email: Edgar Marca <matiskay@gmail.com>
Maintainer-email: Edgar Marca <matiskay@gmail.com>
License-Expression: BSD-3-Clause
Project-URL: Homepage, https://github.com/matiskay/html-similarity
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.14
Description-Content-Type: text/x-rst
License-File: LICENSE
Requires-Dist: lxml>=5.0
Requires-Dist: rapidfuzz>=3.14.5
Dynamic: license-file

===============
HTML Similarity
===============

This package provides a set of functions to measure the similarity between web pages.

Install
=======

The quick way::

    pip install html-similarity

How it works?
=============

Structural Similarity
---------------------

Uses sequence comparison of the html tags to compute the similarity, by default.

We not implement the similarity based on tree edit distance because it is slower than sequence comparison.

``structural_similarity`` accepts an ``algorithm`` keyword to pick the comparison strategy:

- ``indel`` (default): flat tag-sequence comparison using rapidfuzz's bit-parallel Indel/LCS
  implementation. Fastest option, but blind to nesting (e.g. moving an element to a different
  parent without changing the overall tag order won't affect the score).
- ``pq_gram``: tree-structure aware. Compares `pq-gram <https://www.lsi.upc.edu/~nin/teaching/sed/lectures/pqgram-tods.pdf>`_
  profiles, which approximate Tree Edit Distance in roughly linear time while still capturing
  parent/child relationships. Slower than ``indel`` but catches structural changes that a flat
  sequence misses.
- ``difflib``: legacy flat tag-sequence comparison (the original implementation), kept mainly for
  benchmarking against ``indel``.

See ``notebooks/structural_similarity_benchmark.ipynb`` for a notebook that compares the speed and
the structural sensitivity of all three.


Style Similarity
----------------

Extracts css classes of each html document and calculates the jaccard similarity of the sets of classes.


Joint Similarity (Structural Similarity and Style Similarity)
-------------------------------------------------------------

The joint similarity metric is calculated as::

    k * structural_similarity(document_1, document_2) + (1 - k) * style_similarity(document_1, document_2)


All the similarity metrics takes values between 0 and 1.

Recommendations for joint similarity
------------------------------------

Using `k=0.3` give use better results. The style similarity gives more information about the similarity rather than the structural similarity.

Examples
========

Here is a example::

    In [1]: html_1 = '''
    <h1 class="title">First Document</h1>
    <ul class="menu">
        <li class="active">Documents</li>
        <li>Extra</li>
    </ul>
    '''

    In [2]: html_2 = '''
    <h1 class="title">Second document Document</h1>
    <ul class="menu">
        <li class="active">Extra Documents</li>
    </ul>
    '''

    In [3] from html_similarity import style_similarity, structural_similarity, similarity

    In [4]: style_similarity(html_1, html_2)
    Out[4]: 1.0

    In [7]: structural_similarity(html_1, html_2)
    Out[7]: 0.9090909090909091

    In [8]: similarity(html_1, html_2)
    Out[8]: 0.9545454545454546

References
==========

- The idea of sequence comparision was taken from `Page Compare <https://github.com/TeamHG-Memex/page-compare>`_.
- The other ideas were taken from `T. Gowda and C. A. Mattmann, Clustering Web Pages Based on Structure and Style Similarity, 2016 IEEE 17th International Conference on Information Reuse and Integration (IRI), Pittsburgh, PA, 2016, pp. 175-180. <http://ieeexplore.ieee.org/document/7785739/>`_
- Use case `Clustering web pages based on structure and style similarity <https://www.slideshare.net/thammegowda/ieee-iri-16-clustering-web-pages-based-on-structure-and-style-similarity?qid=7deea5f8-157d-4e57-a413-16ec7c6a22d9&v=&b=&from_search=1>`_

Thanks
======

- `@beppler <https://github.com/beppler>`_ for `Support load html from bytes <https://github.com/matiskay/html-similarity/pull/107>`_
- `@catarinaacsilva <https://github.com/catarinaacsilva>`_ for `Fix one error regarding xml code inside html <https://github.com/matiskay/html-similarity/pull/51>`_
