Metadata-Version: 2.4
Name: valentine
Version: 1.1.0
Summary: Valentine: find relationships between columns of different tabular datasets
Author-email: Delft Data <delftdatasystems@gmail.com>
Maintainer-email: Delft Data <delftdatasystems@gmail.com>
License-Expression: Apache-2.0
Project-URL: Homepage, https://delftdata.github.io/valentine/
Project-URL: Documentation, https://delftdata.github.io/valentine/
Project-URL: Source, https://github.com/delftdata/valentine
Project-URL: Issues, https://github.com/delftdata/valentine/issues
Project-URL: Changelog, https://github.com/delftdata/valentine/releases
Keywords: matching,valentine,schema matching,dataset discovery,coma,cupid,similarity flooding
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: 3.15
Classifier: Operating System :: POSIX
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: Microsoft :: Windows
Requires-Python: <3.16,>=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy<3.0,>=2.0
Requires-Dist: pandas<4.0,>=2.0
Requires-Dist: nltk<4.0,>=3.10.3
Requires-Dist: anytree<3.0,>=2.9
Requires-Dist: networkx<4.0,>=2.8
Requires-Dist: chardet<8.0,>=7.0
Requires-Dist: rapidfuzz<4.0,>=3.0
Requires-Dist: PuLP[cbc]<5.0,>=4.0
Requires-Dist: POT<1.0,>=0.9.5
Requires-Dist: scipy<2.0,>=1.12
Requires-Dist: python-dateutil<3.0,>=2.8
Provides-Extra: dev
Requires-Dist: pytest<10.0,>=9.0; extra == "dev"
Requires-Dist: coverage<8.0,>=7.0; extra == "dev"
Requires-Dist: ruff<1.0,>=0.17; extra == "dev"
Provides-Extra: polars
Requires-Dist: polars<3.0,>=1.0; extra == "polars"
Provides-Extra: embeddings
Requires-Dist: sentence-transformers<7.0,>=2.0; extra == "embeddings"
Provides-Extra: docs
Requires-Dist: zensical==0.0.70; extra == "docs"
Requires-Dist: mkdocstrings==1.0.7; extra == "docs"
Requires-Dist: mkdocstrings-python==2.0.9; extra == "docs"
Dynamic: license-file

<h1 align="center">
  <img src="https://raw.githubusercontent.com/delftdata/valentine/master/docs/assets/favicon.png" alt="" width="40" valign="middle">
  Valentine
</h1>
<p align="center"><em>(Schema-) Matching DataFrames Made Easy</em></p>

<p align="center">
  <a href="https://github.com/delftdata/valentine/actions/workflows/build.yml">
    <img src="https://github.com/delftdata/valentine/actions/workflows/build.yml/badge.svg" alt="Build">
  </a>
  <a href="https://codecov.io/gh/delftdata/valentine">
    <img src="https://codecov.io/gh/delftdata/valentine/branch/master/graph/badge.svg?token=4QR0X315CL" alt="codecov">
  </a>
  <a href="https://app.codacy.com/gh/delftdata/valentine/dashboard">
    <img src="https://app.codacy.com/project/badge/Grade/85cfebfc9c6a43359c5b2e56a5fdf3a3" alt="Codacy Badge">
  </a>
  <a href="https://github.com/astral-sh/ruff">
    <img src="https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json" alt="Ruff">
  </a>
  <a href="https://pypi.org/project/valentine/">
    <img src="https://img.shields.io/pypi/v/valentine.svg" alt="PyPI version">
  </a>
  <a href="https://pypi.org/project/valentine/">
    <img src="https://img.shields.io/pypi/pyversions/valentine.svg" alt="Python versions">
  </a>
  <a href="https://pypi.org/project/valentine/">
    <img src="https://static.pepy.tech/badge/valentine/month" alt="PyPI downloads">
  </a>
  <a href="https://github.com/delftdata/valentine/blob/master/LICENSE">
    <img src="https://img.shields.io/github/license/delftdata/valentine.svg" alt="License">
  </a>
  <a href="https://delftdata.github.io/valentine/">
    <img src="https://img.shields.io/badge/docs-GitHub%20Pages-blue.svg" alt="Docs">
  </a>
</p>

---

A Python package for capturing potential relationships among columns of different tabular datasets, given as pandas or Polars DataFrames.
Valentine is based on the paper [**Valentine: Evaluating Matching Techniques for Dataset Discovery**](https://ieeexplore.ieee.org/abstract/document/9458921).

📚 **Full documentation:** <https://delftdata.github.io/valentine/> — getting started, matcher guide, API reference, and migration notes.


## Experimental suite version

The original experimental suite version of Valentine, as first published for the needs of the research paper, can be still found [here](https://github.com/delftdata/valentine/tree/v1.1).

## Installation instructions
### Requirements

*   *Python* >=3.12,<3.16
    *   On Windows and macOS, use Python 3.12–3.14 for now: Valentine's
        dependency PuLP does not yet publish Python 3.15 wheels for those platforms.

To install Valentine simply run:

```shell
pip install valentine
```

To enable **Polars** support, install the optional extra:

```shell
pip install valentine[polars]
```

To enable the **sentence-transformer embedding** distance for `JaccardDistanceMatcher` (see below), install:

```shell
pip install valentine[embeddings]
```


## Usage
Valentine can be used to find matches among columns of a given pair of pandas or Polars DataFrames. You can even mix pandas and Polars frames in the same call — Valentine auto-detects the frame type.

### Matching methods
In order to do so, the user can choose one of the following matching methods:

1.   `Coma(int: max_n, bool: use_instances, bool: use_schema, float: delta, float: threshold)` is a pure Python implementation of the [COMA 3.0](https://sourceforge.net/projects/coma-ce/) schema matching algorithm.
     *    **Parameters**:
           *    **max_n**(*int*) - Maximum number of matches to keep per column, 0 means unlimited (default: 0).
           *    **use_instances**(*bool*) - Whether to use TF-IDF instance-based matching on data values (default: False).
           *    **use_schema**(*bool*) - Whether to use schema-based matching on column names, paths, and structure (default: True).
           *    **delta**(*float*) - Fraction from the best score within which matches are kept (default: 0.15).
           *    **threshold**(*float*) - Absolute minimum similarity score to keep a match (default: 0.0).

2.   `Cupid(float: w_struct, float: leaf_w_struct, float: th_accept)` is the python implementation of the paper [Generic Schema Matching with Cupid](https://www.vldb.org/conf/2001/P049.pdf)
     *    **Parameters**:
          *    **w_struct**(*float*) - Structural similarity threshold, default is 0.2.
          *    **leaf_w_struct**(*float*) - Structural similarity threshold, leaf level, default is 0.2.
          *    **th_accept**(*float*) - Accept similarity threshold, default is 0.7.

3.   `DistributionBased(float: threshold1, float: threshold2)` is the python implementation of the paper [Automatic Discovery of Attributes in Relational Databases](https://dl.acm.org/doi/10.1145/1989323.1989336)
     *    **Parameters**:
          *    **threshold1**(*float*) - The threshold for phase 1 of the method, default is 0.15.
          *    **threshold2**(*float*) - The threshold for phase 2 of the method, default is 0.15.

4.   `JaccardDistanceMatcher(...)` is a baseline method that scores column pairs by **Tversky** similarity over their value sets (Jaccard by default). Element equality between values can be decided by a configurable string distance function, including a sentence-transformer **embedding** option for semantic matching.
     *    **Parameters**:
          *    **threshold_dist**(*float*) - Acceptance threshold above which two values are considered equal under the chosen `distance_fun`, default is 0.8. For embeddings, ~0.7 is a typical operating point.
          *    **distance_fun**(*StringDistanceFunction*) - Per-value similarity function. The enumeration class `StringDistanceFunction` can be imported from `valentine.algorithms.jaccard_distance`. Functions currently supported are:
   		       * `StringDistanceFunction.Levenshtein`: [Levenshtein distance](https://en.wikipedia.org/wiki/Levenshtein_distance) (default)
               * `StringDistanceFunction.DamerauLevenshtein`: [Damerau-Levenshtein distance](https://en.wikipedia.org/wiki/Damerau%E2%80%93Levenshtein_distance)
               * `StringDistanceFunction.Hamming`: [Hamming distance](https://en.wikipedia.org/wiki/Hamming_distance)
               * `StringDistanceFunction.Jaro`: [Jaro distance](https://en.wikipedia.org/wiki/Jaro%E2%80%93Winkler_distance)
               * `StringDistanceFunction.JaroWinkler`: [Jaro-Winkler distance](https://en.wikipedia.org/wiki/Jaro%E2%80%93Winkler_distance)
               * `StringDistanceFunction.Exact`: String equality `==`
               * `StringDistanceFunction.Embedding`: cosine similarity on sentence-transformer embeddings (requires the `valentine[embeddings]` extra)
          *    **tversky_alpha**(*float*) / **tversky_beta**(*float*) - Tversky penalty weights for unmatched values on each side (defaults `1.0`, `1.0`). Defaults give Jaccard; `0.5, 0.5` gives Sørensen-Dice; `1.0, 0.0` (or vice versa) gives set containment — useful when one column is expected to be a subset of the other.
          *    **embedding_model**(*str*) - Sentence-transformers model name when `distance_fun=Embedding` (default `"all-MiniLM-L6-v2"`).
          *    **embedding_device**(*str* or *None*) - Device override (`"cpu"`, `"cuda"`, `"mps"`). `None` (default) auto-picks: cuda → mps → cpu.
          *    **embedding_batch_size**(*int* or *None*) - Encode batch size; `None` uses the sentence-transformers default (32). Larger values amortise per-call overhead on capable hardware.

5.   `SimilarityFlooding(Policy: coeff_policy, Formula: formula, StringMatcher: string_matcher)` is the python implementation of the paper [Similarity Flooding: A Versatile Graph Matching Algorithmand its Application to Schema Matching](https://ieeexplore.ieee.org/document/994702)
     * **Parameters**:
        *    **coeff_policy**(*Policy*) - Policy for deciding the weight coefficients of the propagation graph. `Policy.INVERSE_PRODUCT` or `Policy.INVERSE_AVERAGE` (default).
        *    **formula**(*Formula*) - Formula on which iterative fixpoint computation is based. `Formula.BASIC`, `Formula.FORMULA_A`, `Formula.FORMULA_B`, or `Formula.FORMULA_C` (default).
        *    **string_matcher**(*StringMatcher*) - String matching function for the initial similarity mapping. `StringMatcher.PREFIX_SUFFIX` (default), `StringMatcher.PREFIX_SUFFIX_TFIDF`, or `StringMatcher.LEVENSHTEIN`.

### Matching DataFrames

Pass two or more DataFrames as a list (or any iterable) along with a matcher. Valentine will match columns across all unique pairs. Pandas and Polars frames can be freely mixed:

```python
# Match a pair of DataFrames (pandas, Polars, or mixed)
matches = valentine_match([df1, df2], matcher)

# Match multiple DataFrames (computes all N×(N-1)/2 pairs)
matches = valentine_match([df1, df2, df3], matcher, df_names=["sales", "orders", "products"])
```

Optionally provide `df_names` to label each DataFrame (defaults to "aaa", "bbb", etc. — designed to have zero similarity so they don't influence schema-based matchers). Function `valentine_match` returns a `MatcherResults` object, an immutable mapping from `ColumnPair` to similarity scores with convenience methods for filtering, subsetting, and evaluation.


### MatcherResults and ColumnPair

Results are keyed by `ColumnPair` namedtuples with named fields for easy access:

```python
for pair, score in matches.items():
    print(f"{pair.source_column} <-> {pair.target_column}: {score:.3f}")
    # Also available: pair.source_table, pair.target_table
    # Shorthand tuples: pair.source, pair.target
```

`MatcherResults` provides convenience methods for filtering and subsetting:
```python
top_n_matches = matches.take_top_n(5)
top_n_percent_matches = matches.take_top_percent(25)
high_confidence = matches.filter(min_score=0.7)

# One-to-one selectors — three flavours, pick the one that fits your task:
one_to_one_matches = matches.one_to_one_hungarian()  # globally optimal (default)
one_to_one_strict = matches.one_to_one_hungarian(threshold=0.5)
greedy_legacy = matches.one_to_one_greedy()  # legacy greedy assignment
mutual_only = matches.one_to_one_mutual_top(n=1)  # mutual nearest neighbour
```

`one_to_one_hungarian` (Hungarian assignment via `scipy.optimize.linear_sum_assignment`) is the recommended default and is what `Precision` / `Recall` / `F1Score` apply when their `one_to_one` flag is set. `one_to_one_greedy` preserves the legacy greedy behaviour for backwards compatibility. `one_to_one_mutual_top(n)` keeps a pair only when each side ranks the other in its top-`n` — a high-precision filter that drops one-sided affinities.

### Match details (Coma)

When using the Coma matcher, per-sub-matcher score breakdowns are available via `.details`:

```python
for pair, score in matches.items():
    details = matches.get_details(pair)
    if details:
        print(f"{pair.source_column} <-> {pair.target_column}: {details}")
        # e.g. {'NameCM': 0.72, 'PathCM': 0.65, 'LeavesCM': 0.58, ...}
```


### Measuring effectiveness

```python
metrics = matches.get_metrics(ground_truth)
```

Computes Precision, Recall, F1-score and others as described in the original Valentine paper. The ground truth is a list of `(source_column, target_column)` tuples:

```python
ground_truth = [("emp_id", "employee_number"), ("fname", "first_name"), ...]
```

Custom metrics can be specified, and predefined sets are available:

```python
from valentine.metrics import F1Score, PrecisionTopNPercent, METRICS_PRECISION_INCREASING_N

metrics_custom = matches.get_metrics(
    ground_truth, metrics={F1Score(one_to_one=False), PrecisionTopNPercent(n=70)}
)
metrics_predefined_set = matches.get_metrics(ground_truth, metrics=METRICS_PRECISION_INCREASING_N)
```

The 1:1 selection algorithm used when a metric's `one_to_one` flag is `True` can be overridden per call (default `"hungarian"`):

```python
metrics_strict = matches.get_metrics(
    ground_truth, metrics={F1Score()}, one_to_one_method="mutual_top"
)
metrics_legacy = matches.get_metrics(ground_truth, metrics={F1Score()}, one_to_one_method="greedy")
```

Valid values are `"hungarian"` (default), `"greedy"`, and `"mutual_top"`. Metrics whose `one_to_one` flag is `False` (e.g. `MeanReciprocalRank`, `RecallAtSizeofGroundTruth`) ignore the argument.


### Example
The following block of code shows: 1) how to run a matcher from Valentine on two DataFrames storing information about job candidates, and then 2) how to assess its effectiveness based on a given ground truth. More examples are available in the [`examples/`](https://github.com/delftdata/valentine/tree/master/examples) directory, including a [pandas example](https://github.com/delftdata/valentine/blob/master/examples/valentine_example_pandas.py), a [Polars example](https://github.com/delftdata/valentine/blob/master/examples/valentine_example_polars.py), and a [mixed pandas+Polars example](https://github.com/delftdata/valentine/blob/master/examples/valentine_example_mixed.py).

```python
import pandas as pd
from valentine import valentine_match
from valentine.algorithms import Coma

# Load data using pandas
df1 = pd.read_csv("source_candidates.csv")
df2 = pd.read_csv("target_candidates.csv")

# Instantiate matcher and run
matcher = Coma(use_instances=True)
matches = valentine_match([df1, df2], matcher)

# Iterate over results using ColumnPair named fields
for pair, score in matches.items():
    print(f"{pair.source_column} <-> {pair.target_column}: {score:.3f}")

# If ground truth available valentine could calculate the metrics
ground_truth = [
    ("emp_id", "employee_number"),
    ("fname", "first_name"),
    ("lname", "last_name"),
    ("dept", "department"),
    ("annual_salary", "compensation"),
    ("hire_date", "start_date"),
    ("office_loc", "work_location"),
]

metrics = matches.get_metrics(ground_truth)
print(metrics)
```

## Cite Valentine
```
Original Valentine paper:
@inproceedings{koutras2021valentine,
  title={Valentine: Evaluating Matching Techniques for Dataset Discovery},
  author={Koutras, Christos and Siachamis, George and Ionescu, Andra and Psarakis, Kyriakos and Brons, Jerry and Fragkoulis, Marios and Lofi, Christoph and Bonifati, Angela and Katsifodimos, Asterios},
  booktitle={2021 IEEE 37th International Conference on Data Engineering (ICDE)},
  pages={468--479},
  year={2021},
  organization={IEEE}
}
Demo Paper:
@article{koutras2021demo,
  title={Valentine in Action: Matching Tabular Data at Scale},
  author={Koutras, Christos and Psarakis, Kyriakos and Siachamis, George and Ionescu, Andra and Fragkoulis, Marios and Bonifati, Angela and Katsifodimos, Asterios},
  journal={VLDB},
  volume={14},
  number={12},
  pages={2871--2874},
  year={2021},
  publisher={VLDB Endowment}
}
```
