Metadata-Version: 2.4
Name: wikidata-ner-classifier
Version: 0.3.1
Summary: Deterministic Wikidata NER prediction with description-based branch-local refinement
Author: Roberto Avogadro
License-Expression: MIT
Project-URL: Homepage, https://github.com/roby-avo/ner-wikidata
Project-URL: Issues, https://github.com/roby-avo/ner-wikidata/issues
Project-URL: Source, https://github.com/roby-avo/ner-wikidata
Keywords: wikidata,ner,entity-linking,knowledge-graph,classification
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# Wikidata NER Classifier 0.3.1

Deterministic, explainable prediction of retrieval-oriented entity types from a
Wikidata QID, direct type-class labels, optional ancestor labels, and an entity
description.

The QID is retained as an identifier and is never used as a lookup key.

## What changed in 0.3.1

The classifier now exposes four progressively narrower retrieval levels:

```text
coarse_type -> fine_type -> subtype -> specific_type
```

It also returns every compatible narrow label in `specific_types` and keeps
orthogonal properties such as genre in `facets`.

The default evidence policy is now:

1. `types[].name` selects `coarse_type` and `fine_type`.
2. `ancestor_types[].name`, when supplied, provides lower-weight class ancestry.
3. Direct type labels and `description` refine only the selected branch.
4. `context_string` is ignored by the classifier by default because it often
   contains related people, organizations, countries, genres, and formats.
5. Description evidence cannot change a `FILM` branch into a location, company,
   person, or another unrelated branch.
6. Unsupported specificity is not invented.

The packaged configuration contains:

- 187 fine-type rules;
- 295 structural subtype rules;
- 33 controlled facet rules;
- 6 branch-local composite-type templates.

## Installation

```bash
python -m pip install wikidata-ner-classifier
```

Python 3.10 or newer is required. The library has no runtime dependencies.

## Basic use

```python
from wikidata_ner import WikidataNERClassifier

classifier = WikidataNERClassifier()

prediction = classifier.predict(
    qid="Q3441181",
    types=[{"id": "Q11424", "name": "film"}],
    description="1964 sword-and-sandal film directed by Giuseppe Vari",
)

print(prediction.to_dict())
```

Relevant output:

```json
{
  "coarse_type": "CREATIVE_WORK",
  "fine_type": "FILM",
  "subtype": null,
  "specific_type": "SWORD_AND_SANDAL_FILM",
  "specific_types": [
    "SWORD_AND_SANDAL_FILM"
  ],
  "facets": {
    "genre": [
      "SWORD_AND_SANDAL"
    ]
  },
  "refinement_sources": [
    "description"
  ]
}
```

The description adds specificity only inside the already established `FILM`
branch.

## Alpaca or Elasticsearch entities

Both source objects and complete Elasticsearch hits are accepted:

```python
prediction = classifier.predict_entity(hit_or_source)
```

```python
{
  "qid": "Q3441181",
  "types": [{"name": "film"}],
  "description": "1964 sword-and-sandal film directed by Giuseppe Vari"
}
```

```python
{
  "_id": "Q3441181",
  "_source": {
    "qid": "Q3441181",
    "types": [{"name": "film"}],
    "description": "1964 sword-and-sandal film directed by Giuseppe Vari"
  }
}
```

A complete Elasticsearch response can be processed with:

```python
predictions = classifier.predict_elasticsearch_response(response)
```

## Why both subtype and specific type exist

A subtype describes a structural kind. A facet describes an independent
characteristic. A specific type is a retrieval-oriented composition.

```python
prediction = classifier.predict(
    "Q1",
    [
        {"name": "film"},
        {"name": "feature film"},
        {"name": "comedy film"},
    ],
)
```

This can produce:

```json
{
  "fine_type": "FILM",
  "subtype": "FEATURE_FILM",
  "specific_type": "FEATURE_FILM",
  "specific_types": [
    "FEATURE_FILM",
    "COMEDY_FILM"
  ],
  "facets": {
    "genre": [
      "COMEDY"
    ]
  }
}
```

`FEATURE_FILM` and `COMEDY_FILM` are compatible. They may be combined by the
retriever rather than forced into a single mutually exclusive label.

## Example refinements from the Alpaca query

| Type labels | Description | Fine type | Subtype | Most specific retrieval type |
|---|---|---|---|---|
| `film` | `1951 film directed by Luigi Zampa` | `FILM` | none | `FILM` |
| `film` | `1964 sword-and-sandal film ...` | `FILM` | none | `SWORD_AND_SANDAL_FILM` |
| `album` | `album by Holger Czukay` | `MUSICAL_WORK_SONG_OR_ALBUM` | `MUSIC_ALBUM` | `MUSIC_ALBUM` |
| `literary work` | `Alternative history, military science fiction story` | `BOOK_OR_WRITTEN_WORK` | `FICTION_STORY` | `MILITARY_SCIENCE_FICTION_LITERARY_WORK` |
| `pencil drawing` | `1953 work of art ...` | `VISUAL_ARTWORK_PHOTOGRAPH_OR_COMIC` | `PENCIL_DRAWING` | `PENCIL_DRAWING` |

A generic description cannot justify an invented subtype. A generic film remains
`FILM` when neither its type labels nor description contain a safe refinement.

## Retrieval indexing helpers

Store the prediction alongside each entity using stable keyword fields:

```python
fields = prediction.to_retrieval_fields(prefix="ner")
```

Example fields:

```json
{
  "ner_coarse_type": "CREATIVE_WORK",
  "ner_fine_type": "FILM",
  "ner_subtype": null,
  "ner_specific_type": "SWORD_AND_SANDAL_FILM",
  "ner_specific_types": [
    "SWORD_AND_SANDAL_FILM"
  ],
  "ner_facets": {
    "genre": [
      "SWORD_AND_SANDAL"
    ]
  }
}
```

A deterministic Elasticsearch filter can be generated with:

```python
query_filter = prediction.elasticsearch_filter(
    field="ner_specific_types",
    require_all=True,
)
```

For several compatible specific types, `require_all=True` emits one term filter
per type. Use `require_all=False` to emit a `terms` disjunction.

Index-time and query-time predictions should use the same library and rule-file
version.

## Description and context controls

Description refinement is enabled by default:

```python
classifier = WikidataNERClassifier(
    use_description_for_refinement=True,
    use_context_string_for_refinement=False,
)
```

Disable it when only class labels should be considered:

```python
classifier = WikidataNERClassifier(
    use_description_for_refinement=False,
)
```

Noisy context refinement is available only as an explicit opt-in:

```python
classifier = WikidataNERClassifier(
    use_context_string_for_refinement=True,
)
```

The separate `use_description=True` option allows description text to add
low-weight support to the primary coarse/fine scorer. It is disabled by default.
Descriptions therefore do not rescue a missing or unknown type anchor unless the
caller explicitly changes that policy.

## Live Alpaca notebook

Open `examples/alpaca_live_test.ipynb`.

The notebook:

1. issues the supplied Alpaca Elasticsearch request;
2. extracts QID, type labels, and description;
3. ignores `context_string` during classification;
4. displays `predicted_subtype`, `predicted_specific_type`, all compatible
   `specific_types`, facets, and confidence values;
5. demonstrates a retrieval filter generated from the prediction.

Set the bearer token before starting Jupyter:

```bash
export ALPACA_TOKEN='your-token'
```

The notebook also supports a hidden token prompt when the environment variable is
not set.

## CLI

```bash
wikidata-ner response.json > predictions.json
cat response.json | wikidata-ner
```

Relevant flags:

```text
--no-description-refinement
--context-refinement
--description-for-primary
--entity-label
```

## Validation

The source package includes unit tests for:

- direct type-label classification;
- description-only branch refinement;
- context exclusion by default;
- explicit context opt-in;
- subtype and facet compatibility;
- generic fallback behavior;
- QID independence;
- Elasticsearch hit input;
- retrieval-field generation;
- generated Elasticsearch filters.


## Mention-focused OpenRouter notebook

`examples/openrouter_mention_focused_ner.ipynb` classifies one explicit target
mention at a time from free text, structured records, or table cells. Context is
used only as evidence for that target. Contextual free text requires `mention=`
or an exact span; table helpers make the selected cell the target.
