Metadata-Version: 2.5
Name: pyannotators-entityfishing
Version: 1.6.83
Summary: Annotator based on entity-fishing
Project-URL: Homepage, https://github.com/oterrier/pyannotators_entityfishing/
Author-email: Olivier Terrier <olivier.terrier@kairntech.com>
License: MIT
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Software Development :: Libraries :: Application Frameworks
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.12
Requires-Dist: collections-extended
Requires-Dist: mongoquery
Requires-Dist: pydantic<3.0,>=2.0
Requires-Dist: pymultirole-plugins<1.7.0,>=1.6.0
Requires-Dist: python-singleton-metaclasses
Requires-Dist: requests
Requires-Dist: requests-cache
Requires-Dist: requests-futures
Provides-Extra: dev
Requires-Dist: bump2version; extra == 'dev'
Requires-Dist: pre-commit; extra == 'dev'
Provides-Extra: docs
Requires-Dist: lxml-html-clean; extra == 'docs'
Requires-Dist: myst-parser; extra == 'docs'
Requires-Dist: sphinx; extra == 'docs'
Requires-Dist: sphinx-rtd-theme; extra == 'docs'
Requires-Dist: sphinxcontrib-apidoc; extra == 'docs'
Provides-Extra: sbom
Requires-Dist: cyclonedx-bom; extra == 'sbom'
Requires-Dist: pip-audit; extra == 'sbom'
Provides-Extra: spacy
Requires-Dist: spacy>=3.0; extra == 'spacy'
Provides-Extra: test
Requires-Dist: dirty-equals; extra == 'test'
Requires-Dist: pytest; extra == 'test'
Requires-Dist: pytest-cov; extra == 'test'
Requires-Dist: ruff; extra == 'test'
Description-Content-Type: text/markdown

# pyannotators-entityfishing

Annotator based on [entity-fishing](https://github.com/kermitt2/entity-fishing) for named entity recognition and disambiguation against Wikidata.

## Installation

```bash
pip install pyannotators-entityfishing
```

For noun-form filtering (optional):

```bash
pip install pyannotators-entityfishing[spacy]
python -m spacy download en_core_web_sm  # or other language models
```

## Usage

```python
from pymultirole_plugins.v1.schema import Document
from pyannotators_entityfishing.entityfishing import EntityFishingAnnotator, EntityFishingParameters

annotator = EntityFishingAnnotator()
parameters = EntityFishingParameters(
    default_label="ENTITY",
    minSelectorScore=0.3,
)

docs = annotator.annotate(
    [Document(text="Albert Einstein was born in Ulm.", metadata={"language": "en"})],
    parameters,
)

for ann in docs[0].annotations:
    print(f"{ann.start}:{ann.end} {ann.labelName} {ann.terms[0].identifier}")
```

## Linking candidates (processor `entityfishing_candidates`)

The package also registers a **processor**, `entityfishing_candidates` (group `pyprocessors.plugins`), which does not decide the link: it writes, on the mentions of an upstream NER, the Wikidata concepts each one may refer to, for a decision step (Jev, `pyprocessors_jev`) to choose among them. It implements step 2 of [ADR-0001](docs/adr/0001-linking-candidats-entityfishing-decision-jev.md), and does not call `/disambiguate`, whose selector drops the right candidate, or answers nothing, when its prior is low.

For each group of mentions of the same entity (`properties.entity_group`, written by `pyprocessors_coreference`, else the same label and surface):

1. term lookup (`/kb/term`) on the group's `canonical_form`, falling back on its surfaces: the senses, most frequent first, without those under `min_prior`, up to `top_k`;
2. the concepts of those senses (`/kb/concept`, by Wikipedia page id, in the document's language);
3. the `mapped_labels` expression of the mention's label keeps only the concepts of the right type;
4. every mention of the group gets the candidates in `terms`: `identifier` = QID, `lexicon` = `wikidata`, `preferredForm`, `score` = the prior (`prob_c`), and in `properties` the `wikidataId`, `prob_c`, `pageid`, the first sentence of the `definition` and the requested Wikidata `facts`.

The mentions themselves (span, label) are never changed; a mention with no candidate keeps empty `terms`; terms of other lexicons (AFP...) are kept. It is a processor and not an annotator because Sherpa sends an annotator the text and metadata only, never the annotations of the NER before it.

```python
from pymultirole_plugins.v1.schema import Annotation, Document

from pyannotators_entityfishing.candidates import EntityFishingCandidatesParameters, EntityFishingCandidatesProcessor

text = "Macron à Washington."
document = Document(
    text=text,
    metadata={"language": "fr"},
    annotations=[
        Annotation(start=0, end=6, labelName="Person", text="Macron"),
        Annotation(start=9, end=19, labelName="Location", text="Washington"),
    ],
)
[document] = EntityFishingCandidatesProcessor().process([document], EntityFishingCandidatesParameters(top_k=3))
for ann in document.annotations:
    print(ann.text, [(t.identifier, t.preferredForm, t.score) for t in ann.terms])
```

| Parameter | Default | Description |
|-----------|---------|-------------|
| `labels` | all | Labels of the mentions that get candidates |
| `mapped_labels` | none | Label → mongo-query on the Wikidata concept (as for the annotator, e.g. `tests/data_v2`); a label with no expression is not filtered |
| `top_k` | `8` | Maximum number of candidates per mention |
| `min_prior` | `0.01` | Drop the senses whose prior is above 0 but below this value: mostly Wikipedia link errors (0.0004 for Nicolas Sarkozy as "Emmanuel Macron"), each one costing tokens at the decision step. A prior of exactly 0 is a match on a page title or redirect and is always kept. 0.01 is entity-fishing's own `minSenseProbability` |
| `wikidata_properties` | none | Comma-separated Wikidata properties copied into `properties.facts` |
| `ef_uri` | `APP_EF_URI` | Base URL of the entity-fishing service |

## Development

The build is driven by [Task](https://taskfile.dev) and [uv](https://docs.astral.sh/uv/),
with the shared stages coming from the `python-archetype` submodule.

### Getting started

The stages live in a Git submodule, so **clone with `--recurse-submodules`**:

```bash
git clone --recurse-submodules git@bitbucket.org:kairntech/pyannotators_entityfishing.git
cd pyannotators_entityfishing
sh -c "$(curl -sSL https://taskfile.dev/install.sh)" -- -d -b ~/.local/bin
task
```

Already cloned without it? The submodule directory is empty, and `task` fails on:

```
task: No Taskfile found at ".../submodules/python-archetype/resources/Taskfile.yml"
```

which means exactly that, and nothing worse:

```bash
git submodule update --init
```

**Task is the only manual prerequisite.** An archetype cannot bootstrap itself: uv and the
Python interpreter install themselves on demand (every task that runs uv depends on an
internal `install-python` task), but the thing that runs them does not. Make sure
`~/.local/bin` is on your `PATH` — that is where `task` and `uv` both land.

### Running the pipeline

```bash
task stages          # print the pipeline stages, in order
task                 # run the pipeline up to (but excluding) py:publish
task -- --skip-tests # same, without the test stage
task up-to -- py:lint # run the pipeline up to and including one stage
task jenkins         # run every stage, exactly what Jenkins runs
```

`task` with no argument is safe by construction: it runs every stage but the last, and that
bound is computed from the `STAGES` list rather than written down. The last stage is the
only one with an effect outside your machine.

`STAGES`, declared once in `Taskfile.yml`, is the single definition of the pipeline order —
so what you run locally is what Jenkins runs.

### Individual stages

| Task                            | Description                                           |
|---------------------------------|-------------------------------------------------------|
| `task py:sync`                  | Install the project and its dependencies (uv sync)    |
| `task py:lint`                  | `ruff check` and `ruff format --check`                |
| `task py:format`                | Reformat the code with ruff                           |
| `task py:test`                  | Run the test suite                                    |
| `task py:test-marker -- <m>`    | Run the tests carrying one pytest marker              |
| `task py:sbom`                  | Generate a CycloneDX SBOM of the resolved environment |
| `task py:check-vulnerabilities` | Check for known CVEs                                  |
| `task py:check-updates`         | Check for dependency updates                          |
| `task py:build`                 | Build the wheel and sdist (uv build)                  |
| `task py:publish`               | Publish the distributions (uv publish)                |
| `task py:version-file`          | Print the path of the file carrying `__version__`     |
| `task py:set-version VERSION=x` | Write that version into it                            |

`uv.lock` is not versioned here, so `py:sync` always resolves from scratch (`--upgrade`):
a stale lock lying around on a machine would otherwise make you test and audit versions the
CI never sees.

### SBOM & vulnerability check

`task py:sbom` and `task py:check-vulnerabilities` wrap the underlying tools. To run them
by hand:

```bash
uv sync --extra test --extra sbom
uv run cyclonedx-py environment -o sbom.cdx.json --output-format json
uv run pip-audit --skip-editable --format json --output audit-report.json
```

Ask for **all** the extras, not just `sbom`: `uv sync` synchronises rather than adds, so
whatever the requested extras do not pull in gets removed. `--extra sbom` on its own
uninstalls `ruff`, `pytest-cov`, `coverage` and `dirty-equals`, leaving an environment that
can no longer lint or measure coverage.

`--skip-editable` skips the project itself, which no advisory database can know about. The
tasks do the same, and pip-audit still exits non-zero when it finds a real vulnerability —
which is the contract the CI relies on. Avoid `--strict`: it turns "dependency not found on
PyPI" into a fatal error, so it breaks as soon as the local version is not a published one.
