Metadata-Version: 2.4
Name: shoshan
Version: 0.4.0
Summary: A zero-hallucination Hebrew lemmatizer (retrieve, then transduce).
Author: Noam Ordan
License: MIT
Project-URL: Homepage, https://github.com/ivrit/shoshan
Project-URL: Weights, https://huggingface.co/HebArabNlpProject/shoshan
Project-URL: Demo, https://huggingface.co/spaces/noamor/shoshan-demo
Keywords: hebrew,lemmatization,nlp,retrieval,morphology
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: torch>=2.0
Requires-Dist: transformers>=4.30
Requires-Dist: numpy
Requires-Dist: pandas
Requires-Dist: huggingface_hub>=0.20
Dynamic: license-file

# Shoshan

A Hebrew lemmatizer that does not hallucinate.

Give it a word in a sentence and it returns the word's lemma — its dictionary
form. The trick is that Shoshan never invents a word. It first **retrieves** the
lemma from a fixed bank of real Hebrew lemmas. When the best match does not look
like it could be a form of the input word, it falls back and **transduces** the
lemma by editing the input word itself (stripping a prefix, fixing a suffix).
Either way the output is a real lemma or a bounded edit of the word you gave it.
It cannot return a string that came from nowhere.

- **Weights:** <https://huggingface.co/HebArabNlpProject/shoshan>
- **Demo:** <https://huggingface.co/spaces/noamor/shoshan-demo>

## Install

```bash
pip install git+https://github.com/ivrit/shoshan.git
```

The weights (about 1 GB) download from the Hugging Face Hub the first time you
load the model and are cached after that.

## Use it

Three entry points, one per unit of text:

| call | input | reach for it when you have… |
|---|---|---|
| `lemma(word, sentence)` | one word in context | a single look-up |
| `annotate(sentence)` | one sentence | a sentence to tag |
| `lemmatize_text(source)` | a string, file, or folder | a **whole document** to lemmatize or **index for search** |

### One word, one sentence

```python
from shoshan import Lemmatizer

lz = Lemmatizer.from_pretrained()   # downloads + caches

# lemma(word, sentence): the lemma of that one word,
# read in context; the rest of the sentence is left alone.
lz.lemma("המחברות", "המורה חילקה את המחברות לתלמידים בכיתה.")   # 'מחברת'
lz.lemma("בספרו", "הוא כתב על כך בספרו האחרון.")       # 'ספר'

# annotate(sentence): lemmatize every word in the sentence.
for tok in lz.annotate("הילדים שיחקו בגן"):
    print(tok["form"], "→", tok["lemma"], tok["source"])
```

The Hebrew above, romanized for non-Hebrew readers (Modern-Israeli phonemic):
*המחברות* ha-makhbarot ('the notebooks') → *מחברת* makhberet; *בספרו* bi-sfaro ('in his
book') → *ספר* sefer; and *הילדים שיחקו בגן* ha-yeladim sikhaku ba-gan ('the
children played in the garden') → *ילד* yeled, *שיחק* sikhek, *גן* gan.

`source` is `"retrieved"` when the lemma came from the bank and `"transduced"`
when the edit-script fallback produced it (this is how out-of-vocabulary words
are handled).

For information retrieval you usually drop closed-class function words (prepositions,
pronouns, conjunctions, …). Pass `blank_function_words=True` and those tokens come
back with an empty lemma and `source="function"`, so you can skip them:

```python
lz = Lemmatizer.from_pretrained(blank_function_words=True)
```

### Whole documents and search indexing

`lemmatize_text(source)` is the call for **complete texts**. Give it a raw
string, a file path, or a folder path; it segments paragraphs → sentences →
word tokens and returns a **doc dict**:

```python
doc = lz.lemmatize_text("הילדים שיחקו בגן. הם בנו ארמון בחול.")

doc["analyzed_text"]   # the lemmas in order, as one string
doc["tokens"][0]                          # one record per word (shape):
# {'token': 'הילדים', 'start': 0, 'end': 6, 'lemma': 'ילד',
#  'pos': 'NOUN', 'source': 'bypass', 'score': ..., 'sent_id': 0}
```

The doc dict has five keys:

| key | what it holds |
|---|---|
| `text` | the original input, echoed back |
| `tokens` | one record per word: `token, start, end, lemma, pos, source, score, sent_id` |
| `analyzed_text` | the lemmas in order, space-joined into one string |
| `es_tokens` | an Elasticsearch [`_analyze`](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-analyze.html)-style stream, ready to index |
| `unknown` | out-of-bank words worth annotating (deduped, with counts) |

**Character offsets round-trip.** Every token carries absolute offsets into
`text`, so `text[start:end] == token` — slice the original to highlight a hit
without re-tokenizing. The same holds for `annotate(sentence)`: each row's
`start` indexes the sentence you passed in, so
`sentence[start:start + len(form)] == form`.

**Ready for Elasticsearch.** Each `es_tokens` entry is a lemma with its source
span and position — `{token, start_offset, end_offset, position, type}` — the
same shape Elasticsearch's `_analyze` emits, so you can feed it straight into an
index and search by lemma while highlighting the original surface text.

**`source` tells you where each lemma came from:**

- `retrieved` — pulled from the lemma bank,
- `transduced` — produced by the edit-script fallback (out-of-vocabulary words),
- `bypass` — retrieved from the bank with the coverage gate **skipped**, because the
  form is a pronoun or one of a list of high-frequency closed-class words whose lemma
  the gate reliably distrusts for the wrong reason. This is common: on ordinary text it
  is close to half of all tokens,
- `acronym` — a known Hebrew acronym, kept as its own lemma with the coverage gate
  skipped (`צה"ל` → `צה"ל`),
- `suppletive` — a curated look-up for irregular / closed-class forms whose
  lemma shares too few letters to retrieve (e.g. *היא* → *הוא*),

  > **Lemma convention for pronouns.** Following the MILA canonical-lemma
  > convention, **personal pronouns lemmatize to הוא** — not only *היא*, *הם*
  > and *הן*, but also *אני*, *אתה* and *אנחנו*. This is deliberate, not a bug:
  > the pronoun paradigm is treated as one lexeme with הוא as its citation
  > form. If you need person or number, read them from the surface form or
  > from `pos`, not from the lemma.
  >
  > The look-up is keyed on the surface form **and the predicted POS**, so it
  > applies when the word is actually tagged as a pronoun. In a sentence where
  > the POS head mis-tags one — ill-formed agreement, say — the form comes back
  > unchanged instead. Do not rely on the mapping as an invariant over arbitrary
  > text; check `pos` alongside it.
- `function` — only with `blank_function_words=True`: closed-class stopwords
  come back with an empty lemma and are kept in `tokens` (for provenance) but
  dropped from `es_tokens` and `analyzed_text`.

**Files and folders.** Pass a file path and you get one doc dict back with a
`path` key added; pass a folder and you get `{relative_path: doc dict}` for
every text file under it (`files_glob="*.txt"`, recursive by default):

```python
doc  = lz.lemmatize_text("notes.txt")      # one doc dict, plus doc["path"]
docs = lz.lemmatize_text("corpus/")        # {relative_path: doc dict}
```

From the command line:

```bash
shoshan "המורה חילקה את המחברות לתלמידים בכיתה"
shoshan --csv input.csv output.csv      # input columns: form,sentence[,pos]
```

## How it works

One DictaBERT encoder produces a vector for the target word in its sentence.
Three things hang off that vector:

1. **Retrieval** — cosine similarity against a bank of ~118k pre-encoded lemmas.
2. **A coverage gate** — checks whether the retrieved lemma's characters actually
   appear, in order, inside the surface word. If they do not, the retrieval is
   probably wrong (an unknown word), so the router does not trust it.
3. **An edit-script head** — a learned, rule-free transformation of the *word*
   into its lemma, used only when the gate distrusts retrieval.

Two routes skip step 2 deliberately, and between them they cover a large share of
ordinary text: a curated suppletive look-up and a known-acronym look-up answer directly
(`source` `suppletive` / `acronym`), and a list of pronouns and high-frequency
closed-class forms accepts retrieval without the coverage check (`source` `bypass`).
Those forms are exactly the ones whose lemma shares too few letters with the surface for
the gate to judge fairly. One consequence is worth knowing if you curate: `write_miss_log`
only ever sees tokens the gate actually judged, so a bypassed or suppletive token can
never appear in the worklist.

Because step 3 can only delete from or affix to the input word, the system has a
bounded output: it never produces an unrelated string. Adding vocabulary is just
adding rows to the bank and re-encoding them — no retraining.

### Finding words to curate

The system tells you where it is unsure. A token is flagged when the bank's best lemma is
morphologically implausible for the surface form (the coverage gate distrusts the
retrieval) — i.e. a likely out-of-vocabulary word. Turn on `log_misses` and the model
collects these; `write_miss_log` writes them as a **frequency-sorted worklist** — the
word-forms most worth annotating or adding to the lexicon, commonest first.

```python
lz = Lemmatizer.from_pretrained(log_misses=True)
lz.lemmatize(corpus_rows)               # run over your text
lz.write_miss_log("to_curate.csv")      # see the columns below
```

or from the command line:

```bash
shoshan --csv corpus.csv out.csv --miss-log to_curate.csv
```

The worklist has eight columns, one row per (wordform, predicted POS) pair:

| column | what it holds |
|---|---|
| `wordform` | the surface form, as it appeared in your text |
| `predicted_pos` | the POS the model assigned it |
| `count` | how often the pair occurred; the file is sorted by this, commonest first |
| `predicted_lemma` | the lemma the system finally returned |
| `retrieved_lemma` | the bank's best candidate, which the coverage gate distrusted |
| `mean_coverage` | mean coverage-gate score over the occurrences |
| `mean_sim` | mean retrieval similarity over the occurrences |
| `reason` | why it was flagged (the commonest reason, when they differ) |

Since extending the system is just adding lemmas to the bank and re-encoding (no
retraining), this closes the loop: the model surfaces its own gaps, you curate the
highest-frequency ones, and coverage improves without a training step.

## Results

The released model is trained only on the openly redistributable **Knesset +
Wikipedia** portions of the IAHLT Hebrew UD treebank, plus public lexicons. It is
evaluated on held-out domains it never saw in training.

**Exact-match lemma accuracy** (Shoshan):

| split | overall | seen form | unseen form |
|---|---|---|---|
| in-domain | 94.3% | 95.4% | 85.6% |
| out-of-domain | 92.4% | 95.1% | 81.2% |

Accuracy on words whose surface form was seen in training transfers almost intact
across domains (~95%); the gap sits in the genuinely unseen tail, which is exactly
where the edit-script transducer operates.

**Versus DictaBERT-lex** (out-of-domain). Exact match is not a fair cross-system
metric here: the two systems follow different lemma conventions (article stripping,
multi-word handling), so scoring one against the other's gold understates it. We
compare instead on convention-invariant **B³ consistency** (do inflections of a
word cluster together?) and on **low-overlap errors** (predictions that share too
little with the input word to be one of its forms).

| metric (out-of-domain) | Shoshan | DictaBERT-lex |
|---|---|---|
| B³ precision | **0.965** | 0.906 |
| B³ recall | **0.953** | 0.932 |
| B³ F1 | **0.959** | 0.919 |
| Low-overlap errors, unseen words | **0.0%** | 12.3% |

The last row is the point of the design: on words neither system saw in training,
DictaBERT-lex — which predicts each lemma as a single token from its vocabulary —
rewrites about one in eight into a word the input could not have produced (or emits
an empty token), while Shoshan's bounded output rules that out.
DictaBERT-lex was trained on more data than we use here, including the domains we
hold out, so the comparison is conservative.

The training data and evaluation splits live in a companion dataset,
[`HebArabNlpProject/shoshan-data`](https://huggingface.co/datasets/HebArabNlpProject/shoshan-data), and
download on demand:

```python
from shoshan import data
df = data.load("ood")     # or "train" / "dev" / "test" / "oov" / "ood_bagatz" ...
```

Numbers and the full write-up are in the paper (in preparation).

### Consistency over canonical form

For information retrieval the exact lemma string barely matters. What matters is that
every inflection of a word lands on the *same* label and that different words stay apart.
That is what B³ measures, and it is why we report it — and why it is the fair way to compare
against a system that uses a different lemma convention. Surfaces that look nothing alike
collapse to one label:

```
הורדתי · מוריד · הורידו · יוריד   →   הוריד
```

(*horadeti · morid · horidu · yorid → horid* — four very different surface forms of
"download / take down", all conflated to one lemma.)

B³ is blind to the label itself: a system that consistently used a *non-standard* string for
this cluster would score exactly the same. In that sense the value is stemmer-like — what the
label is called does not matter, only that every form of a word reaches the same one. What B³
penalizes is the reverse: *splitting* one word across several labels. In this system that
happens only at the edit-script fallback, on a rare unseen form it cannot retrieve — it may
guess a stem that does not match the rest of the lexeme. That residue is why B³ recall is
0.953 and not 1.0, and richer context shrinks it by routing more forms back to clean
retrieval.

## About the name

*Shoshan* is for Even-Shoshan, the Hebrew dictionary, since a lemmatizer's whole
job is to hand you the dictionary entry behind an inflected word. It is also a
lily.

## License and credit

Code is MIT (`LICENSE`). The weights and data carry their own terms, recorded in
`NOTICE`: the encoder is fine-tuned from
[DictaBERT](https://huggingface.co/dicta-il/dictabert), and the data comes from the
IAHLT Hebrew UD treebank and public Hebrew lexicons; see `docs/DATA_STATEMENT.md`.
We thank **Avner Algom** and the **IAHLT** for the treebank data and for
permission to release the out-of-domain evaluation sentences.
