The data files in this directory are licensed under the Creative Commons
Attribution 4.0 International License (CC BY 4.0):
https://creativecommons.org/licenses/by/4.0/

Copyright (c) 2026 Ali Julaee Rad. Each file's header names what it was
derived from.

These files quote text from Wikipedia, Wikisource and Ganjoor, or are derived
from the Tajik Shahnameh corpus (ArabovMK/shahnameh-tajik-corpus), and are
licensed under CC BY-SA 4.0 instead (https://creativecommons.org/licenses/by-sa/4.0/):
ezafe-written-dev.jsonl, ezafe-junctions-dev.jsonl, ezafe-lab-1.jsonl,
arabic-spans-lab.jsonl, arabic-lines-lab.jsonl, shahnameh-tajik-test.jsonl.
Each quoted row gives its page's title and url; the authors of a Wikipedia or
Wikisource page are in that page's history. Changes: a short passage is cut
from the page, markup removed, and in the ezafe files the mark under test
removed. The Wikisource and Ganjoor passages are from laws, parliamentary
records and works whose authors died by 1955.

ezafe-heh-pairs.tsv, ezafe-heh-left.tsv and ezafe-heh-words.tsv hold word and
word-pair counts, not passages. Their sources (social media, comments, forums,
news sites, FineWeb2, Persian Wikipedia and Wikibooks) are named in each header;
most state no licence and none of their text is redistributed. Hashtags and
long numbers are left out.

ezafe-junction-counts.tsv.gz is CC BY-SA 4.0: word and word-pair counts from
Persian Wikipedia, Wikibooks, Wikivoyage and Wiktionary (CC BY-SA 4.0), Ganjoor
and Common Voice (CC0).

arabic-model.json.gz is CC BY-SA 4.0: word and character statistics and model
parameters, no source passages and no copy of the Quran. Built from arwikisource,
arwikibooks, Aya, Ganjoor, fawikisource, Dehkhoda, books and fawiki; Quran
wording found with Tanzil's text marked where Arabic is quoted; calibrated on
lines from fawikisource, Naskban, books, Ganjoor, fawiki, fawikiquote, Tasnim,
IRIB News, Mizan, HPLT, Virgool, Telegram channels. Its provenance field lists
each source's recorded licence.

corpus-sample.jsonl.gz is the Sample tier of the Ghazal Corpus 1.0.0, 500
documents, each under the licence in its own `license` field: CC BY-SA 4.0
(Wikimedia text; its authors are in the history of each document's url), CC0 1.0
(Common Voice) or public domain (Ganjoor poems whose poets died long ago).
corpus-sample.json is its manifest, with the full attribution statement.

verb-stems.tsv and verb-preverbs.tsv include lexical facts checked against
English Wiktionary (Wiktionary contributors, via the Kaikki Persian extract)
and are distributed under CC BY-SA 4.0:
https://creativecommons.org/licenses/by-sa/4.0/
https://en.wiktionary.org/
https://kaikki.org/dictionary/Persian/
Dehkhoda infinitive entries were also consulted for stem facts; no dictionary
definitions or corpus passages are reproduced in these tables.

orthography-splits.tsv and orthography-copula.tsv are licensed under CC BY-SA 4.0
instead of the default above. Both contain word forms and
counts from rows of the Ghazal Corpus ingestion (2026-11-09),
not passages. Sources: Persian Wikipedia (CC BY-SA 4.0), Mizan parallel
corpus (CC BY 4.0 dataset), IRIB news, Blogfa, Ganjoor, the books collection
and persian-tweets-2024. The latter five have no licence recorded in the
ingestion; their text is not redistributed. No Tanzil or Sokhan dictionary
material is used. Count selection and thresholds are in each file's header.
orthography-splits.tsv uses the package's verb-stems.tsv, verb-preverbs.tsv
and verb-colloquial.tsv for its finite-verb gate; their inherited sources
include standard paradigms and John Mace's Persian Grammar, Appendix I.
The corpus behind the legacy preverb counts is unspecified.

orthography-function-splits.tsv, orthography-content-splits.tsv,
and orthography-he-endings.tsv are also licensed
under CC BY-SA 4.0 instead of the default. All three use counts from fawiki,
mizan, irib-news, blogfa, books, persian-tweets-2024 and ganjoor. The first
two also use common-word/name evidence from the training splits of UD
Persian Seraji and PerDT (CC BY-SA 4.0 datasets). orthography-he-endings.tsv
and orthography-copula.tsv use the existing orthography-silent-he.tsv as
a candidate gate; that legacy file records 2026 corpus spelling counts
without a complete source list. Each file's header identifies its inputs.

orthography-mi-forms.tsv and orthography-clitic-hosts.tsv are licensed under
CC BY-SA 4.0. Both use word-form counts from fawiki, mizan,
irib-news, blogfa, books, persian-tweets-2024 and ganjoor rows, with the
source licences listed above. No development or test sentences supply counts.
orthography-context-names.tsv is licensed under CC BY-SA 4.0 and uses only
training rows from UD Persian Seraji and PerDT (CC BY-SA 4.0).

vowels-lexicon.tsv, vowels-model.json.gz, vowels-sources.json,
vowels-review.tsv and vowels-attribution.jsonl.gz are combined CC BY-SA 4.0
derivatives. They contain word forms, counts, parameters and attribution
metadata, with no definitions, sentences or poem passages. Underlying CC0 and
public-domain contributions retain those terms. The attribution manifest
identifies contributing document IDs, Wikimedia history URLs and revision IDs
where available, and exact input snapshots by filename, byte size and SHA-256.
Shared input files are counted once in that manifest.

Sources: Ganjoor historical verse (local licence unstated), DehkhodaProject
(the scraper's release is CC0; rights in the dictionary itself are not
established, and only headwords and their readings are used,
https://github.com/DehkhodaProject/DehkhodaProject), Steingass's 1892
dictionary (public domain; read from the Digital Dictionaries of South Asia
edition, https://dsal.uchicago.edu/dictionaries/steingass/, which is offered
under CC BY-NC-ND 2.0, through the steingass-scraper database, whose code
licence is below; only headwords and their readings are used), English Wiktionary via Kaikki
(https://kaikki.org/dictionary/Persian/), Persian and Tajik Wiktionary, Persian
Wikipedia, Wikisource, Wikibooks, Wikiquote and Wikivoyage (CC BY-SA 4.0),
Persian Wikinews (CC BY 2.5), Common Voice (CC0), HomoRich-G2P-Persian
(CC0, https://huggingface.co/datasets/MahtaFetrat/HomoRich-G2P-Persian;
recommended training split only), Hedayat stories and Mohl's Shahnameh edition
(public-domain works; digital witness licences unstated), non-Sokhan Naskban
and reference OCR editions (printed edition licences unstated), and Mizan
(CC BY 4.0 compilation; underlying translations separately protected).

OCR editions and Mizan contribute extracted lexical facts only. This licence
covers the combined derivative where applicable; it does not relicense the
underlying books, translations or digital witnesses. Their individual source
IDs and licence descriptions remain in vowels-sources.json. Sokhan, Tanzil and
Tashkeela are excluded. Public benchmark text is used only to exclude training
sentence matches and to measure the finished artifacts; no benchmark labels
enter training or calibration. Dictionary headword overlap is expected.

Steingass digital transcription: steingass-scraper, snapshot d2fd36bf577b.
MIT License
Copyright (c) 2024 T. S. Beers

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

vowels-context.json.gz is also CC BY-SA 4.0. It holds word forms, counts and
context parameters from HomoRich training, Ganjoor editorial marks, Hedayat
stories, Persian Wikisource and Wikipedia. Reviewed lexical sense families
are included as word facts. The fallback uses the word lexicon listed above;
its separately calibrated scores are stored without a second copy of the lexicon.
Controlled grammar training/calibration examples are original CC0 material;
only their derived parameters are included. Biological compound readings are
normalized during training; public benchmark reference phonemes are unchanged.
Its provenance object names source snapshots, licences and the lexical input hash.
No sentences or poem passages are included. SentenceBench and HomoRich test
are exclusion and measurement inputs only. Held-out documents, development
documents and matching public sentences are excluded from training. Dictionary
headword exclusions apply at dictionary entry reads; they do not remove that
word's independent corpus evidence. Authored evaluation sentences supply no
training or calibration labels.
