Third-party licences
====================

splintr's own code is MIT (see LICENSE). This file covers what splintr
redistributes but did not write: the bundled tokenizer vocabularies.

Each vocabulary is a separate crate under `crates/vocab-*`, carrying the
vocabulary's own licence text in its own LICENSE file and its provenance in its
README. That per-crate copy is the one that travels with the data; this file is
the index.

None of these is a model. A vocabulary is the list of byte strings and merge
ranks a tokenizer splits text on — repository data, not weights. Where an
upstream project licenses code and weights separately (DeepSeek does), it is the
code licence that applies.

Every family is also loadable from its own `tokenizer.json` at runtime, via
`from_json_path`, which redistributes nothing.

  crate                    vocabulary            source                                    licence
  ---------------------------------------------------------------------------------------------------------------
  splintr-vocab-cl100k     cl100k_base           OpenAI tiktoken encodings                 MIT
  splintr-vocab-o200k      o200k_base            OpenAI tiktoken encodings                 MIT
  splintr-vocab-whisper    Whisper               openai/whisper-large-v3                   Apache-2.0
  splintr-vocab-qwen       Qwen 2/3              Qwen/Qwen3-8B                             Apache-2.0
  splintr-vocab-mistral    Mistral V1/V2/V3      mistralai/Mistral-7B-Instruct-v0.3,       Apache-2.0
                                                 mistralai/Mistral-Nemo-Instruct-2407
  splintr-vocab-modernbert ModernBERT            answerdotai/ModernBERT-base               Apache-2.0
  splintr-vocab-deepseek   DeepSeek V3           deepseek-ai/DeepSeek-V3                   MIT (LICENSE-CODE)
  splintr-vocab-glm        GLM-4/4.5             zai-org/GLM-4.5                           MIT
  splintr-vocab-kimi       Kimi K2/K3            moonshotai/Kimi-K2-Instruct               Modified MIT
  splintr-vocab-llama3     Llama 3/3.1/3.2/3.3   meta-llama/Llama-3.1                      Llama 3.1 Community
  splintr-vocab-llama2     Llama 2, Code Llama   codellama/CodeLlama-7b-hf                 Llama 2 Community
  splintr-vocab-gemma2     Gemma 2               Google's Gemma 2 tokenizer.model          Gemma Terms of Use
  splintr-vocab-gemma3     Gemma 3               Google's Gemma 3 tokenizer.model          Gemma Terms of Use
  splintr-vocab-gemma4     Gemma 4               google/gemma-4-12B-it                     Apache-2.0

Three bundled families redistribute nothing and so appear in no row above:
`gpt-oss` states o200k_base's ranks, `phi4` and `olmo2` state cl100k_base's,
each id for id. What they add is a table of special-token names, which is
splintr's own code.


Conditions beyond MIT and Apache-2.0
------------------------------------

Four of these carry terms that do not stop at attribution. They are reproduced
here so they are visible before the dependency is added, not so they are
accepted on anyone's behalf.

Gemma 2 and Gemma 3 — Gemma Terms of Use
  Section 3.1 permits redistribution, on four conditions, and each crate
  carries what they require: a copy of the agreement (`LICENSE`), the notice
  text file naming ai.google.dev/gemma/terms (`NOTICE`), a prominent statement
  that the shipped `.spm` is a converted form of Google's `tokenizer.model`
  rather than the original (`NOTICE` and `README.md`), and notice to
  subsequent users that the Section 3.2 use restrictions apply and must be
  passed on.

  Gemma 4 is NOT under these terms. Google licenses it separately under
  Apache-2.0 (ai.google.dev/gemma/apache_2), and the Gemma Terms' own appendix
  lists Gemma 1, 2, 3 and EmbeddingGemma but not Gemma 4 — which is why it is
  its own crate rather than a fourth payload beside the others.

Llama 2 / Code Llama — Llama 2 Community License
  Llama 2 is licensed under the Llama 2 Community License, Copyright (c) Meta
  Platforms, Inc. All Rights Reserved. The full agreement is in
  `crates/vocab-llama2/LICENSE`.

  The same shape as the Llama 3.1 agreement below: redistributing the
  vocabulary carries the agreement with it, the "Built with Llama 2" and
  naming duties attach to a model created or improved with the materials
  rather than to shipping this file, and Meta requires a separate licence
  above 700 million monthly active users. Meta's Acceptable Use Policy applies
  throughout.

Llama 3 — Llama 3.1 Community License
  Llama 3.1 is licensed under the Llama 3.1 Community License, Copyright (c)
  Meta Platforms, Inc. All Rights Reserved. The full agreement is in
  `crates/vocab-llama3/LICENSE`.

  Redistributing the vocabulary carries the agreement with it, which is why the
  crate ships it. The agreement's other duties fall on what a licensee *does*
  with the materials rather than on shipping this file: displaying "Built with
  Llama" and prefixing a model's name with "Llama" apply to an AI model created
  or improved with them, and Meta requires a separate licence above 700 million
  monthly active users. Meta's Acceptable Use Policy applies throughout.

Kimi — Modified MIT
  MIT plus one clause: a commercial product above 100 million monthly active
  users or $20 million monthly revenue must display "Kimi K2" in its user
  interface. Full text in `crates/vocab-kimi/LICENSE`.


Notes on provenance
-------------------

OpenAI (cl100k_base, o200k_base)
  Published by OpenAI at
  https://openaipublic.blob.core.windows.net/encodings/<name>.tiktoken. These
  rank files are not in the openai/tiktoken repository, and OpenAI states no
  licence for them separately; the MIT licence reproduced in the crates is
  tiktoken's own. The same files are redistributed on the same footing by the
  `tiktoken-rs` and `bpe-openai` crates.

DeepSeek V3
  Licensed as code (LICENSE-CODE, MIT, reproduced in the crate) and weights
  (a separate Model Agreement). The tokenizer is repository code.

GLM-4/4.5
  The HuggingFace model card declares `license: mit` and the repository carries
  no licence file; the project's GitHub repository carries Apache-2.0. Both
  permit redistribution with notice. The discrepancy is recorded rather than
  resolved on their behalf.

Llama 2 / Code Llama
  Taken from `codellama/CodeLlama-7b-hf`, not from a Llama 2 repository: Meta
  publishes Llama 2's `tokenizer.model` only behind a gated repository, which
  cannot be read without accepting terms on an individual's behalf. Code Llama's
  is ungated, carries the Llama 2 Community License in the same repository, and
  extended Llama 2's SentencePiece model in place — its first 32,000 ids are
  Llama 2's pieces and Llama 2's scores exactly, checked against Meta's file
  (md5 `eeec4125e9c7560836b4873b6f8e3025`) as republished verbatim by
  `TinyLlama/TinyLlama-1.1B-Chat-v1.0`. Only Code Llama's file ships; the
  Llama 2 vocabulary is its first 32,000 lines, derived at build time.

Gemma
  Gemma 2's and Gemma 3's `tokenizer.model` are published only behind gated
  repositories. Gating governs who may obtain the file, not who may pass it on:
  Section 3.1 permits redistribution outright, so the two are bundled under the
  conditions recorded above. Both are also published standalone, converted to
  the same `.spm` text, at huggingface.co/fs90/gemma-2-tokenizer-spm and
  .../gemma-3-tokenizer-spm.

  Gemma 4 needed none of that: `google/gemma-4-12B-it` is ungated and
  Apache-2.0, and its `tokenizer.json` is shipped byte for byte (SHA-256
  cc8d3a0ce36466ccc1278bf987df5f71db1719b9ca6b4118264f45cb627bfe0f), so the
  crate can be checked against upstream with `sha256sum` alone.
