Metadata-Version: 2.4
Name: toktier-fastokens
Version: 0.3.1.1
License-File: LICENSE
License-File: NOTICE
License-File: NOTICES.txt
License-File: CHANGES-toktier.md
License-File: PATCHES/0001-F040-fix-bpe-resolve-out-of-vocabulary-characters.patch
License-File: PATCHES/0002-F041-fix-pcre2-align-unicode-class-semantics.patch
License-File: PATCHES/0003-F042-fix-scan-chunk-boundaries-inside-pretokens.patch
License-File: PATCHES/0004-F045-fix-scan-accept-long-s-in-contractions.patch
License-File: PATCHES/0005-F046-fix-split-rescan-tail-after-merge.patch
Summary: A pinned build of fastokens 0.3.1 with the toktier patch set 1 applied, maintained by the toktier project and not affiliated with or endorsed by the upstream project
Home-Page: https://github.com/asu-idi/toktier
License-Expression: Apache-2.0
Requires-Python: >=3.9
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM

# toktier-fastokens

A pinned build of [fastokens](https://github.com/crusoecloud/fastokens) 0.3.1
with five patches maintained by the toktier project. It is **not affiliated
with or endorsed by the upstream project**.

It provides the same import name as upstream:

```python
import fastokens
```

so it is a drop-in for code written against fastokens 0.3.1.

## Install only one of the two

Because both distributions install a top-level `fastokens` package, installing
this one *and* upstream `fastokens` into the same environment leaves whichever
was installed last in place, silently. Pick one:

```bash
pip uninstall -y fastokens
pip install toktier-fastokens
```

or, if you want upstream instead:

```bash
pip uninstall -y toktier-fastokens
pip install fastokens
```

## What the patches change

Five patches close cases where the engine returned a token-id list that
differs from the reference tokenizer for the same input, with no error
raised. `CHANGES-toktier.md` lists them one by one, `NOTICE` states which
files each one touches, and `PATCHES/` carries the patches verbatim.

Applying the five patches in order to upstream commit
`fe854299553524f2156a22036a2cb4d1f2ef4d97` (tag `v0.3.1`) reproduces the exact
source tree this build comes from: git tree hash
`aa1924284ec4abaedcc8ed5823ee17e7959c55c5`.

## Rebuilding

`build_pinned.sh` in the toktier repository rebuilds this distribution from a
clean upstream checkout with a pinned Rust toolchain and a pinned maturin. It
checks the patch digests and the resulting tree hash before it builds.

## Licensing

Upstream fastokens is Apache-2.0; the upstream `LICENSE` text is included
verbatim, as is the upstream `NOTICES.txt` (reproduced inside `NOTICE`,
which also carries the statement of modification required by section 4(b)).

## Scope of the guarantee

The readings that motivated this build were taken on the source tree named
above, with `tokenizers==0.22.2` as the reference. They describe that tree,
not every possible build of it: a wheel built elsewhere is a different
artifact with a different content digest. toktier recognises the builds it
publishes by content digest and says so explicitly; see the toktier
documentation for what each state reports.

