Metadata-Version: 2.4
Name: gpt-tokenizer
Version: 1.0.0
Summary: A terminal-based GPT token visualiser built on tiktoken.
Project-URL: Homepage, https://github.com/dev-amanydv/llm-tokenizer
Project-URL: Repository, https://github.com/dev-amanydv/llm-tokenizer
Project-URL: Issues, https://github.com/dev-amanydv/llm-tokenizer/issues
Author-email: Aman Yadav <ay.work07@gmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: bpe,cli,gpt,llm,nlp,terminal,tiktoken,tokenizer,tokens,visualiser
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Education
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Topic :: Utilities
Requires-Python: >=3.10
Requires-Dist: prompt-toolkit>=3.0
Requires-Dist: tiktoken>=0.7
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == 'dev'
Provides-Extra: examples
Requires-Dist: regex>=2024.0; extra == 'examples'
Description-Content-Type: text/markdown

# GPT-Tokenizer

A terminal-based GPT **token visualiser**. Type text into an interactive REPL and
each token is printed on a colored background so you can see exactly where the
tokenizer splits your input. Backed by [tiktoken](https://github.com/openai/tiktoken).

## Install

Requires Python 3.10+.

```bash
python -m venv .venv
source .venv/bin/activate
pip install -e .
```

## Usage

```bash
gpt-tokenizer          # console entry point
# or
python -m gpt_tokenizer
```

Type text at the prompt and press **Enter** to tokenize it. To write **multiple
lines or paragraphs**, press **Shift+Enter** (or **Option/Alt+Enter**) to insert
a newline without submitting — the whole block is tokenized together. Press
**Ctrl-D** to quit.

> Shift+Enter relies on your terminal reporting it as a distinct key (iTerm2,
> kitty, WezTerm, and the VS Code terminal do; macOS Terminal.app does not).
> Option/Alt+Enter works everywhere.

A line on its own starting with `/` is a command (runs on Enter):

| Command    | Action                                                            |
| ---------- | ----------------------------------------------------------------- |
| `/model` | Switch model (`gpt2`, `gpt-3.5-turbo`, `gpt-4`, `gpt-4o`) |
| `/clear` | Clear the screen and redraw the banner                            |
| `/help`  | Show the command list                                             |
| `/bye`   | Exit (or Ctrl-C, or Ctrl-D on an empty prompt)                    |

Newlines in your text are shown as a dim `⏎` glyph so the original layout stays
visible. The default model is `gpt-4o`.

## Project layout

```
src/gpt_tokenizer/     Installable package
  cli.py               REPL loop and command dispatch
  tokenizer.py         tiktoken encode wrapper -> list of (id, bytes)
  models.py            Supported models + interactive selection
  render.py            Banner and colorized token output
  ansi.py              ANSI escape-sequence helpers
examples/
  bpe_from_scratch.py  Standalone, educational BPE walkthrough (Karpathy minbpe style)
data/                  GPT-2 encoder.json / vocab.bpe (used only by the example)
tests/                 pytest suite
```

## Development

```bash
pip install -e ".[dev]"      # installs pytest
pytest

pip install -e ".[examples]" # installs regex, needed by the BPE example
python examples/bpe_from_scratch.py
```
