Metadata-Version: 2.4
Name: tiktoken-tools
Version: 0.1.0
Summary: Token counting tools for text, PDF, images, xlsx, csv, and docx files
Author: raffishquartan
License: MIT License
        
        Copyright (c) 2026 raffishquartan
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://github.com/raffishquartan/tiktoken-tools
Project-URL: Repository, https://github.com/raffishquartan/tiktoken-tools
Project-URL: Issues, https://github.com/raffishquartan/tiktoken-tools/issues
Keywords: tiktoken,tokens,token-counting,llm,openai,anthropic
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Text Processing
Classifier: Topic :: Utilities
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: tiktoken>=0.7
Requires-Dist: Pillow>=10.0
Requires-Dist: PyMuPDF>=1.24
Requires-Dist: pypdf>=4.0
Requires-Dist: openpyxl>=3.1
Requires-Dist: python-docx>=1.1
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Dynamic: license-file

<!--
Copyright (c) 2026 raffishquartan. Licensed under the MIT License.
-->

# tiktoken-tools

A set of command-line tools for estimating how many tokens a file will consume when sent to an LLM API. Useful for cost estimation, context window planning, and pre-processing workflows.

## Why this exists

The primary use case is **token-free assessment of document token costs** - whether you're checking a single file or scanning tens of thousands. This makes it practical to build pre-processing pipelines that gate on token budgets before paying API costs, without making any API calls at all.

I use tiktoken-tools extensively in my own Claude Code workflow. The included skills are integrated into many of my personal automation scripts to drive efficient token usage at scale - flagging large files before they're read into context, estimating API costs before batch jobs, and keeping sessions within context-window limits.

Supports text files, PDFs (both text-extractable and scanned), images, Excel spreadsheets, CSVs, and Word documents (including embedded images). Text tokenization uses OpenAI's [tiktoken](https://github.com/openai/tiktoken) library with the `o200k_base` encoding (GPT-4o family). Image token costs are calculated using the published formulas for OpenAI and Anthropic APIs.

## Supported file types

| Type | Extensions | What it counts |
|------|-----------|----------------|
| Text | .txt, .md, .py, .js, etc. | tiktoken tokens |
| PDF (text) | .pdf | tiktoken tokens from extracted text |
| PDF (scanned) | .pdf | Image tokens per rendered page |
| Image | .png, .jpg, .gif, .webp, etc. | OpenAI/Anthropic image tokens |
| XLSX | .xlsx | tiktoken tokens from CSV text representation |
| CSV | .csv | tiktoken tokens + row/col stats |
| DOCX | .docx | tiktoken tokens (text) + image tokens (embedded images) |

## Installation

### Full install with Claude Code integration (recommended)

Clone the repo and run the installer. This installs the CLI tools, symlinks the Claude Code skills into `~/.claude/skills/`, and adds a token-awareness section to `~/.claude/CLAUDE.md`:

```bash
git clone https://github.com/raffishquartan/tiktoken-tools.git
cd tiktoken-tools
./install-everything.sh
```

Requires [uv](https://docs.astral.sh/uv/getting-started/installation/).

### CLI only

If you only want the command-line tools without the Claude Code skills, install from PyPI:

```bash
uv tool install tiktoken-tools
# or
pip install tiktoken-tools
```

To install the latest unreleased code straight from GitHub instead:

```bash
uv tool install git+https://github.com/raffishquartan/tiktoken-tools.git
```

## Usage

### Generic dispatcher (auto-detects file type)

```bash
count-tokens -f document.pdf
count-tokens -f photo.png
count-tokens -f data.xlsx

# Directory - recursively counts all files and outputs CSV to stdout
count-tokens -f ./my-project/
count-tokens -f ./my-project/ > token-counts.csv

# Stdin
echo "hello world" | count-tokens -f -
```

### Type-specific commands

```bash
# Text files (supports stdin via -f -)
count-text-tokens -f readme.md
echo "hello world" | count-text-tokens -f -

# PDF (text-based)
count-pdf-tokens -f document.pdf --csv page-stats.csv

# Images
count-image-tokens -f photo.png
count-image-tokens -f photo.png --provider anthropic
count-image-tokens -f photo.png --detail low

# Scanned/image PDFs
count-image-pdf-tokens -f scanned.pdf --provider openai --dpi 150

# Excel
count-xlsx-tokens -f spreadsheet.xlsx

# CSV
count-csv-tokens -f data.csv

# Word documents (text + embedded images)
count-docx-tokens -f report.docx --provider anthropic
```

### Common flags

| Flag | Description | Applies to |
|------|-------------|------------|
| `-f, --file` | Path to file, directory, or `-` for stdin | All |
| `-m, --model` | Model for tiktoken encoding (default: gpt-4o-mini) | Text-based types |
| `-p, --provider` | `openai` or `anthropic` (default: highest count) | Image-based types |
| `-d, --detail` | OpenAI detail level: `low` or `high` (default: high) | Image-based types |

## Image token formulas

### OpenAI (GPT-4o / GPT-4o-mini) - high detail

1. Scale longest side to <= 2048px (downscale only)
2. Scale shortest side to <= 768px (downscale only)
3. Tile into 512x512 blocks
4. `tokens = tiles * 170 + 85`

Low detail: flat 85 tokens.

### Anthropic (Claude 3+)

1. Scale so neither dimension exceeds 1568px (downscale only)
2. `tokens = ceil(width * height / 750)`

## A note on Anthropic token counts

Anthropic's Claude models use a custom BPE tokenizer that is not publicly documented. tiktoken uses OpenAI's `o200k_base` encoding, which differs from Claude's tokenizer. In practice the counts are directionally accurate - differences for English prose are typically small (within a few percent) - but can be more pronounced for code, non-English text, or content heavy with special characters.

Treat tiktoken counts as planning estimates ("will this fit in my context window?", "roughly how much will this batch cost?") rather than exact billing figures. The Anthropic API returns authoritative `input_tokens` counts in every response; use those for precise accounting.

## PDF classification

The generic `count-tokens` command auto-detects whether a PDF contains extractable text or is scanned/image-based. If the average extracted text per page is less than 50 characters, it's treated as an image PDF and pages are rendered at 150 DPI for token counting.

## Development

To fork, develop, or contribute:

```bash
git clone https://github.com/raffishquartan/tiktoken-tools.git
cd tiktoken-tools
uv pip install -e ".[dev]"
pytest
```

## Claude Code integration

tiktoken-tools includes four Claude Code skills that let Claude proactively monitor and manage token usage during sessions:

| Skill | Purpose |
|-------|---------|
| `count-file-tokens` | Count tokens in any file before reading it into context |
| `estimate-session-tokens` | Estimate how many tokens the current session has consumed |
| `estimate-api-cost` | Estimate the cost of API calls given a token count |
| `check-token-fit-for-session` | Check whether a set of files will fit in the remaining context |

The easiest way to install them is `./install-everything.sh` (see Installation above). The skills are symlinked, so pulling repo updates automatically updates the installed skills.

To configure Claude to use the skills proactively, add this to your `~/.claude/CLAUDE.md` (the installer does this for you):

```markdown
# Token awareness

Use the tiktoken-tools skills (count-file-tokens, estimate-session-tokens, estimate-api-cost,
check-token-fit-for-session) proactively to monitor and manage token usage. Before reading large
files, check their token count with /count-file-tokens. When the session has been running for a
while, use /estimate-session-tokens to check context usage and warn if approaching limits. When
the user is planning API calls or batch processing, use /estimate-api-cost and
/check-token-fit-for-session to help them understand costs and capacity. Always present token
counts concisely and suggest ways to reduce token usage when files are large (e.g. converting
xlsx to CSV, using low-detail mode for images, extracting relevant pages from PDFs).
```

For full skill installation details, see [claude-skills/README.md](claude-skills/README.md).

## Dependencies

- [tiktoken](https://github.com/openai/tiktoken) - text tokenization
- [Pillow](https://python-pillow.org/) - image dimension reading
- [PyMuPDF](https://pymupdf.readthedocs.io/) - PDF text extraction and page rendering
- [pypdf](https://pypdf.readthedocs.io/) - fallback PDF text extraction
- [openpyxl](https://openpyxl.readthedocs.io/) - xlsx reading
- [python-docx](https://python-docx.readthedocs.io/) - docx reading
