Metadata-Version: 2.4
Name: pytokencalc
Version: 1.0.1
Summary: Universal token counting for ANY LLM with embedded StatGuardian quality validation: 8 cloud providers, 7 local inference engines, custom/proprietary models, and BYOM (Bring Your Own Model). Pattern-based forward-compatibility, model discovery, platform-aware tracking, 99%+ accuracy.
Author-email: Georgi Mammen Mullassery <mullassery@gmail.com>
Maintainer-email: Georgi Mammen Mullassery <mullassery@gmail.com>
License: MIT License
        
        Copyright (c) 2026 Georgi Mammen Mullassery
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://github.com/Mullassery/PyTokenCalc
Project-URL: Repository, https://github.com/Mullassery/PyTokenCalc
Project-URL: Bug Tracker, https://github.com/Mullassery/PyTokenCalc/issues
Project-URL: Documentation, https://github.com/Mullassery/PyTokenCalc#readme
Project-URL: Changelog, https://github.com/Mullassery/PyTokenCalc/releases
Project-URL: Source Code, https://github.com/Mullassery/PyTokenCalc/tree/main
Project-URL: Discussions, https://github.com/Mullassery/PyTokenCalc/discussions
Keywords: llm,token-counting,tokenizer,multi-provider,gpt,claude,gemini,llama,mistral,groq,deepinfra,token-counter,transformer,tiktoken
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Natural Language :: English
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Utilities
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pydantic>=2.0
Provides-Extra: tokenizers
Requires-Dist: tiktoken>=0.5.0; extra == "tokenizers"
Requires-Dist: transformers>=4.30.0; extra == "tokenizers"
Requires-Dist: sentencepiece>=0.1.99; extra == "tokenizers"
Provides-Extra: quality
Requires-Dist: pyyaml>=6.0; extra == "quality"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0; extra == "dev"
Dynamic: license-file

# PyTokenCalc: Unified Token Counting for Multi-Provider LLMs

[![PyPI version](https://badge.fury.io/py/pytokencalc.svg)](https://pypi.org/project/pytokencalc/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Python 3.9+](https://img.shields.io/badge/Python-3.9+-blue.svg)](https://www.python.org/downloads/)
![Status: v0.9 Released](https://img.shields.io/badge/Status-v0.9%20Released-brightgreen)
![104 Tests Passing](https://img.shields.io/badge/Tests-104%2F104%20Passing-brightgreen)

---

## Why Star This?

- **One library, 20+ LLM providers, 99%+ accuracy** — Replace OpenAI, Anthropic, Google, Ollama, and 15+ more with a single validated API
- **Quality validation built-in** — Detect counting drift, validate model baselines, and catch silent errors before cost reconciliation
- **No vendor lock-in** — Switch providers by changing one string; same code for all
- **Self-healing** — Automatic fallback to pattern-based counting when providers go down

---

## Release Status

### v0.9.0 RELEASED (July 18, 2026)
**Universal Token Counting for ANY LLM**

**Providers Supported:**
- 8 cloud providers (OpenAI, Anthropic, Google, Cohere, Azure, RunPod, Together AI, Replicate)
- 7 local inference engines (Ollama, LM Studio, LocalAI, Llama.cpp, GPT4All, Text Gen WebUI, Jan)
- Custom provider registration (BYOM - Bring Your Own Model)
- Docker/Kubernetes deployment ready

**Key Features:**
- Pattern-based forward compatibility (claude-*, gemini-*, command-*)
- Model discovery system (automatic provider suggestion)
- Platform-aware tracking (same model, different platforms)
- Temporal variation monitoring (timestamp + session tracking)
- 99%+ accuracy vs official counters
- 104 tests passing, zero breaking changes

**Documentation:**
- CUSTOM_PROVIDERS.md (200+ lines, 10+ provider examples including Docker/Kubernetes)
- CHANGELOG.md (complete version history)
- RELEASE_NOTES_v0.9.0.md (comprehensive release documentation)

---

## The Problem

You're building an LLM application. You need to count tokens accurately, predict costs, validate model efficiency, and know which providers are degrading. But right now:

- **You don't trust your token counts.** Are they accurate? Off by how much? No idea.
- **Cost predictions are guesses.** You budget $X, spend $Y. No visibility into why.
- **Model baselines are stale.** When did you last verify they're still valid?
- **Provider health is invisible.** You find out when your app breaks, not before.
- **10+ libraries to integrate.** Each with different APIs and update cycles.

The nightmare:
- No validation layer between your code and provider APIs
- Silent errors: wrong counts go undetected until cost reconciliation
- Vendor lock-in: switching providers requires rewriting integrations
- Zero audit trail: no way to trace which decision used which counts
- Dead time: managing integrations instead of building features

**You need a trusted, validated token oracle. Not just a counter.**

---

## The Solution: PyTokenCalc

**Self-validating token intelligence for LLM applications.**

One library. One API. All providers. **With quality guarantees.**

```python
from pytokencalc.tokenizers import TokenCounterRegistry

registry = TokenCounterRegistry()

# Same code for every provider
tokens = registry.count_tokens("gpt-4o", "Your prompt")
tokens = registry.count_tokens("claude-3-5-sonnet", "Your prompt")
tokens = registry.count_tokens("llama-70b", "Your prompt")

print(f"Tokens: {tokens.input_tokens}")
```

That's it. No provider-specific code. No multiple libraries. No headaches.

---

## Why PyTokenCalc?

| Feature | PyTokenCalc | OpenAI library | Anthropic SDK | Others (Ollama, etc.) |
|---------|---------|---------|---------|---------|
| **Providers supported** | 20+ | OpenAI only | Anthropic only | 1 each |
| **Unified API** | One call | Different per provider | Different | All different |
| **Accuracy** | 99%+ validated | Reference | Reference | Varies |
| **Quality validation** | Built-in drift detection | None | None | None |
| **Cost prediction** | Yes | Basic | Basic | No |
| **Fallback counting** | Pattern-based | None | None | None |
| **Model discovery** | Auto-detect | Manual | Manual | Manual |
| **Custom providers** | Easy registration | Not supported | Not supported | No |
| **Audit trail** | Full history | None | None | None |

---

## Quick Start: Terminal One-Liners

Don't want to write Python? Use the `pycount` command directly:

```bash
# Count tokens instantly
pycount "Hello world"
# Output: 2 tokens (input: 2)

# Pipe from stdin
echo "Your text here" | pycount

# Use a different model
pycount -m claude-3-sonnet "Your text"

# JSON output for scripting
pycount -j "Your text"
# {"model": "gpt-4", "tokens": 2, "input": 2, ...}
```

** [Full CLI Guide](QUICK_START_CLI.md)**

---

## What You Get

### 1. **Validated Token Counts**
Every token count is validated against historical baselines, provider reliability, and model efficiency standards. Know your data is accurate before it affects your decisions.

### 2. **Self-Validating API**
Built-in quality gates catch errors before they cascade:
- Token accuracy checks (5% deviation detection)
- Model efficiency baselines (50+ sample statistical validation)
- Cost prediction confidence (15% accuracy validation)
- Provider health monitoring (uptime, response time, consistency)

### 3. **True Unified Interface**
One function for 20+ providers (cloud and open-source).

```python
# Works for everything
models = [
 "gpt-4o",
 "claude-3-5-sonnet",
 "gemini-2-flash",
 "llama-70b",
 "mistral-large",
]

for model in models:
 result = registry.count_tokens(model, "Your text")
 print(f"{model}: {result.input_tokens} tokens")
```

### 2. **Smart Routing: Fast When You Can, Accurate When You Need To**
- **Local tokenizers** for OpenAI GPT and open-source models (5-10ms)
- **Cached API calls** for proprietary models like Claude and Gemini (0ms if cached, 200ms first time)
- **Automatic** — you don't think about it

```python
# Automatically uses tiktoken (fast, local, free)
gpt_tokens = registry.count_tokens("gpt-4o", text) # 5ms

# Automatically caches and reuses
gpt_tokens = registry.count_tokens("gpt-4o", text) # 0ms (cached)
```

### 3. **99%+ Accuracy vs Official Counts**
Local tokenizers are 100% accurate for their models. API-backed counts match official numbers exactly. No guessing.

### 4. **70-80% Fewer API Calls**
Intelligent caching means you don't hammer provider APIs just to count tokens.

### 5. **Batch Operations**
Count tokens for multiple prompts at once.

```python
results = registry.count_batch([
 {"model": "gpt-4o", "text": "Prompt 1"},
 {"model": "llama-70b", "text": "Prompt 2"},
 {"model": "claude-3-5-sonnet", "text": "Prompt 3"},
])

for result in results:
 print(f"{result.input_tokens} tokens")
```

---

## Installation

### For Users

**Basic (Local Tokenizers Only)**
```bash
pip install pytokencalc
```

This gives you token counting for OpenAI GPT and Llama/Mistral models (covers 70% of use cases).

**Full (Recommended)**
```bash
pip install "pytokencalc[tokenizers]"
```

Installs optional dependencies for broader model support:
- `tiktoken` — OpenAI GPT models
- `transformers` — Llama, Mistral, and 1000+ HuggingFace models
- `sentencepiece` — Additional model support

### For Developers

```bash
# Clone the repository
git clone https://github.com/Mullassery/pytokencalc.git
cd pytokencalc

# Create and activate virtual environment
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate

# Install in development mode with dev dependencies
pip install -e ".[dev,tokenizers]"

# Run tests
pytest tests/
make test # or use Makefile
```

**Requirements:**
- Python 3.9+
- pip or uv

---

## Quick Example

```python
from pytokencalc.tokenizers import TokenCounterRegistry

# Initialize once
registry = TokenCounterRegistry()

# Use anywhere
prompt = "Write me a Python function that..."
result = registry.count_tokens("gpt-4o", prompt)

print(f"Input tokens: {result.input_tokens}")
print(f"Latency: {result.latency_ms}ms")
print(f"Was cached: {result.cached}")
print(f"Source: {result.source}") # "local", "api", or "formula"
```

Output:
```
Input tokens: 42
Latency: 5.2ms
Was cached: False
Source: local
```

Run it again with the same text:
```python
result = registry.count_tokens("gpt-4o", prompt)
print(f"Latency: {result.latency_ms}ms (cached!)")
```

Output:
```
Latency: 0.3ms (cached!)
```

---

## Why This Matters

### Before PyTokenCalc
```python
# Your code without PyTokenCalc
import tiktoken
from transformers import AutoTokenizer
import anthropic

# Different library, different API for each
enc = tiktoken.encoding_for_model("gpt-4o")
gpt_tokens = len(enc.encode(text))

llama_tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-70b")
llama_tokens = len(llama_tokenizer.encode(text))

client = anthropic.Anthropic()
claude_response = client.messages.count_tokens(
 model="claude-3-5-sonnet",
 messages=[{"role": "user", "content": text}]
)
claude_tokens = claude_response.input_tokens

# Three different patterns, three different libraries
```

### With PyTokenCalc
```python
from pytokencalc.tokenizers import TokenCounterRegistry

registry = TokenCounterRegistry()

gpt_tokens = registry.count_tokens("gpt-4o", text).input_tokens
llama_tokens = registry.count_tokens("llama-70b", text).input_tokens
claude_tokens = registry.count_tokens("claude-3-5-sonnet", text).input_tokens

# One pattern. One library. Done.
```

---

## Supported Providers

### Cloud LLM APIs (20+)
**Explicit Support:**
- OpenAI (GPT-4, GPT-3.5-turbo)
- Anthropic (Claude 3 + pattern: `claude-*`)
- Google (Gemini, pattern: `gemini-*`)
- Cohere (Command, pattern: `command-*`)
- Azure OpenAI (same as OpenAI)
- Plus: RunPod, Together AI, Replicate, HuggingFace Inference, and more

**Dynamic Support:**
Any cloud provider with an API endpoint (RunPod, Llama Labs, custom backends)

### Local Inference Engines (7+)
**Auto-Detected:**
- LM Studio (localhost:1234)
- LocalAI (localhost:8080)
- Llama.cpp (localhost:8000)
- GPT4All (localhost:4891)
- Text Generation WebUI (localhost:5000)
- Jan (localhost:1337)
- Vllm (localhost:8000)

### Open-Source Models
**Built-in Ollama Support:**
- Any model you pull into Ollama
- Automatic model discovery
- No model list required

**HuggingFace Direct:**
- 14+ known models supported
- Any HuggingFace model works

### Bring Your Own Model (BYOM)
**Completely Custom:**
- Your fine-tuned models
- Your proprietary models
- Your locally-trained models
- Any model running on any GPU/CPU

Just expose an API and register it with PyTokenCalc.

### Model Discovery
Don't know which provider? PyTokenCalc discovers it:
```python
from pytokencalc.model_discovery import ModelDiscovery

# Automatic discovery
result = ModelDiscovery.lookup_model("llama-2-7b")
# Returns: ["ollama", "huggingface", "runpod", "together-ai"]

# Get setup instructions
report = ModelDiscovery.get_discovery_report("mistral-7b")
# Prints: Install, setup steps for each provider
```

---

## Advanced Features

### Custom Provider Registration
Use any LLM provider, including proprietary or self-hosted:

```python
from pytokencalc.tokenizers.custom_provider_counter import (
 CustomProviderCounter,
 register_custom_provider,
)

# Register your RunPod endpoint
runpod = CustomProviderCounter(
 provider_name="runpod",
 base_url="https://api.runpod.io/v2/YOUR_ID",
 api_key="your-key"
)
runpod.register_models(["llama-2-7b", "mistral-7b"])
register_custom_provider(runpod)

# Now use it like any other provider
registry.count_tokens("llama-2-7b", text, provider="runpod")
```

See [CUSTOM_PROVIDERS.md](CUSTOM_PROVIDERS.md) for 10+ examples.

### Platform-Aware Token Counting
 **Important:** Same model on different platforms may have different token counts

```python
# Same text, different platforms = different results
result_local = registry.count_tokens("llama-2-7b", text, provider="ollama")
result_cloud = registry.count_tokens("llama-2-7b", text, provider="runpod")

print(result_local.platform) # "ollama"
print(result_cloud.platform) # "runpod"
print(result_local.timestamp) # When it was counted
```

Results are kept separate by platform and timestamp. Never aggregate or mix.

### Temporal Variation Tracking
Cloud infrastructure changes over time. PyTokenCalc tracks this:

```python
# First session
result1 = registry.count_tokens("gpt-4o", text)
print(f"Tokens: {result1.input_tokens}, Latency: {result1.latency_ms}ms")
# Output: Tokens: 100, Latency: 50ms

# Later session (after infrastructure update)
result2 = registry.count_tokens("gpt-4o", text)
print(f"Tokens: {result2.input_tokens}, Latency: {result2.latency_ms}ms")
# Output: Tokens: 101, Latency: 120ms (different!)

# Both are tracked with timestamps
print(result1.timestamp, result2.timestamp) # Different times
```

---

## API Reference

### Count Tokens

```python
from pytokencalc.tokenizers import TokenCounterRegistry

registry = TokenCounterRegistry()

# Single count
result = registry.count_tokens(
 model="gpt-4o",
 text="Your prompt here"
)

print(result.input_tokens) # int - token count
print(result.latency_ms) # float - milliseconds taken
print(result.cached) # bool - came from cache?
print(result.source) # str - "local", "api", or "formula"
```

### Batch Count

```python
# Multiple counts in one call
results = registry.count_batch([
 {"model": "gpt-4o", "text": "Text 1"},
 {"model": "llama-70b", "text": "Text 2"},
])

for result in results:
 print(f"{result.input_tokens} tokens")
```

### Cache Stats

```python
cache = registry.tokenizers[0].cache
stats = cache.get_stats()

print(stats.hit_rate) # Cache hit rate (0-1)
print(stats.size) # Number of cached entries
print(stats.total_calls) # Total counts requested
```

---

## Real-World Use Cases

### Use Case 1: Monitor Token Usage
```python
def log_request(model, prompt, response):
 tokens = registry.count_tokens(model, prompt)
 print(f"Request used {tokens.input_tokens} tokens")

# Call it automatically on every LLM invocation
log_request("gpt-4o", "Hello", response)
```

### Use Case 2: Token Budget Management
```python
MAX_TOKENS_PER_HOUR = 1_000_000
used_tokens = 0

for prompt in user_prompts:
 token_count = registry.count_tokens("claude-3-5-sonnet", prompt)
 if used_tokens + token_count.input_tokens > MAX_TOKENS_PER_HOUR:
 print("Budget exceeded!")
 break
 
 used_tokens += token_count.input_tokens
 response = call_llm(prompt)
```

### Use Case 3: Model Comparison
```python
text = "Your 1000-word essay..."

for model in ["gpt-4o", "claude-3-5-sonnet", "gemini-2-flash"]:
 tokens = registry.count_tokens(model, text).input_tokens
 print(f"{model}: {tokens} tokens")

# See which model would be cheapest
```

---

## Performance

| Scenario | Latency | Notes |
|----------|---------|-------|
| First local count (GPT) | 5ms | tiktoken initialization |
| Cached local count | 0.3ms | Memory lookup |
| First API count (Claude) | 200-300ms | Network call |
| Cached API count | 0.2ms | Memory lookup |
| Batch of 10 (mixed) | ~10ms | Parallel processing |

**TL;DR: Local is instant. API counts are cached aggressively. You rarely wait.**

---

## Database & Storage

PyTokenCalc maintains a database to store token counts and enable reconciliation across API calls:

**PyTokenCalc Standalone:** Manages its own database for token accounting. OpenAnchor is optional.

**PyTokenCalc + OpenAnchor:** When you install OpenAnchor, it automatically includes PyTokenCalc. Both use the same database:
- PyTokenCalc stores: Raw token counts (token_events)
- OpenAnchor stores: Analysis & recommendations (attribution, patterns, etc)
- Single instance: Both projects share one database
- OpenAnchor does NOT create its own database—it enriches PyTokenCalc's database with intelligence

**Important:** OpenAnchor requires PyTokenCalc to be installed. You cannot use OpenAnchor without PyTokenCalc. However, you can use PyTokenCalc alone without OpenAnchor.

---

## Token Counting Scope

PyTokenCalc counts tokens for **all operations**, including:

### Inference (Text Generation)
```python
# Count tokens for chat/completion endpoints
result = registry.count_tokens("gpt-4o", "Write a story about...")
# Tokens used for generating responses
```

### Embeddings (Vector Representation)
```python
# Count tokens for embedding API calls
result = registry.count_tokens("text-embedding-3-large", "Hello world")
# Tokens used for creating embeddings
```

### Any Token Consumption
- Chat completions
- Text embeddings
- Image descriptions
- Code generation
- Summarization
- Classification
- Any model operation that consumes tokens

**One function. All operations.**

---

## What's NOT Included

PyTokenCalc counts tokens accurately. That's what it does.

For advanced use cases like token intelligence, optimization, or analytics, use OpenAnchor (which bundles PyTokenCalc) or build on PyTokenCalc's token counting API:
- Token intelligence & attribution (use OpenAnchor)
- Pattern detection & anomaly alerts (use OpenAnchor)
- Optimization recommendations (use OpenAnchor)
- Plan capacity and model comparison (build on PyTokenCalc)

---

## Contributing

Want to add support for a new provider? See [ADDING_PROVIDERS.md](ADDING_PROVIDERS.md) for step-by-step guide.

---

## License

MIT License. See [LICENSE](LICENSE) for details.

---

## About

**PyTokenCalc** solves one problem perfectly: unified token counting across all LLM providers.

**Author**: Georgi Mammen Mullassery (@Mullassery) 
**Repository**: https://github.com/Mullassery/PyTokenCalc

---

**Stop integrating tokenizers. Start counting tokens.**
