Metadata-Version: 2.4
Name: repotoken
Version: 0.1.0
Summary: Count tokens across any code repository — fast, local, no API calls
License: MIT
Project-URL: Homepage, https://github.com/Shardul37/repotoken
Requires-Python: >=3.8
Description-Content-Type: text/markdown
Requires-Dist: tiktoken

# repotoken

Count tokens across any code repository — fast, local, no API calls.

```
$ repotoken .

scanning /your/project ...

  src/parser.py          3.1K tokens
  src/main.py            2.8K tokens
  src/utils.py           1.2K tokens
  tests/test_parser.py     890 tokens
  README.md                430 tokens

  ──────────────────────────────────
  TOTAL                    8.4K tokens
  files scanned                   5
  skipped (binary)                3
```

## Why?

LLMs have context limits — GPT-4o tops out at 128K tokens, most Claude models at 200K. If you're working with AI tooling on a codebase, you need to know how big your repo actually is in token terms — not lines, not bytes, but **tokens**.

`repotoken` tells you exactly that, per file and in total, so you can make informed decisions about chunking, batching, or summarisation.

## Install

```bash
pip install repotoken
```

## Usage

```bash
# scan current directory
repotoken .

# scan any repo
repotoken /path/to/any/repo
```

## Notes

- Uses `cl100k_base` encoding (GPT-4 / GPT-4o tokenizer) — accurate for most modern LLMs within ~10-15%
- Detects binaries using null-byte scanning (same method as git) — no hardcoded extension lists
- Automatically skips `.git`, `node_modules`, `__pycache__`, `.venv`, and other noise
- Sorts output by token count so the heaviest files are at the top
- No API calls, no network requests, runs entirely locally

## License

MIT
