Metadata-Version: 2.4
Name: repotoken
Version: 0.1.2
Summary: Count tokens across any code repository — fast, local, no API calls
License: MIT
Project-URL: Homepage, https://github.com/Shardul37/repotoken
Requires-Python: >=3.8
Description-Content-Type: text/markdown
Requires-Dist: tiktoken

# repotoken

Count tokens across any code repository — fast, local, no API calls.

```
$ repotoken .

scanning /your/project ...

  file                      tokens     lines
  ────────────────────────  ────────  ──────
  src/parser.py               3.1K      140
  src/main.py                 2.8K      120
  src/utils.py                1.2K       55
  tests/test_parser.py         890       40
  README.md                    430       30

  ──────────────────────────────────────────
  TOTAL                       8.4K      385
  files                                   5
  skipped (binary)                        3
```

## Why?

LLMs have context limits — GPT-4o tops out at 128K tokens, most Claude models at 200K. If you're working with AI tooling on a codebase, you need to know how big your repo actually is in token terms — not lines, not bytes, but **tokens**.

`repotoken` tells you exactly that, per file and in total, so you can make informed decisions about chunking, batching, or summarisation.

## Install

```bash
pip install repotoken
```

## Usage

```bash
# scan current directory
repotoken .

# scan any repo
repotoken /path/to/any/repo
```

## Notes

- Uses `cl100k_base` encoding (GPT-4 / GPT-4o tokenizer) — accurate for most modern LLMs within ~10-15%
- Detects binaries using null-byte scanning (same method as git) — no hardcoded extension lists
- Automatically skips `.git`, `node_modules`, `__pycache__`, `.venv`, and other noise
- Sorts output by token count so the heaviest files are at the top
- No API calls, no network requests, runs entirely locally

## License

MIT
