Metadata-Version: 2.5
Name: tokens-md
Version: 0.2.0
Summary: Convert files to token-efficient Markdown for LLM prompts.
Author: tokens.md
License: AGPL-3.
License-File: LICENSE
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Text Processing :: Markup :: Markdown
Classifier: Topic :: Utilities
Requires-Python: >=3.10
Requires-Dist: extract-msg>=0.48
Requires-Dist: fastapi
Requires-Dist: httpx
Requires-Dist: mathml-to-latex>=1.0
Requires-Dist: mobi>=0.3.3
Requires-Dist: odfpy>=1.4
Requires-Dist: openpyxl>=3.1
Requires-Dist: pathspec
Requires-Dist: pymupdf4llm
Requires-Dist: pyperclip
Requires-Dist: python-docx>=1.2
Requires-Dist: python-multipart
Requires-Dist: python-pptx>=0.6.23
Requires-Dist: rich
Requires-Dist: striprtf>=0.0.26
Requires-Dist: sumy
Requires-Dist: tiktoken
Requires-Dist: trafilatura
Requires-Dist: typer>=0.12
Requires-Dist: uvicorn[standard]
Requires-Dist: watchdog
Provides-Extra: dev
Requires-Dist: httpx; extra == 'dev'
Requires-Dist: mypy; extra == 'dev'
Requires-Dist: pytest; extra == 'dev'
Requires-Dist: pytest-cov; extra == 'dev'
Requires-Dist: pytest-instafail; extra == 'dev'
Requires-Dist: pytest-sugar; extra == 'dev'
Requires-Dist: ruff; extra == 'dev'
Provides-Extra: web
Requires-Dist: fastapi; extra == 'web'
Requires-Dist: python-multipart; extra == 'web'
Requires-Dist: uvicorn[standard]; extra == 'web'
Description-Content-Type: text/markdown

# tokens.md

![The Vision of the Colosseum. The last martyr by José Benlliure y Gil - 1885](assets/rm/cover.jpg)

Tokens.md is my tool for saving tokens when speaking to chatbots by converting files en-masse to Markdown. It turns PDFs, Office documents, e-books, structured data, HTML, web pages, and whole code repositories into clean, token-efficient Markdown you can paste straight into an LLM.

OpenAI's `tiktoken` is used to estimate how many tokens are saved (pretty accurately). In some preliminary results, this conversion is usually anywhere between 60% and 95% of tokens saved (smaller files benefit "more" since a larger proportion of their data is overhead).

## Outline

- [Features](#features)
- [Install](#install)
- [Running](#running)
  - [1. CLI Usage (`tmd`)](#1-cli-usage-tmd)
  - [2. Simple Script Execution (`python src/main.py`)](#2-simple-script-execution-python-srcmainpy)
  - [3. Web Front End (`tmd ui` & Next.js)](#3-web-front-end-tmd-ui--nextjs)
- [Supported Formats](#supported-formats)
- [Development](#development)
- [Documentation](#documentation)
- [License](#license)

## Features

The features of this tool are all encompassed by the `tmd` CLI commands. The front end displays this with an accessible GUI. 

- **`tmd convert`**: convert files to Markdown.
- **`tmd clip`**: convert files to markdown and copy the result to your clipboard.
- **`tmd watch`**: watch a hot folder and auto-convert new files as they appear.
- **`tmd fetch`**: pull a web page and save clean article Markdown.
- **`tmd repo`**: collapse an entire code repository into a single Markdown manifest.
- **`tmd merge`**: combine many files into one master document with a Table of Contents.
- **`tmd delta`**: show how many tokens were saved.
- **`--budget`**: an extra argument to force output size in tokens.

See [the usage guide](#running) for examples

## Install

### Pre-Game

First, make sure you have Python 3.13+, uv, and TypeScript support on your machine. Clone this repository:

```
git clone https://github.com/intelligent-username/tokens.md
```

And create the environment:

```bash
uv venv --python=3.13 .venv

# On Linux or Mac
source .venv/bin/activate

# On Windows
.venv\Scripts\activate

uv pip install -e .               # editable install (provides `tmd`)
```

To update dependencies, just run `uv sync`.

Or install the runtime dependencies directly:

```bash
uv pip install -r requirements.txt
```

## Running

![Quick lil demo](assets/rm/example.webp)

You can run `tokens.md` in three distinct ways depending on your workflow preference. Here they are, ordered from easiest to hardest to use.

### 1. Simple Script Execution (`python src/main.py`)

If you want to run the program directly without installing it as a package, run `src/main.py` using your Python interpreter:

```bash
python src/main.py
```

You can pass standard CLI arguments directly to the script:

```bash
python src/main.py convert input/ -o output/
```

Both bare `tmd` and `python src/main.py` automatically resolve default `input/` and `output/` folders relative to the project root.


### 2. Web Front End (`tmd ui` & Next.js)

The single-page web interface wraps all `tmd` capabilities into an intuitive side-by-side visual workbench featuring drag-and-drop file upload, URL fetching, clipboard copying, and live token compression flow meters.

**Method A: Single CLI Command**
```bash
uv pip install -e ".[web]"
tmd ui
```
This launches the backend API on `http://127.0.0.1:8642` and opens the browser interface.

**Method B: Development Server (Frontend + Backend)**
1. Start the FastAPI backend server:
   ```bash
   python -m backend
   ```
2. In a separate terminal, start the Next.js frontend dev server:
   ```bash
   cd frontend
   npm install
   npm run dev
   ```
3. Open `http://localhost:3000` in your web browser.

### 3. CLI Usage (`tmd`)

After installing via `pip install -e .`, the `tmd` command is registered in your environment.

Navigate to the folder containing the files you want to convert and run the following commands. You may manipulate sub-folders and file names, or omit them to use the defaults.

```bash
# Convert a folder or file to Markdown
tmd convert input/ -o output/

# Bare `tmd` command uses default input/ and output/ directories
tmd

# Other CLI subcommands
tmd merge input/ -o output/merged.md
tmd fetch https://example.com/article -o output/article.md
tmd repo . -o output/repo.md
```

## Development

Install dev dependencies and run the checks:

```bash
uv pip install -e ".[dev]"
pytest
ruff check .
mypy src
```

## Documentation

- [`docs/USAGE.md`](docs/USAGE.md): full usage guide for every `tmd` subcommand.
- [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md): how the converter registry works and how to add new formats.

## License

This project is licensed under the GNU AFFERO GENERAL PUBLIC LICENSE. See more [here](LICENSE)
