Metadata-Version: 2.4
Name: tokensafe
Version: 1.3.0
Summary: Autonomous Zero-Loss Token Economy & Context Optimizer for AI Coding Agents
Author-email: TokenSafe Team <support@tokensafe.dev>
License-Expression: MIT
Project-URL: Homepage, https://tokensafe.dev
Project-URL: Repository, https://github.com/NekoSin4eG/tokensafe
Project-URL: Documentation, https://tokensafe.dev
Project-URL: Issues, https://github.com/NekoSin4eG/tokensafe/issues
Keywords: ai,llm,tokens,token-optimization,context-window,cursor,claude,antigravity,codex,mcp,mcp-server,ast,proxy,jwt,ed25519,zero-trust
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Topic :: Software Development :: Build Tools
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# ⚡ TokenSafe (`tokensafe`)

[![npm version](https://img.shields.io/npm/v/tokensafe.svg?color=CB3837)](https://www.npmjs.com/package/tokensafe)
[![PyPI version](https://img.shields.io/pypi/v/tokensafe.svg?color=3775A9)](https://pypi.org/project/tokensafe/)
[![License: MIT](https://img.shields.io/badge/License-MIT-emerald.svg)](https://opensource.org/licenses/MIT)
[![SWE-bench: Verified](https://img.shields.io/badge/SWE--bench-up_to_95%25_Savings-06B6D4.svg)](https://tokensafe.dev#benchmarks)
[![Python: 3.10+](https://img.shields.io/badge/Python-3.10+-3B82F6.svg)](https://python.org)
[![Trial: 7 Days Free](https://img.shields.io/badge/Trial-7_Days_Free-10B981.svg)](https://tokensafe.dev#pricing)

> **Autonomous Zero-Loss Token Economy & Context Optimizer for AI Coding Agents.**  
> Slashes LLM token costs by **up to 95%** with zero quality loss. Built for **Google Antigravity, OpenAI Codex, Cursor, Claude Desktop, Z.ai Code, Windsurf & Cline**.

🌐 **Official Website:** [https://tokensafe.dev](https://tokensafe.dev)

---

## ⚡ 1-Click Universal Install

Connect TokenSafe to all detected AI coding agents on your machine in seconds:

```bash
# Option 1: Zero-Install via npx (Recommended for Cursor & Claude Desktop)
npx tokensafe

# Option 2: Python / PyPI
pip install tokensafe && tokensafe

# Option 3: Direct Shell Script (macOS / Linux)
curl -fsSL https://tokensafe.dev/install.sh | bash
```

---

## 🔌 Universal MCP Server Configuration

Connect TokenSafe directly to **Claude Desktop** (`claude_desktop_config.json`) or **Cursor** MCP Settings:

```json
{
  "mcpServers": {
    "tokensafe": {
      "command": "npx",
      "args": ["-y", "tokensafe", "mcp"]
    }
  }
}
```

---

## 🚀 The Breakthrough: How It Saves 60%–90% Tokens

Traditional AI code interactions suffer from massive context bloat:
1. **The BPE Token Penalty:** Verbose natural language prompts and filler waste 40%–70% of ingress context.
2. **Bulk Tool Dumps:** Reading whole 800+ line files burns 5,000+ tokens per turn when only 20 lines are needed.
3. **Stuck Loops & Bloated Diffs:** Terminal `git diff` and repetitive failed commands spiral context into overflow.

### TokenSafe Solves All Three Autonomously:
- **Graph-Guided AST Skeletonization:** Folds inactive function bodies into signatures, keeping target functions and 1-hop dependencies full while collapsing unrelated code (`-85% to -92%`).
- **God Nodes & Architectural Hub Detection:** Calculates Degree Centrality to identify structural project hubs and injects topology awareness into agent sessions without expensive file crawling.
- **TOCP Prompt Distillation:** Intercepts prompts and distills intent into lean S-Expressions via **Google Gemini (2.0 Flash-Lite), OpenAI ChatGPT (gpt-4o-mini), or DeepSeek-Flash** (`-40% to -70%`). Automatically detects `GEMINI_API_KEY`, `OPENAI_API_KEY`, or `DEEPSEEK_API_KEY`.
- **API Surface Map Generator:** Generates high-density `tokensafe-out/API_SURFACE.md` (~1.5k tokens) for instant agent codebase onboarding.
- **Predictive Dependency Prefetcher:** Attaches imported symbol signatures to eliminate downstream exploratory `read_file` roundtrips.
- **Local Secret Redactor:** Sanitizes Stripe, AWS, JWT, and database credentials before network transmission.
- **Output Guard & Loop Guard:** Blocks destructive full-file rewrites on files $\ge 40$ lines and halts command loops.

---

## ⚡ TokenSafe vs. Graphify: Dynamic In-Flight Defense vs. Static Reports

How TokenSafe complements or supersedes static graph tools like **Graphify** (`graphify-labs/graphify`):

| Capability | Graphify | TokenSafe |
|---|---|---|
| **Primary Focus** | Static knowledge graph reports & MCP queries | **In-flight real-time token compression & defense** |
| **Indexing Cost** | Expensive LLM tokens / heavy graph extraction | **0 tokens, 100% deterministic local AST (< 30ms)** |
| **Execution Latency** | Offline / CLI-initiated build | **Sub-millisecond inline proxy & lifecycle hooks** |
| **Active Slicing** | Static file output (`GRAPH_REPORT.md`) | **Dynamic call-graph skeletonization (85–92% saved)** |
| **Multi-Project Workspace** | Requires manual directory targeting | **Auto-detects project roots per file on the fly** |
| **Agent Prompt Optimization** | None | **TOCP S-Expression prompt distillation (-40% to -70%)** |
| **Safety & Runaway Prevention** | None | **Output rewrite guard, loop detector, secret scrubbing** |

> **Complements & Replaces**: If you already use Graphify for repository mapping, TokenSafe works alongside it by compressing real-time agent tool reads on the wire. Or use TokenSafe standalone for a zero-configuration, zero-cost, sub-30ms architecture engine out of the box.

---

## 🤖 Universal LLM Distillation Providers

TokenSafe supports any key you already have:

| Provider | Supported Model | Environment Variable | Config Key |
|---|---|---|---|
| **Google Gemini** | `gemini-2.0-flash-lite`, `gemini-2.0-flash` | `GEMINI_API_KEY` | `gemini_api_key` |
| **OpenAI ChatGPT** | `gpt-4o-mini`, `gpt-4o` | `OPENAI_API_KEY` | `openai_api_key` |
| **DeepSeek** | `deepseek-flash`, `deepseek-chat` | `DEEPSEEK_API_KEY` | `deepseek_api_key` |

Check active providers anytime with:
```bash
tokensafe providers
```

---

## 📊 Zero-Loss SWE-bench Benchmark Results

Verified across 8 real-world production software engineering scenarios (131,604 baseline tokens down to 7,251 tokens, **-94.5% net token reduction**, **+28.1% turn latency speedup** with **100% Zero-Loss accuracy retention**):

| Benchmark Scenario | Task Complexity | Baseline Tokens | TokenSafe Tokens | Reduction | Turn Speedup | Accuracy |
|---|---|---|---|---|---|---|
| **Zero-Token Architecture Discovery** | Full repo God Node indexing | 49,716 | **486** | **-99.0%** | **6 ➔ 1 turns (+83%)** | 100% Pass |
| **Graph-Guided AST Symbol Slicing** | Target function + 1-hop callees | 8,016 | **451** | **-94.4%** | **4 ➔ 2 turns (+50%)** | 100% Pass |
| **Monorepo Lockfile Grep Leak** | `git grep` across lockfiles | 10,111 | **491** | **-95.1%** | 2 turns | 100% Pass |
| **Monorepo Lockfile Git Diff** | Monorepo branch diff | 18,911 | **761** | **-96.0%** | 2 turns | 100% Pass |
| **Multi-Turn History Compaction** | 8-turn session history fold | 31,010 | **3,130** | **-89.9%** | 8 turns | 100% Pass |
| **Race Condition / Distributed Lock** | AST query & Redis redlock | 5,074 | **404** | **-92.0%** | **4 ➔ 3 turns (+25%)** | 100% Pass |
| **OAuth JWT Expiration Triage** | Controller bug investigation | 3,757 | **401** | **-89.3%** | **3 ➔ 2 turns (+33%)** | 100% Pass |
| **CI Test Failure Extraction** | Full test log & stack trace | 5,009 | **1,127** | **-77.5%** | 3 turns | 100% Pass |

---

## 💎 Pricing & 7-Day Free Trial

| Tier | Price | Highlights |
|---|---|---|
| **Hacker (Open-Core)** | **$0** (Free Forever) | 100k tokens/mo, Local Regex compactor, use own DeepSeek / Gemini key |
| **Solo Developer** | **$9 / month** | **7-Day Free Trial**, Unlimited Managed DeepSeek-Flash / Gemini Flash-Lite, AST Prefetcher, Output Guards |
| **Engineering Team** | **$29 / dev / mo** | **7-Day Free Trial**, Team Budget Caps, Centralized Ledger, B2B Docker Proxy |
| **Enterprise** | Custom | On-Premise Nuitka C-compiled Docker Proxy, Custom Compliance, SAML SSO |

👉 **[Start your 7-Day Free Trial on our website](https://tokensafe.dev#pricing)**

---

## 🛠️ CLI Quickstart

```bash
# Analyze repository God Nodes and dependency topology
tokensafe graph

# Generate compact tokensafe-out/API_SURFACE.md interface snapshot
tokensafe map

# View live savings dashboard in your terminal
tokensafe stats

# Open interactive Apple dark-mode HTML dashboard
tokensafe html

# Run the 5-scenario benchmark suite
tokensafe benchmark

# Run the local transparent AI proxy (127.0.0.1:8765/v1)
tokensafe proxy

# Activate your Pro License key (TS2 Asymmetric Ed25519 Key)
tokensafe activate TS2.eyJlbWFpbCI6...
```

---

## 🔒 Security Architecture & Cloud API Endpoints

TokenSafe Cloud Backend enforces enterprise zero-trust security across all communications:

### 1. Cryptographic Authentication & Licensing
- **Asymmetric RFC 8032 Ed25519 Signatures (`TS2.*`)**: Client instances verify digital signatures offline using an embedded 32-byte public key. The private signing key never leaves the secure cloud enclave.
- **JWT Session Tokens**: Cloud endpoints authenticate sessions via standard HS256 JWT bearer tokens or Clerk session tokens.
- **Dual Bearer Support**: Client endpoints accept either a signed license key (`TS2.*` / `TS1.*`) or a JWT in the `Authorization: Bearer <token>` header.
- **Zero Hardcoded Secrets**: All SMTP, Stripe, Admin, and signing keys are strictly injected via environment variables.

### 2. Cloud API Endpoint Reference

| Method | Endpoint | Auth Required | Headers / Parameters | Description |
|---|---|---|---|---|
| `GET` | `/api/health` | Public | None | Healthcheck and service version status. |
| `GET` | `/api/dashboard` | **Bearer Auth** | `Authorization: Bearer <jwt\|key>` or `?license=<key>` | Authenticated user metrics, token savings, and license status. |
| `POST` | `/api/team/seat/create` | **Bearer Auth** | `Authorization: Bearer <jwt\|key>`, JSON `{ owner_email, member_email }` | Generates a cryptographically signed seat license for a team member. |
| `POST` | `/api/team/seat/revoke` | **Bearer Auth** | `Authorization: Bearer <jwt\|key>`, JSON `{ owner_email, license_key }` | Revokes an active team seat license key. |
| `POST` | `/api/telemetry/sync` | **License Auth** | JSON `{ license_key, email, events: [...] }` | Syncs batch telemetry events (enforces max 50 events/batch). |
| `POST` | `/webhook/stripe` | **HMAC Sig** | `Stripe-Signature: t=...,v1=...` | Verifies Stripe webhook payload against `STRIPE_WEBHOOK_SECRET`. |
| `GET` | `/api/admin/overview` | **Admin Only** | `X-Admin-Secret` or Admin JWT | Global overview of MRR, seats, active licenses, and system health. |
| `POST` | `/api/admin/license/issue`| **Admin Only** | `X-Admin-Secret` or Admin JWT | Manually issues or provisions a custom tier license. |
| `POST` | `/api/admin/license/revoke`| **Admin Only**| `X-Admin-Secret` or Admin JWT | Immediately revokes a license. |
| `POST` | `/api/admin/license/extend`| **Admin Only**| `X-Admin-Secret` or Admin JWT | Extends an active license duration. |

### 3. Server Configuration & Environment Variables

| Variable | Required | Default / Description |
|---|---|---|
| `TOKENSAFE_PRIVATE_KEY` | Production | 64-character hex Ed25519 seed for issuing `TS2.` licenses. |
| `ADMIN_SECRET_KEY` | Yes | High-entropy secret for administrative API access (`X-Admin-Secret`). |
| `ADMIN_EMAILS` | Optional | Comma-separated admin emails (e.g. `nekosin4eg@gmail.com`). |
| `STRIPE_WEBHOOK_SECRET` | Production | Webhook secret (`whsec_...`) used to verify Stripe webhooks. |
| `SMTP_PASS` | Production | Password/token for transactional mail delivery via ImprovMX SMTP. |
| `JWT_SECRET` | Optional | Shared HMAC secret for verifying custom JWT tokens. |


---

## 🧪 Unit Testing

```bash
python3 -m unittest discover tests
```
*Zero external runtime dependencies — runs on standard Python 3.10+.*

---

## 📄 License
MIT License. Commercial enterprise licenses available for on-premise B2B deployments.

### Export Markdown Report
```bash
./bin/agy-tokens markdown
```

---

## 📂 Project Structure

```text
agy-token-optimizer/
├── plugin.json                 # Antigravity plugin manifest
├── hooks.json                  # Lifecycle hook declarations (PreInvocation, PreToolUse, PostToolUse)
├── rules/
│   └── AGENTS.md               # TOCP Protocol directives for LLM (prompt-cached)
├── skills/
│   └── token-optimizer/
│       └── SKILL.md            # On-demand Antigravity skill
├── proto/
│   └── tocp.proto              # Protocol Buffers contract
├── src/
│   ├── tokenizer.py            # Zero-dependency calibrated BPE token counter
│   ├── compactor.py            # Semantic compactor (Natural Language -> TOCP S-Expr)
│   ├── slicer.py               # AST & Line windowing tool optimizer
│   ├── metrics.py              # Persistent SQLite metrics engine
│   └── dashboard.py            # Visual ANSI terminal & Markdown generator
├── hooks/
│   ├── pre_invocation.py       # PreInvocation hook entry point
│   ├── pre_tool_use.py         # PreToolUse hook entry point
│   └── post_tool_use.py        # PostToolUse hook entry point
├── bin/
│   └── agy-tokens              # Executable CLI
├── tests/                      # Unit test suite (100% passing)
├── install.sh                  # One-click installer for Antigravity
└── README.md
```

---

## 🛠️ Installation & Antigravity Setup

Run the installer:
```bash
./install.sh
```
This symlinks the plugin directly into `~/.gemini/config/plugins/agy-token-optimizer`.  
Antigravity automatically loads it on startup.

---

## 🧪 Running Unit Tests

```bash
python3 -m unittest discover tests
```
No external packages required — runs on standard Python 3.10+.
