Metadata-Version: 2.4
Name: savetoken
Version: 0.1.1
Summary: Convert Repomix/repominify output into compact AI-friendly codebase summaries using free LLMs.
License: MIT
Project-URL: Homepage, https://github.com/yourname/savetoken
Project-URL: Issues, https://github.com/yourname/savetoken/issues
Keywords: ai,llm,codebase,repomix,prompt,tokens,summarizer
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Documentation
Classifier: Topic :: Utilities
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Requires-Dist: mypy; extra == "dev"

# savetoken 🪙

> Transform your Repomix codebase dump into a high-fidelity AI context brief —
> **~83% fewer tokens** than sending the raw zip, **~88% fidelity** to the original.

Instead of pasting hundreds of thousands of tokens of source code into an AI,
`savetoken` produces a structured `codebrief.md` that the AI can reason about
just as effectively, at a fraction of the cost.

---

## How it fits in your workflow

```
Your project
    ↓
npx repomix                          # pack codebase → repomix-output.xml
    ↓
savetoken summarize repomix-output.xml   # analyse → codebrief.md
    ↓
Paste codebrief.md into your AI prompt   # 17% of tokens, ~88% fidelity
```

---

## Install

```bash
pip install savetoken
```

Requires Python 3.10+. **Zero runtime dependencies** — stdlib only (`ast`, `urllib`, `json`, `hashlib`, `re`).

---

## Quick start

```bash
# 1. Pack your codebase with Repomix
npx repomix

# 2. Generate context brief (Gemini Flash is free and recommended)
export GEMINI_API_KEY=your_key_here
savetoken summarize repomix-output.xml

# Output: codebrief.md — paste into any AI prompt
```

### Python API

```python
from savetoken import SaveToken

st = SaveToken(provider="gemini", api_key="...")
summary = st.summarize("repomix-output.xml")
st.save(summary, "codebrief.md")
```

---

## What's in the output

savetoken extracts seven categories of information, each via the most
accurate method available:

| Section | Source | Fidelity |
|---|---|---|
| Strategic files (schemas, configs, entry points, tests) | verbatim | 100% |
| Hot functions (most-called by PageRank) | verbatim | 100% |
| File docs (module/class/fn docstrings, signatures, constants) | AST | ~95% |
| Execution flows (entry → service → repo → db) | call graph trace | ~90% |
| Call graph (cross-file function calls, prod/test split) | AST | ~90% |
| Environment contract (all env vars, defaults, required) | AST | 100% |
| SQL schema (tables, columns, FKs, indexes) | AST | ~95% |
| Critical blocks (HACK/FIXME with ±10 lines context) | verbatim | 100% |

**No LLM is called for Python files.** Docstrings and signatures are
extracted directly from source — zero hallucination, instant, cacheable.
LLM calls are only used for: project overview, flow descriptions, and
non-Python file summaries.

---

## Providers

| Provider | Model | Free tier | Env var |
|---|---|---|---|
| `gemini` ⭐ | Gemini 2.0 Flash | ✅ Generous, 1M ctx | `GEMINI_API_KEY` |
| `groq` | Llama 3.3 70B | ✅ Fast | `GROQ_API_KEY` |
| `mistral` | Mistral Small Latest | ✅ Available | `MISTRAL_API_KEY` |
| `openai` | gpt-4o-mini | Paid | `OPENAI_API_KEY` |
| `deepseek` | deepseek-chat | Very cheap | `DEEPSEEK_API_KEY` |

Gemini is recommended — 1M token context handles any codebase size.

---

## CLI reference

```bash
savetoken summarize repomix-output.xml
savetoken summarize repomix-output.xml \
  --provider groq \
  --output codebrief.md \
  --format markdown \
  --lang pt \
  --target-ratio 0.50 \
  --flow-depth 5 \
  --force        # ignore cache, regenerate everything
savetoken cache-stats
savetoken clear-cache
```

| Flag | Default | Description |
|---|---|---|
| `--provider` | `gemini` | LLM provider |
| `--output` | `codebrief.md` | Output path |
| `--format` | `markdown` | `markdown` or `json` |
| `--lang` | `en` | Language for generated descriptions |
| `--target-ratio` | `0.50` | Max output/input token ratio |
| `--flow-depth` | `5` | Max hops in execution flow traces |
| `--force` | off | Bypass cache |
| `--cache-dir` | `.savetoken_cache` | Cache directory |
| `--quiet` | off | Suppress per-file progress |

---

## Caching

Every file is hashed by content. On re-runs, only changed files are
re-processed. The cache lives in `.savetoken_cache/` — commit it to share
across your team. Cache keys are schema-versioned: any model change
automatically invalidates stale entries.

```bash
savetoken cache-stats   # entries, size, schema version
savetoken clear-cache
```

---

## Real-world numbers (datalock — 59 files, 271k tokens)

```
Raw codebase (zip):  271,252 tokens   100%   fidelity 100%
savetoken output:     47,204 tokens    17%   fidelity  ~88%
Savings:                              83%
```

Token breakdown:
- Strategic verbatim (11 files): 26%
- File Docs / AST (48 files):    44%
- Call graph (1,139 edges):      25%
- Flows, env, overhead:           5%

---

## VS Code extension

Located in `vscode-extension/`. Commands:
- **SaveToken: Summarize Codebase** — runs on `repomix-output.xml`
- **SaveToken: Open Summary**
- **SaveToken: Clear Cache**

Settings: `savetoken.provider`, `savetoken.apiKey`, `savetoken.language`,
`savetoken.outputFile`, `savetoken.autoSummarizeOnRepomix`.

---

## Project structure

```
savetoken/
├── savetoken/
│   ├── core.py                 Orchestrator — full pipeline
│   ├── models.py               All dataclasses (FileSummary, Flow, etc.)
│   ├── classifier.py           STRATEGIC / AST / REGULAR / SKIP per file
│   ├── budget.py               Token budget allocation (post-classification)
│   ├── analyzer.py             AST call graph + inheritance (prod/test split)
│   ├── ast_types.py            Function signature extraction via AST
│   ├── docstring_extractor.py  Module/class/fn docstrings + constants (NEW)
│   ├── ranker.py               PageRank centrality → hot functions
│   ├── flows.py                Execution flow tracer (BFS on call graph)
│   ├── env_extractor.py        Environment variable extractor
│   ├── schema_sql.py           SQL schema (Django/SQLAlchemy/Alembic/raw SQL)
│   ├── critical_context.py     HACK/FIXME blocks with ±10 line context
│   ├── deduplicator.py         Remove cross-section redundancy
│   ├── output.py               Markdown + JSON renderers
│   ├── cache.py                File-based cache with schema versioning
│   ├── cli.py                  CLI entry point
│   ├── exceptions.py           SaveTokenError hierarchy
│   └── providers/
│       ├── base.py             Abstract provider + prompts
│       ├── gemini.py           Gemini 2.0 Flash
│       ├── groq.py             Groq / Llama 3.3 70B
│       ├── mistral.py          Mistral Small
│       └── openai.py           OpenAI-compatible (OpenAI, DeepSeek, Ollama)
├── vscode-extension/           VS Code extension
├── tests/                      117 unit + integration tests
├── examples/                   Usage examples
└── pyproject.toml
```

---

## License

MIT
