Metadata-Version: 2.4
Name: hot-context
Version: 0.1.0
Summary: LRU-based intelligent context manager for LLM agents. Keeps hot context, evicts cold.
Author: Ankitkumar Patel
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: agents,context-management,conversation,llm,lru,strands
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: strands-agents>=0.1.0
Provides-Extra: dev
Requires-Dist: pytest-cov; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Provides-Extra: embeddings
Requires-Dist: numpy>=1.24; extra == 'embeddings'
Requires-Dist: sentence-transformers>=2.0; extra == 'embeddings'
Description-Content-Type: text/markdown

# 🔥 hot-context

**LRU-based intelligent context manager for LLM agents.** Keeps hot context, evicts cold.

Most conversation managers use a naive sliding window — drop the oldest messages when context fills up. `hot-context` treats your context window like an intelligent cache: every chunk is scored by relevance, recency, and access frequency, and only the highest-value context survives.

## Key Features

- **Relevance-aware trimming** — trims tool results to only the lines the LLM actually used, instead of evicting entire entries
- **Adaptive token calibration** — learns the real chars-per-token ratio from actual LLM feedback, self-corrects over time
- **Entity echo feedback loop** — uses the LLM's own response as a proxy for attention weights, zero extra API calls
- **Turn-aware eviction** — never evicts during a turn (LLM needs full context to reason), only trims between turns
- **Framework-agnostic core** with drop-in adapters for Strands, LangChain, and OpenAI

## How It Works

```
Turn N:
  User query → Tools return data → LLM sees FULL context → Answers
                                                              ↓
                                          Entity echo: what did the LLM use?
                                          Trimmer: keep only relevant lines from tool results
                                          Calibrator: learn real token ratio from Bedrock
                                                              ↓
Turn N+1:
  Context has trimmed tool results (5k instead of 80k) + all assistant answers preserved
```

**Scoring formula (no LLM needed):**
```
score = α·similarity(chunk, query) + β·recency(chunk) + γ·access_frequency(chunk) - δ·size_penalty(chunk)
```

- **Similarity**: Jaccard overlap on extracted entities (BM25-inspired, zero API calls)
- **Recency**: Exponential decay by turn distance
- **Access frequency**: Time-decayed count of how often the LLM actually used this chunk
- **Size penalty**: Large tool results are penalized proportionally to budget

**Topic shift detection**: When the user changes topics, similarity weight increases and frequency weight decreases — preventing stale popular chunks from dominating.

## Results

Tested against Strands `SlidingWindowConversationManager` on a real multi-turn agent with MCP tools (SQL queries, knowledge base lookups):

| Metric | SlidingWindow | hot-context |
|--------|--------------|-------------|
| Q3 (heavy query) | **CRASHED** — context overflow | ✅ Answered (2831 chars) |
| Peak context Q2 | Unbounded (168k+) | **105k** (trimmed) |
| Peak context Q3 | Overflow (>200k) | **113k** (controlled) |
| Queries completed | 2.5 of 3 | **3 of 3** |

Peak context went DOWN from Q1→Q2 (128k→105k) — the trimmer removed irrelevant lines from Q1's tool result.

## Quick Start

### Strands (drop-in replacement)

```python
from strands import Agent
from strands.models.bedrock import BedrockModel
from hot_context.adapters.strands import StrandsHotContextManager
from hot_context import ScoringWeights

agent = Agent(
    model=BedrockModel(model_id="us.anthropic.claude-sonnet-4-5-20250929-v1:0"),
    conversation_manager=StrandsHotContextManager(
        token_budget=100_000,
        weights=ScoringWeights.tool_agent(),
    ),
)
```

### Framework-agnostic core

```python
from hot_context import HotContextManager, ScoringWeights

manager = HotContextManager(
    token_budget=100_000,
    weights=ScoringWeights.tool_agent(),
    pin_last_n=2,
)

# Before each query
manager.on_query(user_input)

# After LLM responds
manager.on_response(str(result))
```

### Adaptive calibration (Strands)

```python
# Token estimator self-calibrates from actual Bedrock token counts.
# After each LLM call, feed back the reported input_tokens:
cm = agent.conversation_manager
cm.calibrate(agent.messages, actual_input_tokens=87450)
# Future estimates are now more accurate — no manual tuning needed.
```

### Weight Presets

```python
ScoringWeights.tool_agent()     # high similarity — for agents with many tool calls
ScoringWeights.conversational() # high recency + frequency — for multi-turn chat
ScoringWeights.research()       # high similarity + frequency — for RAG/research agents
```

### Custom Entity Patterns

```python
import re
from hot_context import HotContextManager

manager = HotContextManager(
    extra_patterns=[
        ("order_id", re.compile(r"\b\d{3}-\d{7}-\d{7}\b")),
        ("asin", re.compile(r"\bB[A-Z0-9]{9}\b")),
    ]
)
```

## Architecture

```
┌──────────────────────────────────────────────────────────────┐
│                    HotContextManager                         │
│                                                              │
│  TokenEstimator ──→ EntityExtractor ──→ ContextScorer        │
│       ↑                                      │               │
│  calibrate()                            score + rank         │
│  (from LLM)                                  │               │
│                                              ↓               │
│  Trimmer ←── FeedbackLoop ←── LLM Response ←── Evictor      │
│  (line-level    (entity echo)                (entry-level)   │
│   relevance)                                                 │
└──────────────────────────────────────────────────────────────┘

Adapters: Strands | LangChain | OpenAI
```

## Why Not Just Summarize?

| Approach | Token cost | Latency | Quality |
|----------|-----------|---------|---------|
| Sliding window | Free | None | Poor — drops relevant old context |
| LLM summarization | High | +500ms/turn | Good — but expensive at scale |
| **hot-context** | **Free** | **~1ms/turn** | **Good — keeps what matters, trims what doesn't** |

## Roadmap

### P2 — Benchmarks & Paper
- [ ] **Benchmark harness** — synthetic multi-turn conversations with known answers. Measure token savings + answer quality (ROUGE/BERTScore on trimmed vs full context).
- [ ] **Standard benchmarks** — LongBench, SCROLLS, QuALITY, NarrativeQA for long-context evaluation.
- [ ] **Ablation studies** — trimmer-only vs scorer-only vs full pipeline vs baselines (MemGPT, LLMLingua, attention sink).
- [ ] **Token savings curves** — plot peak context vs conversation length across strategies.

## Components

| File | Purpose |
|------|---------|
| `manager.py` | Core orchestrator: `on_query()` → score + evict, `on_response()` → feedback |
| `scorer.py` | Scoring formula with topic shift detection and pluggable similarity |
| `token_estimator.py` | Adaptive chars-per-token calibration from real LLM feedback |
| `trimmer.py` | Line-level relevance trimming of tool results |
| `chunker.py` | Smart chunking respecting tool-use/tool-result pairing |
| `entity_extractor.py` | Regex-based entity extraction (dates, IDs, numbers, custom patterns) |
| `feedback.py` | Entity echo feedback loop — updates access logs from LLM response |
| `types.py` | `ContextEntry`, `AccessRecord`, `ScoringWeights` with presets |
| `adapters/strands.py` | Drop-in `ConversationManager` for Strands agents |
| `adapters/langchain.py` | `BaseChatMemory` adapter for LangChain |
| `adapters/openai.py` | Message list manager for OpenAI Chat API |


[![PyPI version](https://badge.fury.io/py/hot-context.svg)](...)
[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
[![arXiv](https://img.shields.io/badge/arXiv-2026.XXXXX-b31b1b.svg)](...)