Metadata-Version: 2.4
Name: writing-context-rtfm
Version: 0.6.1
Summary: MCP extension for writing-context retrieval relying on RTFM
Author: João Carlos N. Bittencourt
License: MIT
License-File: LICENSE
Keywords: context-reduction,llm-writing,mcp,mcp-server,rtfm
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.13
Requires-Dist: mcp>=1.27.1
Requires-Dist: pylatexenc
Requires-Dist: pyyaml
Requires-Dist: rtfm-ai[embeddings]
Provides-Extra: dev
Requires-Dist: basedpyright>=1.0; extra == 'dev'
Requires-Dist: mypy>=1.0; extra == 'dev'
Requires-Dist: pre-commit>=3.0; extra == 'dev'
Requires-Dist: pytest-cov>=4.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.8.0; extra == 'dev'
Requires-Dist: types-pyyaml; extra == 'dev'
Provides-Extra: tiktoken
Requires-Dist: tiktoken; extra == 'tiktoken'
Description-Content-Type: text/markdown

<!-- mcp-name: writing-context-rtfm -->
<div align="center">

***Surgical Context for Writing Agents***

Stop giving your AI agent the entire manuscript to write one section. Give it the exact paragraphs, constraints, and dependencies it needs to succeed. No token bloat. No hallucinations.

**`Lightweight · Task-Focused · Extension · MIT`**

<br>

[![PyPI Version](https://img.shields.io/pypi/v/writing-context-rtfm.svg)](https://pypi.org/project/writing-context-rtfm/) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) [![Python](https://img.shields.io/badge/Python-3.11+-blue.svg)](https://www.python.org/) [![MCP](https://img.shields.io/badge/MCP-2026-green.svg)](https://modelcontextprotocol.io/) [![Powered by RTFM](https://img.shields.io/badge/Powered%20by-RTFM-purple.svg)](https://github.com/roomi-fields/rtfm)

</div>

---

<!-- ─────────── TIER 1 — Pain & promise ─────────── -->

Your writing agent is drowning in tokens.

You ask Claude or Cursor to "Write the methodology section." To give it context, you feed it your 50-page manuscript, your related works, and your notes. The agent gets overwhelmed by the global narrative, loses track of the specific hyper-parameters you wanted to include, and writes a generic, repetitive summary that reads like a high-school essay. 

The bottleneck isn't the model's writing ability — it's the **noise**.

**`writing-context-rtfm` fixes the noise.** It is a lightweight MCP extension built on top of `rtfm-ai`. Instead of letting the agent grep freely, it acts as a gatekeeper. It takes the agent's task, queries the underlying RTFM index, aggressively filters out background noise, and packs only the *essential* and *supporting* source chunks into a tight, highly-focused prompt.

```bash
writing-context-rtfm pack \
  --task "Write the methodology section detailing dataset and quantization" \
  --target sections/methodology.tex \
  --budget 4000
```

3 seconds later, the agent receives a compact context pack containing exactly the paragraphs and key terms needed, alongside stylistic constraints for the target section. The agent writes perfectly.

> **Token budgets respected. Constraints enforced. Progressive disclosure over context dumps.**

---

## Installation & Onboarding

`writing-context-rtfm` is published on PyPI and runs as a Model Context Protocol (MCP) server.

### 1. Install writing-context-rtfm
You can install the package globally or in your virtual environment:

```bash
# Using uv (recommended)
uv tool install writing-context-rtfm

# Using pipx
pipx install writing-context-rtfm
```

### 2. Install the RTFM CLI (Retrieval Engine)
Since `writing-context-rtfm` queries and relies on the `rtfm-ai` database, you must install the `rtfm-ai` command-line tool to initialize and synchronize your manuscript's retrieval index:

```bash
# Using uv (recommended)
uv tool install "rtfm-ai[embeddings]"

# Using pipx
pipx install "rtfm-ai[embeddings]"
```
*(Note: If you are setting up inside a local virtual environment, running `uv pip install "writing-context-rtfm[tiktoken]"` will automatically pull in `rtfm-ai[embeddings]` as a library dependency, but installing it globally ensures the `rtfm` binary is available on your PATH).*

### 3. Install Zotero MCP (Semantic Literature Grounding)
`writing-context-rtfm` uses `zotero-mcp` to ground your agent in your local literature library. It supports dynamic semantic search over your PDFs and metadata.
You must install `zotero-mcp` and have Zotero Desktop running for literature grounding to work:

```bash
# Install zotero-mcp globally using uv
uv tool install zotero-mcp
```
*(Make sure Zotero Desktop is open while the agent is running so it can connect to the local SQLite database).*

---

### 4. Quick Project Onboarding
To integrate the server into your manuscript repository, run the following commands:

#### Step A: Initialize configuration and editor rules
```bash
writing-context-rtfm init
```
This command non-destructively:
* Creates a self-documenting `.writing-context/config.yaml` file template showing how to tune token budgets and role weights.
* Appends the cache database path to your `.gitignore`.
* Updates your local `.mcp.json` to register the MCP server automatically.
* Adds **Agent Rules of Thumb** blocks into `CLAUDE.md`, `AGENTS.md`, and `GEMINI.md` to guide AI agents on retrieving context first and respecting LaTeX boundaries.

#### Step B: Auto-scaffold your section cards
```bash
writing-context-rtfm cards build
```
This scans your workspace for LaTeX files, parses `\input` structures, maps section dependencies, and uses model-assisted inference to automatically scaffold purposes, key terms, and constraints. It outputs the generated structure to `.writing-context/cards.generated.yaml`. 

*(Note: If you do not have or want to use an OpenAI API key, `writing-context-rtfm` supports a model fallback chain for card scaffolding: OpenAI API -> Hugging Face Serverless Inference API (requires `HF_TOKEN`, defaults to `Qwen/Qwen2.5-Coder-7B-Instruct`) -> Local Ollama server (running at `http://localhost:11434`, defaults to `qwen2.5-coder` or `phi3`) -> Deterministic Offline Scan fallback).*

#### Step C: Initialize, Sync and Setup Embeddings
Initialize the RTFM index inside your repository and generate the semantic search embeddings:
```bash
# 1. Initialize RTFM configuration
rtfm init

# 2. Run the initial sync to build the index database
rtfm sync
```
*(Note: `writing-context-rtfm init` only configures the writing-context settings, cards, and agent rules; it does not automatically initialize or sync the underlying RTFM database. This setup assumes you already have at least part of the `.tex` files in your repository—if starting from an empty repository or using Overleaf, ensure your files are placed locally first).*

##### Baseline Model Embeddings
* **Default Local Model**: By default, RTFM automatically generates embeddings for all document chunks. It uses a fast, lightweight multilingual model (`sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2`) which runs completely locally on your CPU/GPU and downloads automatically from Hugging Face on the first sync. No external API keys are required.
* **Customizing Models**: If you want to use a larger or different model, you can run the embedding step explicitly:
  ```bash
  # Options: fast (default), balanced (BAAI/bge-base-en-v1.5), quality (mixedbread-ai/mxbai-embed-large-v1)
  rtfm embed --embed-model balanced
  ```

##### OpenAI Semantic Search (Optional Extension)
If you prefer to leverage OpenAI embeddings for semantic expansion instead of running local transformer models, `writing-context-rtfm` includes a built-in provider that seamlessly overlays OpenAI vectors onto RTFM's index.
1. Securely save your API key to the local cache: `writing-context-rtfm auth openai_semantic "sk-..."`
2. Enable it in your `.writing-context/config.yaml`:
   ```yaml
   providers:
     openai_semantic:
       enabled: true
       model: "text-embedding-3-small"
       auto_sync: false # Set to true to embed all files automatically during `rtfm sync`
   ```
*Note: This architecture uses an ultra-fast `numpy` in-memory comparison, requiring zero C++ SQLite VSS extensions, ensuring maximum compatibility across all operating systems in your laboratory.*

##### Card Scaffolding Generator Configuration (Optional)
To customize the model or API endpoint used during `cards build` / `cards infer` (for example, to use a local Ollama instance or the Hugging Face Inference API instead of OpenAI), configure the `generator` block in your `.writing-context/config.yaml`:
```yaml
generator:
  # The model to use (e.g., gpt-4o-mini, Qwen/Qwen2.5-Coder-7B-Instruct, phi3)
  model: "Qwen/Qwen2.5-Coder-7B-Instruct"
  # The API endpoint base URL (e.g. https://api-inference.huggingface.co/v1, http://localhost:11434/v1)
  api_base: "https://api-inference.huggingface.co/v1"
  # The API key/token (optional; falls back to environment variables or local auth cache)
  # api_key: "your-token"
```
*   **Hugging Face Inference API**: Query using a free Hugging Face token. You can save your token locally using: `writing-context-rtfm auth huggingface "hf_..."`.
*   **Local Ollama Server**: Start Ollama and run `ollama pull phi3` or `ollama pull qwen2.5-coder`. Point `api_base` to `http://localhost:11434/v1` and set `model` to your pulled model name. No API key is required.

---

## MCP Server Integration

### 1. Claude Desktop
Add this to your `claude_desktop_config.json` (on macOS: `~/Library/Application Support/Claude/claude_desktop_config.json`):

```json
{
  "mcpServers": {
    "writing-context-rtfm": {
      "command": "writing-context-rtfm",
      "args": [
        "serve"
      ]
    }
  }
}
```

### 2. Cursor IDE
1. Open Cursor Settings (`Cmd + ,`).
2. Navigate to **Features** > **MCP** and click **+ Add New MCP Server**.
3. **Name:** `writing-context-rtfm`
4. **Type:** `command`
5. **Command:** `writing-context-rtfm serve`

### 3. VS Code Extensions (Cline, Roo Code)
Update your MCP settings file (e.g., `cline_mcp_settings.json`):

```json
{
  "mcpServers": {
    "writing-context-rtfm": {
      "command": "writing-context-rtfm",
      "args": [
        "serve"
      ]
    }
  }
}
```

### 4. Claude Code (Anthropic CLI Agent)
```bash
# Global configuration
claude mcp add --scope user --transport stdio writing-context-rtfm -- writing-context-rtfm serve

# Repository-local configuration
claude mcp add --scope local --transport stdio writing-context-rtfm -- writing-context-rtfm serve
```

---

<!-- ─────────── TIER 2 — Positioning & buzz ─────────── -->

## The Core Philosophy: RTFM Retrieves, We Pack

| **Tool** | **Role** | **Action** | **Output** |
|----------|----------|------------|------------|
| `rtfm-ai` | The Retrieval Layer | Indexes everything, runs FTS/Semantic search, returns raw hits. | 25 raw chunks |
| `writing-context-rtfm` | The Curation Layer | Filters noise, applies constraints, ranks by structural priority. | 4 essential chunks |

We do **not** replace or fork RTFM. We wrap it. RTFM is built to fetch memory. `writing-context-rtfm` is built to decide *what is enough memory to write a specific section*. 

---

## Features

### 1. Agent Self-Correction Loop
If the requested token budget is too small to fit the necessary target context, the packer returns a `"status": "degraded"` and appends a warning to the `warnings` list recommending a specific minimum budget:
* **Writing Packs (`pack`)**: `To resolve this, call the tool with a larger token_budget of at least X.`
* **Proofreading Packs (`proofread-pack`)**: `To resolve this, call the tool with a larger max_tokens value of at least X.`

AI client agents are instructed via the auto-injected guidelines to parse this warning, extract the recommended value `X`, and automatically retry the call with the new budget to get the missing context.

### 2. LaTeX Safety Checks
The extension automatically parses the target text and detects LaTeX math environments, macro calls, and cross-references (e.g., `\ref{...}`, `\begin{equation} ... \end{equation}`). It issues safety warnings to the AI writing agent containing the exact code patterns that must not be deleted or broken during edits, maintaining compilation safety.

### 3. SQLite Local Caching
To optimize token and response latency, generated context packs are hashed and cached locally in `.writing-context/context_cache.sqlite`. Cache keys are dynamically invalidated whenever the project configuration, section cards, or underlying RTFM indexes are updated.

### 4. Semantic Zotero Grounding
The extension dynamically routes literature queries to `zotero-mcp`. 
* **Semantic Routing**: High-level section intents (e.g., "Discuss how smart cities use IoT") are routed to ChromaDB-powered semantic searches.
* **Proofread Protection**: If you run a `proofread-pack`, the engine intelligently skips open-ended searches to prevent "context contamination" (hallucinating new ideas into your polish phase), only resolving the explicit `\cite{}` keys already present in the draft.

---

## The Split-Cards Pattern (Overrides & Generated)

To give writing agents context and rules, we define manuscript metadata. Rather than forcing you to maintain a single massive YAML configuration manually, `writing-context-rtfm` splits section cards into two layers:

1. **`cards.generated.yaml` (Machine-Written)**: Generated automatically by `writing-context-rtfm cards build`. The tool scans your LaTeX file tree, maps structural hierarchies, and uses local/remote models to extract default purposes, key terms, facts, and constraints. **Do not modify this file.**
2. **`cards.overrides.yaml` (Human-Controlled)**: The user control panel. Create or edit this file to override generated settings or declare global document parameters (like style guidelines, project-wide glossary, or specific section rules).

At runtime, the extension automatically overlays `cards.overrides.yaml` on top of the generated metadata, compiling them into a single unified context card database.

### Override File Example (`cards.overrides.yaml`)
```yaml
version: 2

# Project-wide global context rules
document:
  title: "A New Approach to Manuscript Curation"
  thesis: "Surgical context selection using a gatekeeping protocol reduces LLM token overhead."
  writing_style:
    tone: "Academic, precise, third-person"
    avoid_words: ["groundbreaking", "revolutionary", "game-changing"]
  terminology:
    Context Pack: "A compact JSON structure containing prioritized source spans and constraints."

# Override specific sections generated by the tool
sections:
  section_methodology:
    title: "Proposed Methodology"
    purpose: "Detail the system architecture and context selection algorithms."
    depends_on:
      - section_introduction
    must_preserve:
      - "Token budget formula is B_usable = B_total * (1 - margin)"
    avoid: ["premature results discussion"]
    constraints:
      - "Write equations using LaTeX align environments"
```

### Configurable Zotero Grounding
If Zotero is enabled in your `.writing-context/config.yaml`, the provider routes keyword/semantic queries to Zotero. To prevent out-of-domain paper recommendations from cluttering your prompt, configure Zotero's semantic filtering under `extra` in `config.yaml`:
```yaml
providers:
  zotero:
    enabled: true
    mcp_server:
      command: zotero-mcp
    extra:
      # Optimal threshold: -0.4. Discards negative similarity noise (like off-topic papers) 
      # while preserving relevant priority queueing and network scheduling matches.
      similarity_threshold: -0.4
      include_abstract: false  # Set to true to include full abstracts in Zotero spans
```

### How the Agent Applies Cards
When the agent requests context to write `section_methodology`, the system:
1. **Expands the query**: Uses the target section title, override keywords, and task text to search.
2. **Enforces Exclusions (Avoids)**: Instantly discards any retrieved text matching your defined `avoid` list.
3. **Injects Rules & Terminology**: Directs the agent to respect constraints (e.g. `must_preserve`) and glossary terms during writing.

> [!IMPORTANT]
> **Modular documents are required.** This extension's noise-reduction algorithms heavily rely on the `path` defined in your section cards to perform "Target Boosts" and semantic scoping. If your entire manuscript is just a single monolithic `main.tex` or `main.md` file, the packer won't be able to distinguish the target section from background noise. Keep your writing modular (e.g., `sections/01_intro.tex`, `sections/02_methodology.tex`) for optimal results.

---

## CLI Reference

```bash
# Initialize project config, gitignore, and editor rules
writing-context-rtfm init

# Build section cards (scan, infer, and update in sequence)
writing-context-rtfm cards build

# Deterministically scan the manuscript structure
writing-context-rtfm cards scan

# Interactively review generated card candidates
writing-context-rtfm cards review

# Initialize the local SQLite cache database (.writing-context/context_cache.sqlite)
writing-context-rtfm init-db

# Run diagnostics health checks on databases and configuration files
writing-context-rtfm doctor

# Sync the underlying RTFM index
writing-context-rtfm sync

# Generate a context pack directly in the terminal
writing-context-rtfm pack \
  --task "Update the introduction" \
  --target sections/introduction.tex \
  --budget 4000

# Generate a proofreading context pack
writing-context-rtfm proofread-pack sections/abstract.tex --line-start 1 --line-end 10 --max-tokens 3000

# Inspect configured rules and details for a specific section card
writing-context-rtfm inspect-target --target section_abstract

# Look up a term in the document glossary config
writing-context-rtfm get-term "Context Pack"

# Show the LaTeX reference graph and section dependencies
writing-context-rtfm show-graph

# Clear the cached context packs
writing-context-rtfm cache clear

# Show cache database size and run statistics
writing-context-rtfm cache stats

# Start MCP Server
writing-context-rtfm serve
```

---

## Where this fits

```
┌─────────────────────────────────┐
│       AI Agent / LLM Client     │  ← Execution (Cursor, Claude)
├─────────────────────────────────┤
│     writing-context-rtfm        │  ← Curation (Packs, Filters, Rules)
├─────────────────────────────────┤
│           rtfm-ai               │  ← Retrieval (Index, FTS, Semantic)
└─────────────────────────────────┘
```

Without the context packer, your agent retrieves 50 documents and hopes for the best. With it, the agent receives a surgically precise, prioritized briefing.

## License
[MIT License](LICENSE) — use it, fork it, extend it.
