Metadata-Version: 2.4
Name: opencode-semantic-memory
Version: 0.5.0
Summary: Persistent semantic memory system for OpenCode sessions
Author-email: Gregory Havenga <ghavenga@gitlab.com>
License-Expression: MIT
Keywords: ai,assistant,mcp,memory,opencode
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.11
Requires-Dist: httpx>=0.27.0
Requires-Dist: lancedb>=0.4.0
Requires-Dist: markdown-it-py>=3.0.0
Requires-Dist: mcp>=1.0.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: pylance>=0.20.0
Requires-Dist: sentence-transformers>=2.2.0
Requires-Dist: starlette>=0.36.0
Requires-Dist: tomli>=2.0.0
Requires-Dist: tree-sitter-javascript>=0.21.0
Requires-Dist: tree-sitter-python>=0.21.0
Requires-Dist: tree-sitter-ruby>=0.21.0
Requires-Dist: tree-sitter>=0.21.0
Requires-Dist: uvicorn>=0.27.0
Requires-Dist: watchdog>=3.0.0
Provides-Extra: dev
Requires-Dist: mypy>=1.0.0; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.21.0; extra == 'dev'
Requires-Dist: pytest>=7.0.0; extra == 'dev'
Requires-Dist: ruff>=0.1.0; extra == 'dev'
Description-Content-Type: text/markdown

# opencode-memory

Persistent semantic memory system for [OpenCode](https://opencode.ai) AI assistant sessions.

## Overview

opencode-memory is an MCP (Model Context Protocol) server that provides long-term memory capabilities for OpenCode sessions. It automatically learns from your workflow, stores decisions and procedures, and provides relevant context when needed.

**Key benefits:**
- Remember decisions, blockers, procedures, and facts across sessions
- Semantic search finds relevant memories even with different wording
- Session coordination prevents conflicts when running multiple OpenCode instances
- Project-scoped directives for per-project instructions
- Zero manual effort - learns passively from your workflow

## Features

- **Hybrid retrieval**: Combines full-text search (FTS5) with vector similarity search
- **Instant storage**: Memories stored immediately, embeddings computed async
- **Session coordination**: Tracks active sessions, supports item claiming to prevent conflicts
- **Contextual directives**: Global + project-specific instructions loaded at boot
- **Entity tracking**: Links memories to MRs, issues, epics (!123, #456, &789)
- **GitLab enrichment**: Fetches metadata for entities from GitLab API
- **Memory age**: All outputs show age to identify potentially outdated information

## Prerequisites

- **Python 3.11+** (check with `python3 --version`)
- **git** (for cloning the repository)
- **venv** module (usually included with Python, or install via `python3-venv` package)

On Ubuntu/Debian:
```bash
sudo apt install python3.11 python3.11-venv git
```

On macOS (with Homebrew):
```bash
brew install python@3.11 git
```

## Installation

### From PyPI

```bash
pip install opencode-semantic-memory
```

### Quick Install Script

Works on both Linux and macOS, sets up the daemon as a background service:

```bash
curl -fsSL https://gitlab.com/ghavenga/opencode-memory/-/raw/master/scripts/setup.sh | bash
```

This will:
- Clone the repository to `~/.local/share/opencode-memory-install`
- Create a virtual environment and install dependencies
- Download the embedding model (~90MB, one-time, runs locally)
- Set up a background service:
  - **Linux**: systemd user service
  - **macOS**: launchd LaunchAgent
- Configure OpenCode integration
- Create `~/.config/opencode/AGENTS.md` with memory bootstrap
- Optionally bootstrap core directives (requires confirmation due to LLM costs)

### From Source

```bash
git clone https://gitlab.com/ghavenga/opencode-memory.git
cd opencode-memory
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

# Download the embedding model (required, ~90MB, runs locally)
python -c "from sentence_transformers import SentenceTransformer; SentenceTransformer('all-MiniLM-L6-v2')"
```

**Important:** After installing from source, you must manually set up the service (see Quick Start below). The service files in the repo assume the quick install path - you'll need to update the paths to match your clone location.

### Cost Information

**Default background processing is free** - The memory daemon uses:
- Local embedding model (sentence-transformers, no API calls)
- Pattern-based extraction (regex, no LLM)
- Session summaries (stores conversations as-is, no LLM processing)

**LLM-based knowledge extraction is OFF by default** - There is an optional
feature that uses `opencode run` to analyze old conversations and extract
procedures, decisions, and facts. This:
- Runs every 6 hours, processing up to 50 conversations per cycle
- Makes LLM API calls for each conversation (significant cost potential)
- Is **disabled by default** to avoid unexpected charges

To enable LLM extraction (if you want it), add to `~/.config/opencode-memory/config.toml`:
```toml
[ingestion]
llm_extraction = true
```

## Quick Start

### 1. Start the Memory Services

The recommended setup uses 3 background services (see [ARCHITECTURE.md](ARCHITECTURE.md)):

**Using the setup script (recommended):**
```bash
# The setup script handles everything including service installation
curl -fsSL https://gitlab.com/ghavenga/opencode-memory/-/raw/master/scripts/setup.sh | bash
```

**Manual start (development/testing):**
```bash
source .venv/bin/activate

# Terminal 1: HTTP/MCP server
python -m opencode_memory.http_server

# Terminal 2: File watcher daemon
python -m opencode_memory.daemon

# Terminal 3: Background worker
python -m opencode_memory.background_worker
```

**Install services manually (Linux systemd):**
```bash
mkdir -p ~/.config/systemd/user
INSTALL_DIR="$PWD"  # Run from your clone directory

# Copy service files
for svc in opencode-memory opencode-memory-daemon opencode-memory-worker; do
    sed "s|%h|$HOME|g" "$INSTALL_DIR/${svc}.service" | \
    sed "s|\.local/share/opencode-memory-install|${INSTALL_DIR#$HOME/}|g" \
    > ~/.config/systemd/user/${svc}.service
done

# Enable and start
systemctl --user daemon-reload
systemctl --user enable --now opencode-memory opencode-memory-daemon opencode-memory-worker

# Check status
systemctl --user status opencode-memory*
```

**Install services manually (macOS launchd):**

See the setup script for plist templates, or run the setup script which handles this automatically.

### 2. Configure OpenCode

Add to `~/.config/opencode/opencode.json`:

```json
{
  "mcp": {
    "memory": {
      "type": "remote",
      "url": "http://localhost:9824/mcp"
    }
  }
}
```

### 3. Configure the Agent

Add to `~/.config/opencode/AGENTS.md`:

```markdown
# Agent Instructions

At session start, always call `memory_get_boot_context` to load prior context, active sessions, and blockers.

## Memory System

The memory system provides persistent context across sessions.

### Proactive Usage

Call `memory_remember` when:
- You make a decision about approach
- You hit a blocker
- You learn how to do something
- You discover an interesting fact

Call `memory_recall(query)` before:
- Using an unfamiliar API
- Making assumptions about how something works

Call `memory_get_context(entity_ref)` before working on any MR/issue/epic.

Categories: decision, blocker, procedure, fact, event, directive
```

## Configuration

Create `~/.config/opencode-memory/config.toml`:

```toml
[identity]
user = "your-gitlab-username"  # Optional, auto-detected from git
instance = "gitlab.com"

[boot]
identity = true
active_sessions = true
hot_items = true
unresolved_blockers = true
recent_decisions = false
max_hot_items = 5

[ingestion]
watch_paths = ["~/.local/share/opencode/opencode.db"]
db_poll_interval = 30
llm_extraction = false
working_directory = "/path/to/your/projects"

[storage]
path = "~/.local/share/opencode-memory"
```

## MCP Tools

> **Note:** Tools are registered with the `memory_` prefix when accessed through the "memory" MCP server.
> For example, `recall` becomes `memory_recall` in OpenCode sessions.

### Core Memory Tools

| Tool | Description |
|------|-------------|
| `memory_recall(query, limit?, project?, category?, compact?)` | Semantic search across memories |
| `memory_remember(content, category, what?, why?, learned?, entities?, project?)` | Store a memory |
| `memory_get_context(entity_ref)` | Get all memories for !MR, #issue, or &epic |
| `memory_get_boot_context()` | Load startup context (identity, blockers, directives) |
| `memory_get_linked_memories(memory_id, link_types?)` | Get memories linked to a specific memory |

### Session Coordination

| Tool | Description |
|------|-------------|
| `memory_session_start(session_id, working_on?)` | Register session |
| `memory_session_end(session_id, summary?)` | End session with summary |
| `memory_session_heartbeat(session_id)` | Keep session alive |
| `memory_get_active_sessions()` | List active sessions |
| `memory_claim_item(session_id, item_ref)` | Claim exclusive ownership |
| `memory_release_item(session_id, item_ref)` | Release claimed item |

### Memory Management

| Tool | Description |
|------|-------------|
| `memory_resolve_blocker(memory_id)` | Mark blocker as resolved |
| `memory_unresolve_blocker(memory_id)` | Reopen a blocker |
| `memory_archive_memory(memory_id, reason?)` | Archive outdated memory (soft delete) |
| `memory_delete_memory(memory_id, also_delete_vector?)` | Permanently delete a memory |
| `memory_edit_memory(memory_id, content?, what?, why?, learned?)` | Edit memory content or metadata |
| `memory_bulk_archive(memory_ids?, category?, older_than_days?, reason)` | Archive multiple memories |
| `memory_consolidate_memory(days_stale?, project?)` | Find stale/duplicate memories |
| `memory_search_history(query, category?, limit?)` | Search with category filter |

### Backup & Transfer

| Tool | Description |
|------|-------------|
| `memory_export_memories(output_path?, project?, categories?, since_days?)` | Export to JSON |
| `memory_import_memories(input_path, dry_run?, skip_duplicates?)` | Import from JSON |

### Utilities

| Tool | Description |
|------|-------------|
| `memory_ingest_file(file_path)` | Manually ingest a markdown file |
| `memory_enrich_entity(entity_ref, project?)` | Fetch GitLab metadata |
| `memory_bootstrap_memory(path?)` | Scan project files for initial facts |
| `memory_log_session(summary, learnings?, entities?)` | Log session summary |
| `memory_memory_status()` | Check system health and queue status |

## Memory Categories

| Category | Use For |
|----------|---------|
| `boot_gate` | STOP trigger shown every session (≤400 chars, points to directive) |
| `directive` | Full standing instructions (loaded via recall) |
| `procedure` | How-to knowledge, workflows |
| `decision` | Architectural choices, design decisions |
| `blocker` | Obstacles preventing progress |
| `fact` | Project-specific information |
| `plan` | Sequence of steps with end state |
| `goal` | Sustained target/threshold (ongoing, no end state) |
| `idea` | Future possibilities, deferred considerations |
| `event` | Significant occurrences |
| `conversation` | Full conversation content (auto-generated) |

## Storage

All data is stored locally:

- **SQLite**: `~/.local/share/opencode-memory/memory.db` - Memories, entities, sessions, FTS index
- **LanceDB**: `~/.local/share/opencode-memory/vectors/` - Vector embeddings for semantic search

The daemon automatically:
- Cleans up old resolved blockers (>90 days)
- Archives old conversations (>180 days)
- Compacts LanceDB versions (keeps last 10)

## Environment Variables

| Variable | Description |
|----------|-------------|
| `GITLAB_TOKEN` | Enable GitLab entity enrichment |
| `HF_HUB_OFFLINE=1` | Prevent model downloads (use cached) |
| `TRANSFORMERS_OFFLINE=1` | Prevent model downloads |
| `OPENCODE_MEMORY_HOST` | HTTP server bind address (default: 127.0.0.1) |
| `OPENCODE_MEMORY_PORT` | HTTP server port (default: 9824) |
| `OPENCODE_MEMORY_API_KEY` | Optional API key for authentication |
| `OPENCODE_MEMORY_RATE_LIMIT` | Requests per minute per client (default: 60) |

## Development

```bash
# Activate environment
source .venv/bin/activate

# Run tests
python -m pytest tests/ -v

# Lint
ruff check src/

# Type check
mypy src/

# Check memory stats
python -m opencode_memory.cli stats
```

## Architecture

opencode-memory uses a **3-process architecture** to separate concerns:

```
┌─────────────────────────────────────────────────────────────────────────────┐
│                           opencode-memory                                   │
├─────────────────────────────────────────────────────────────────────────────┤
│  ┌─────────────────────┐  ┌─────────────────────┐  ┌─────────────────────┐  │
│  │   HTTP/MCP Server   │  │       Daemon        │  │  Background Worker  │  │
│  │  (opencode-memory)  │  │ (opencode-memory-   │  │ (opencode-memory-   │  │
│  │                     │  │       daemon)       │  │       worker)       │  │
│  ├─────────────────────┤  ├─────────────────────┤  ├─────────────────────┤  │
│  │ • MCP tool handlers │  │ • File watching     │  │ • Embedding compute │  │
│  │ • Request/response  │  │ • OpenCode DB poll  │  │ • GitLab enrichment │  │
│  │ • Health endpoints  │  │ • Queue work items  │  │ • Memory linking    │  │
│  │                     │  │                     │  │ • Cleanup/archival  │  │
│  │                     │  │                     │  │ • LLM extraction    │  │
│  └─────────┬───────────┘  └─────────┬───────────┘  └─────────┬───────────┘  │
│            │                        │                        │              │
│            └────────────────────────┼────────────────────────┘              │
│                          ┌──────────▼──────────┐                            │
│                          │   Shared Storage    │                            │
│                          ├─────────────────────┤                            │
│                          │ • SQLite + FTS5     │                            │
│                          │ • LanceDB (vectors) │                            │
│                          │ • Work queues       │                            │
│                          └─────────────────────┘                            │
└─────────────────────────────────────────────────────────────────────────────┘
```

| Process | Purpose | Module |
|---------|---------|--------|
| **HTTP Server** | MCP tools, health endpoints | `opencode_memory.http_server` |
| **Daemon** | File watching, DB polling | `opencode_memory.daemon` |
| **Worker** | All heavy background processing | `opencode_memory.background_worker` |

This separation ensures:
- API requests are never blocked by background processing
- Worker can crash/restart without affecting the API
- Multiple workers can run in parallel for scaling

See [ARCHITECTURE.md](ARCHITECTURE.md) for detailed documentation.

## Operations

### Log Rotation

When running as a systemd service, logs are managed by journald. To configure log rotation:

```bash
# Check current log size
journalctl --user -u opencode-memory --disk-usage

# Set max journal size (add to ~/.config/systemd/user.conf or override)
# Or create ~/.config/systemd/journald.conf.d/opencode-memory.conf:
cat > ~/.config/systemd/journald.conf.d/opencode-memory.conf << 'EOF'
[Journal]
SystemMaxUse=100M
MaxRetentionSec=7d
EOF

# Or manually vacuum old logs
journalctl --user --vacuum-time=7d
journalctl --user --vacuum-size=100M
```

For file-based logging, configure rotation in the systemd service:

```ini
[Service]
StandardOutput=append:/var/log/opencode-memory/server.log
StandardError=append:/var/log/opencode-memory/error.log
```

Then use logrotate (`/etc/logrotate.d/opencode-memory`):

```
/var/log/opencode-memory/*.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    create 0640 $USER $USER
}
```

### Monitoring

The HTTP server exposes several endpoints:

| Endpoint | Description |
|----------|-------------|
| `/health` | Basic health check (returns 200 if healthy, 503 if degraded) |
| `/stats` | Detailed statistics (memories, cache, queue, links) |
| `/metrics` | Prometheus-format metrics |

Example monitoring with curl:
```bash
# Health check (for load balancers/monitoring)
curl -s http://localhost:9824/health | jq .

# Full stats
curl -s http://localhost:9824/stats | jq .

# Prometheus metrics
curl -s http://localhost:9824/metrics
```

### CLI Tools

```bash
# Show comprehensive statistics
python -m opencode_memory.cli stats

# Ingest markdown files
python -m opencode_memory.cli ingest /path/to/notes --recursive

# Archive old memories
python -m opencode_memory.cli cleanup --dry-run

# Enrich entities with GitLab metadata
python -m opencode_memory.cli enrich --limit 100
```

## Troubleshooting

### Check service status

**Linux (systemd):**
```bash
# Check all services
systemctl --user status opencode-memory*

# View logs for specific service
journalctl --user -u opencode-memory -f         # Server logs
journalctl --user -u opencode-memory-daemon -f  # Daemon logs
journalctl --user -u opencode-memory-worker -f  # Worker logs

# Restart all services
systemctl --user restart opencode-memory opencode-memory-daemon opencode-memory-worker
```

**macOS (launchd):**
```bash
# Check all services
launchctl list | grep opencode

# View logs
tail -f ~/.local/state/opencode-memory/server.log
tail -f ~/.local/state/opencode-memory/daemon.log
tail -f ~/.local/state/opencode-memory/worker.log

# Restart all services
for svc in com.opencode.memory com.opencode.memory.daemon com.opencode.memory.worker; do
    launchctl unload ~/Library/LaunchAgents/${svc}.plist 2>/dev/null
    launchctl load ~/Library/LaunchAgents/${svc}.plist
done
```

### Check memory system health
Use `memory_status` tool or:
```bash
# HTTP health endpoint
curl http://localhost:9824/health

# Detailed stats
curl http://localhost:9824/stats
```

### Prometheus metrics
The HTTP server exposes metrics at `/metrics`:
```bash
curl http://localhost:9824/metrics
```

### Backup and restore
```bash
# Export all memories
memory_export_memories(output_path="/backup/memories.json")

# Import on new machine
memory_import_memories(input_path="/backup/memories.json", dry_run=True)  # Preview
memory_import_memories(input_path="/backup/memories.json")  # Actually import
```

### Reset memory database

**Linux:**
```bash
systemctl --user stop opencode-memory opencode-memory-daemon opencode-memory-worker
rm -rf ~/.local/share/opencode-memory/
systemctl --user start opencode-memory opencode-memory-daemon opencode-memory-worker
```

**macOS:**
```bash
for svc in com.opencode.memory com.opencode.memory.daemon com.opencode.memory.worker; do
    launchctl unload ~/Library/LaunchAgents/${svc}.plist 2>/dev/null
done
rm -rf ~/.local/share/opencode-memory/
for svc in com.opencode.memory com.opencode.memory.daemon com.opencode.memory.worker; do
    launchctl load ~/Library/LaunchAgents/${svc}.plist
done
```

### Upgrade from older versions

If you're upgrading from a version that used `opencode-memory-enrich.service`, the setup script will automatically migrate to the new 3-process architecture. Run the setup script again:

```bash
curl -fsSL https://gitlab.com/ghavenga/opencode-memory/-/raw/master/scripts/setup.sh | bash
```

This will:
1. Stop and disable the old `opencode-memory-enrich` service
2. Install the new `opencode-memory-daemon` and `opencode-memory-worker` services
3. Start all services

## License

MIT
