Metadata-Version: 2.4
Name: dagestan
Version: 1.0.0
Summary: Temporal graph memory layer for large language models
Author-email: Nabin <nabinoli2004@gmail.com>
License: MIT
Project-URL: Homepage, https://github.com/nabin/dagestan
Project-URL: Repository, https://github.com/nabin/dagestan
Project-URL: Issues, https://github.com/nabin/dagestan/issues
Keywords: llm,memory,graph,temporal,knowledge-graph
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: chromadb>=1.5.5
Requires-Dist: datasets>=4.8.4
Requires-Dist: dotenv>=0.9.9
Requires-Dist: google-generativeai>=0.8.6
Requires-Dist: loadenv>=0.1.1
Requires-Dist: pydantic>=2.12.5
Requires-Dist: pytest-asyncio>=1.3.0
Requires-Dist: pyyaml>=6.0.1
Requires-Dist: sentence-transformers>=5.3.0
Provides-Extra: openai
Requires-Dist: openai>=1.0.0; extra == "openai"
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.18.0; extra == "anthropic"
Provides-Extra: all
Requires-Dist: openai>=1.0.0; extra == "all"
Requires-Dist: anthropic>=0.18.0; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0; extra == "dev"
Dynamic: license-file

# Dagestan


**Temporal Graph Memory Layer for LLMs**

Dagestan stores LLM memory as a typed temporal knowledge graph instead of flat vector embeddings. It tracks entities, concepts, events, preferences, and goals — with time-aware confidence decay, contradiction detection, and relationship-based retrieval.

## Why

Current LLM memory solutions (vector DBs) treat memory as a pile of embeddings retrieved by similarity. They don't understand:

- **Time** — old information and new information are treated the same
- **Relationships** — there's no structure between memories
- **Contradictions** — conflicting facts coexist silently
- **Decay** — nothing fades; nothing is curated

Dagestan addresses these gaps with a graph-based approach where memory has structure, relationships, and temporal awareness.

## Status

**v0.1 — Foundation release.** Working memory layer with:

- Typed temporal graph (Entity, Concept, Event, Preference, Goal nodes)
- LLM-based knowledge extraction from conversations
- Contradiction detection between conflicting preferences/goals
- Exponential temporal decay on confidence scores
- Gap detection (incomplete entity profiles)
- Bridge node detection (cross-cluster connections)
- Centrality-based importance scoring
- Query-driven graph retrieval (keyword + structure, no embeddings)
- JSON persistence
- CLI for graph inspection
- 63 passing unit tests

## Install

```bash
# Core (no LLM dependencies)
pip install -e .

# With OpenAI
pip install -e ".[openai]"

# With Anthropic
pip install -e ".[anthropic]"

# Development
pip install -e ".[dev]"
```

## Quick Start

```python
from dagestan import Dagestan

# Initialize with an LLM provider for extraction
mem = Dagestan(provider="openai", db_path="./memory.json")

# Ingest conversation text
mem.ingest("User mentioned they love Python and want to build a startup focused on graph databases.")

# Retrieve relevant context
context = mem.retrieve("What does the user care about?")
print(context)

# Run curation (applies decay, finds contradictions)
report = mem.curate()
print(f"Contradictions found: {report.contradictions_found}")

# Get structured context for next conversation
strategy = mem.strategy()
print(strategy)
```

### Without an LLM (manual graph building)

```python
from dagestan import Dagestan, Node, Edge, NodeType, EdgeType

mem = Dagestan(db_path="./memory.json", auto_save=True)

# Add nodes directly
user = mem.add_node(Node(type=NodeType.ENTITY, label="User"))
pref = mem.add_node(Node(type=NodeType.PREFERENCE, label="Prefers tea"))
goal = mem.add_node(Node(type=NodeType.GOAL, label="Build Dagestan"))

mem.add_edge(Edge(source_id=user.id, target_id=pref.id, type=EdgeType.HAS_PREFERENCE))
mem.add_edge(Edge(source_id=user.id, target_id=goal.id, type=EdgeType.WANTS))

# Query the graph
results = mem.retrieve("user preferences")
print(results)
```

## CLI

```bash
# Show graph summary
dagestan info --db ./memory.json

# List all nodes
dagestan nodes --db ./memory.json

# Filter by type
dagestan nodes --type preference --db ./memory.json

# Query the graph
dagestan retrieve "user goals" --db ./memory.json

# Run curation
dagestan curate --db ./memory.json

# Export full graph
dagestan export --db ./memory.json
```

## Architecture

```
Conversation → Extraction (LLM) → Temporal Graph → Operations → Retrieval
                                       ↓
                                   Curation
                                 (decay, contradictions,
                                  gaps, bridges)
```

**Node Types:** Entity, Concept, Event, Preference, Goal

**Edge Types:** relates_to, caused, contradicts, happened_before, has_preference, wants

**Temporal Metadata:** Every node carries `created_at`, `last_reinforced`, `confidence_score`, and `decay_rate`. Confidence degrades exponentially unless reinforced by new conversation.

## Graph Operations

These run on the graph structure — no LLM needed for computation.

| Operation | What it does |
|-----------|-------------|
| Contradiction Detection | Finds conflicting preferences/goals for the same entity |
| Temporal Decay | Reduces confidence based on time since last reinforcement |
| Gap Detection | Identifies entities with incomplete knowledge profiles |
| Bridge Detection | Finds nodes connecting otherwise disconnected clusters |
| Centrality Scoring | Ranks nodes by connection count + recency |

## Project Structure

```
dagestan/
├── __init__.py              # Main Dagestan class (public API)
├── cli.py                   # Command-line interface
├── graph/
│   ├── schema.py            # Node, Edge, NodeType, EdgeType
│   ├── temporal_graph.py    # Core graph with CRUD + snapshots
│   └── operations.py        # Contradiction, decay, gap, bridge, centrality
├── extraction/
│   ├── extractor.py         # Conversation → graph via LLM
│   └── prompts.py           # Extraction prompt templates
├── curation/
│   ├── curator.py           # Curation pipeline orchestrator
│   └── strategy.py          # Context strategy generation
├── retrieval/
│   └── retriever.py         # Query-driven graph traversal
├── storage/
│   └── store.py             # JSON persistence (SQLite planned for v0.2)
├── integrations/            # Planned: drop-in OpenAI/Anthropic wrappers
viz/
├── __init__.py
├── __main__.py              # Entry point (python -m viz)
├── server.py                # Stdlib HTTP server + REST API + SSE
├── watcher.py               # File-change polling (no deps)
├── export.py                # LaTeX/TikZ, DOT, CSV export
├── generate_demo.py         # Demo graph generator
└── static/
    ├── index.html           # Main UI page
    ├── style.css            # Dark-theme styling
    └── app.js               # vis-network graph rendering
```

## Visualization

Dagestan ships with a **zero-dependency interactive graph visualizer** for debugging and exploring your knowledge graphs in the browser.

```bash
# Launch with auto-detected graph file
python -m viz.server

# Or point at a specific file and port
python -m viz.server --file demo_memory.json --port 8765
```

Then open **http://localhost:8765**.

### Features

| Feature | Description |
|---------|-------------|
| **Live Graph Rendering** | Interactive force-directed layout via vis-network |
| **Auto-Refresh / SSE** | File watcher detects changes and pushes updates via Server-Sent Events |
| **Diff Highlighting** | New nodes glow green with a ✦ marker; removed counts shown in dashboard |
| **Node & Edge Inspector** | Click any element to view full metadata, confidence, decay rate |
| **Neighborhood Explorer** | Double-click a node to zoom into its local neighborhood |
| **Stats Dashboard** | Node/edge counts, type distributions, confidence histograms |
| **Temporal Decay View** | Color-coded confidence levels (green → yellow → red) |
| **Filter by Type** | Toggle node types on/off |
| **Search** | Find nodes by label (keyboard shortcut: `/`) |
| **LaTeX / DOT / CSV Export** | Export graphs for research papers (TikZ, Graphviz DOT, CSV) |
| **File Switcher** | Browse and switch between graph JSON files in the project |
| **Keyboard Shortcuts** | `R` refresh, `F` fit view, `/` search, `?` help, `Esc` clear |

### Generate a Demo Graph

```bash
python -m viz.generate_demo --output viz_demo_graph.json --nodes 25
python -m viz.server --file viz_demo_graph.json
```

See [viz/README.md](viz/README.md) for full details.

## Roadmap

| Version | Focus | Status |
|---------|-------|--------|
| v0.1 | Core graph, extraction, basic operations, JSON storage | **Done** |
| v0.2 | SQLite backend, all 5 operations fully tuned | Planned |
| v0.3 | Query-driven traversal improvements, context compression | Planned |
| v1.0 | Benchmarks, paper, honest evaluation vs baselines | Planned |
| v2.0 | Graph operations as reasoning substrate | Research |

## Evaluation (LOCOMO Benchmarks)

Dagestan includes an evaluation harness for benchmarking against the
[LOCOMO benchmark](https://snap-research.github.io/locomo).

Evaluation code/config lives under `evals/`:

- `evals/run_locomo_eval.py` — main evaluation runner (QA, summarization, coherence)
- `evals/run_ablations.py` — ablation study runner
- `evals/configs/*.yaml` — model/data settings (e.g., `smoke_test.yaml`, `locomo_gemini_flash.yaml`)

Quick commands (from repo root):

```bash
# Install eval-only dependencies (dataset download / caching)
pip install datasets huggingface_hub pyyaml

# Download/cache the LOCOMO dataset locally
make download-locomo

# Smoke run (fast dry-run)
make smoke

# Full benchmark run
make locomo

# Single tasks
make qa
make summarization
make coherence

# Ablations
make ablations
make ablations-baselines
```

Notes:

- The runner writes outputs under `evals/results/<run_id>/`.
- The `make smoke` target uses the `provider: stub` setting from
  `evals/configs/smoke_test.yaml`. If your current Dagestan backend does not
  support `provider="stub"` for extraction/ingestion, run with a real provider
  (e.g., `openai` / `anthropic`) by editing the config.

Performance claims should be made only after you complete the benchmark runs.

## Constraints

- **Zero GPU required** — runs on CPU only
- **Minimal dependencies** — core is stdlib only, LLM client is optional
- **LLM-agnostic** — works with any provider via simple callable interface
- **Storage-agnostic** — JSON now, SQLite/Neo4j planned
- **Honest evaluation** — we measure where it fails, not just where it wins

## License

MIT

