Metadata-Version: 2.4
Name: deepcq
Version: 0.2.1
Summary: Deep Context Query — intent-aware memory querying for LLM agents
License: MIT
Project-URL: Homepage, https://github.com/ternary-ai/deepcq
Project-URL: Repository, https://github.com/ternary-ai/deepcq
Project-URL: Issues, https://github.com/ternary-ai/deepcq/issues
Keywords: llm,agents,memory,context,rag,prompt-engineering
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Dynamic: license-file

# deepcq

**Deep Context Query** — intent-aware memory querying for LLM agents.

deepcq sits between an agent's memory and its prompt. Instead of stuffing an
entire memory blob into every LLM call, it scores which sections of memory
are actually relevant to the current query and returns only those — cutting
prompt size while keeping recall.

No embeddings, no vector store, no external dependencies. Relevance is
computed from the query's own tokens against text already in memory, using
plain regex and a lightweight stemmer.

## Install

```bash
pip install deepcq
```

For local development from a clone:

```bash
cd deepcq
pip install -e ".[dev]"
```

## Core idea

You declare a `MemorySchema`: a list of section names such as
`["status", "errors", "metrics"]`. deepcq doesn't need to know what those
sections mean — it derives what's relevant from the words in each query
itself ("query-term-first"), rather than from a hand-built keyword table.

```python
from deepcq import DeepCQ, MemorySchema

schema = MemorySchema(sections=["status", "errors", "metrics"])
cq = DeepCQ(schema=schema)

cq.state.set("errors",  "[errors]\n- OOM at 02:15\n")
cq.state.set("metrics", "[metrics]\nCPU: 78%\nMemory: 4.2GB\n")
cq.state.set("status",  "[status]\nAll services running\n")

result = cq.query("any memory or cpu spikes?")
print(result.content)                       # only the metrics section
print(result.sections_included)             # ["metrics"]
print(f"{result.metrics.reduction_pct:.1f}% smaller")
```

### `locate()` — finding relevant chunks in a text blob

For the common case of "which part of this document answers this query"
(no schema, no memory tiers, just text), `locate()` wraps `DeepCQ` into a
one-shot convenience function:

```python
from deepcq import locate

doc = (
    "The authentication service validates tokens on every request.\n\n"
    "Rate limiting is enforced per API key using a sliding window counter.\n\n"
    "The database migration tool applies schema changes in order."
)

for candidate in locate(doc, "how does rate limiting work?"):
    print(candidate.index, candidate.score, candidate.text)
```

`locate()` splits `text` on `chunk_on` (default `"\n\n"`), scores each chunk
against `query`, and returns `Candidate(index, text, score)` objects for
chunks at or above `confidence_threshold`, ordered by position in the
original text. `text` is returned unmodified — no headers, no wrapping — so
it's safe to use directly for verbatim matching against the source document.
Pass `max_candidates` to cap the result to the highest-scoring chunks.

## How it works

### 1. Five memory tiers

`DeepCQ` manages five tiers, each suited to a different memory lifetime:

| Tier | Class | Lifetime | Purpose |
|------|-------|----------|---------|
| T1 | `AppState` | live | Current state of the running application (sections keyed by name) |
| T2 | `SessionCache` | per-session, bounded | Ephemeral key/value scratch space, deduped, LRU-evicted |
| T3 | `History` | sliding window | Last N conversation turns |
| T4 | `LongTermStore` | persistent | Append-only notes, optionally file-backed (JSON) |
| T5 | `ReferenceData` | persistent, lazy | Static config/reference data, optionally file-backed |

Each tier renders itself to text via `as_text()`. `DeepCQ.query()` pulls text
from all five tiers before scoring.

### 2. Tokenizing the query (`tokens.py`)

The query is lowercased, split into words, stripped of stopwords, and each
surviving token is paired with a suffix-stripped stem (`spikes` → `spike`,
`running` → `run`). No external NLP dependency — it's a small suffix-rule
table plus a doubled-consonant cleanup (`running` → `runn` → `run`).

### 3. Scoring sections (`intent.py`)

For each section in the schema, deepcq builds a vocabulary from:
1. The section name itself (tokenized/stemmed).
2. Optional `keywords` you declare for that section in the schema (useful
   when a section name is jargon, e.g. `"db"` → `["database", "postgres"]`).
3. Whatever text is currently stored under that section's header across all
   tiers.

The section's score is the fraction of unique query tokens found in that
vocabulary. Sections scoring at or above `schema.confidence_threshold`
(default `0.3`) are "included".

### 4. Extracting content (`schema.py`, `regex_gen.py`)

Sections are delimited in stored text by a header, `"[{name}]"` by default
(configurable via `section_header_format`, e.g. `"## {name}"`). Each section
compiles to a regex that captures from its header up to the next header or
end of string — this is how `MemorySchema.extraction_pattern(section)` pulls
a section's block out of raw tier text.

For the two free-text tiers (`cache`, `history`, which aren't
section-delimited), deepcq instead builds a single regex out of all query
tokens and includes that tier's text only if any token matches
(`build_content_pattern` in `regex_gen.py`).

### 5. Result (`result.py`)

`query()` returns a `QueryResult`:

```python
@dataclass
class QueryResult:
    content: str                      # the filtered, concatenated text
    intent: dict[str, float]          # per-section relevance scores
    sections_included: list[str]
    sections_excluded: list[str]
    content_pattern: str              # the regex used against cache/history
    metrics: OptimizationMetrics      # original_chars, filtered_chars, reduction_pct
```

### Fallback safety

If no section clears the confidence threshold and
`schema.fallback_include_all` is `True` (the default), `query()` returns the
full unfiltered text instead of nothing — this prevents a vague query from
silently discarding memory the agent might actually need.

## Package layout

```
deepcq/
├── pyproject.toml
├── README.md
├── deepcq/                      # the installable package
│   ├── __init__.py              # public API: DeepCQ, MemorySchema, QueryResult, ...
│   ├── core.py                  # DeepCQ orchestrator (query())
│   ├── schema.py                # MemorySchema + extraction regex compilation
│   ├── intent.py                # score_sections()
│   ├── tokens.py                # tokenize() / stem()
│   ├── regex_gen.py              # query-token regex builder for free-text tiers
│   ├── result.py                # QueryResult / OptimizationMetrics dataclasses
│   ├── memory/
│   │   ├── state.py             # T1 AppState
│   │   ├── session.py           # T2 SessionCache, T3 History
│   │   └── store.py             # T4 LongTermStore, T5 ReferenceData
│   └── integrations/
│       └── ternary/             # optional adapter for ternary-ai agents/prompts
│           ├── adapters.py      # ToolContextProvider, LangChainMemoryAdapter
│           └── prompt_library.py # PromptLibrary (loads ternary-ai/prompts CSVs)
├── examples/
│   ├── openai_agents_sdk.py
│   ├── claude_agents_sdk.py
│   ├── qwen_agents_sdk.py
│   └── google_a2a.py
└── tests/
```

## Optional integrations

`deepcq.integrations.ternary` is an optional adapter layer for using deepcq
as a memory/context backend in agent frameworks. It has no hard dependency
on any framework — both adapters duck-type the interface they mimic:

- **`ToolContextProvider`** — formats a `QueryResult` as a plain string to
  inject into a system prompt or tool call.
- **`LangChainMemoryAdapter`** — duck-types LangChain's `BaseChatMemory`
  (`save_context` / `load_memory_variables` / `clear`), so it can be passed
  directly as a chain's `memory=` argument without installing LangChain.
- **`PromptLibrary`** — loads the CSV prompt templates from the
  [ternary-ai/prompts](https://github.com/ternary-ai/prompts) repo (locally
  or via `urllib` from raw GitHub URLs) into a deepcq schema, so prompt
  templates themselves can be retrieved by intent.

```python
from deepcq import DeepCQ, MemorySchema
from deepcq.integrations.ternary import ToolContextProvider, PromptLibrary

cq = DeepCQ(schema=MemorySchema(sections=["research", "risk-management"]))
provider = ToolContextProvider(cq)
context = provider.get_context("what is my portfolio concentration risk?")
```

See `examples/openai_agents_sdk.py`, `examples/claude_agents_sdk.py`,
`examples/qwen_agents_sdk.py`, and `examples/google_a2a.py` for focused
integration examples. They are designed as SDK wiring references, so you
may need to adjust the exact import path to match the SDK version you have
installed.

## Testing

```bash
pip install -e ".[dev]"
pytest
```

## Moving this into its own repository

This directory is self-contained — nothing outside `deepcq/` is required.
To split it out:

```bash
cp -r deepcq/ /path/to/new/deepcq-repo
cd /path/to/new/deepcq-repo
git init
pip install -e ".[dev]"
pytest
```
