The design follows Cal Paterson's memoryfield specification: memory is a data format, not a pipeline. The canonical data is the pages; the index is a derived cache that can be deleted at any time and rebuilt from the pages.
git pull are searchable immediately.| Choice | Why |
|---|---|
Flat directory of Markdown pages with YAML frontmatter (title, uuid, summary, created, updated) | Agents are already fluent in files and Markdown; the memory can be read with cat, edited in an editor, versioned with git, and carried between models. Filenames are restricted to [a-z0-9-] per the spec; symlinks, sub-directories and editor debris are ignored. |
| One SQLite file per embedding model, named after it | Indexes from different models never mix; the file records its model_code and refuses to serve a different model. Deleting the file loses nothing. |
Exhaustive cosine scan in Python (math.sumprod) instead of sqlite-vec | The spec itself notes a full scan is fine at memory scale, and it measured at 19 ms for 300 pages of 1536 dimensions. The macOS system Python cannot load SQLite extensions, so avoiding sqlite-vec keeps the prototype dependency-free and portable. |
| Incremental sync keyed by sha256 | Only new or edited pages are re-embedded. The 300-page corpus embedded once in 64 s for about one cent; subsequent syncs are a few milliseconds. |
Embedding text-embedding-3-small through the framework's existing Model.get_embedding | No new client code; any catalogued embedding model (Gemini, BGE via Together) works by name. An offline feature-hashing embedder is included as a fallback and as a baseline. |
| Embed the title, summary and body, capped at 8 KB | Title and summary act as "key expansion" for retrieval (the same trick LongMemEval found helpful); the cap matches the spec and the model's 8192-token input. (The prototype embedded the whole file including frontmatter; the volatile uuid and timestamps were later excluded because they made vectors nondeterministic and added noise.) |
from kiss.core.memoryfield import MEMORY_PROTOCOL, MemoryTools
from kiss.core.kiss_agent import KISSAgent
tools = MemoryTools("~/.kiss/memories") # default embedder: text-embedding-3-small
agent = KISSAgent("assistant")
agent.run(
model_name="claude-fable-5-1",
prompt_template="Set up the postgres channel agent for the staging database.",
system_prompt=MEMORY_PROTOCOL + "\n" + your_system_prompt,
tools=[*tools.tools(), *other_tools],
)MEMORY_PROTOCOL is a nine-line instruction block: search memory before starting, record durable lessons (one topic per page, with sources) while working, never store secrets, delete pages that turn out wrong. In the live test, a Haiku agent given only these tools stored a fact in one session and a fresh agent recalled it by paraphrased question in the next.
The question a memory has to answer is: given something the user asks later, does search surface the page that holds the answer? To test that on real material rather than toy pages, the evaluation script (kiss.core.memoryfield.evaluate) turns finished tasks from ~/.kiss/sorcar.db into memory pages, one per task, containing the request and the final result as plain text, and then fires probe questions at the index.
| Corpus | The 300 most recent successful top-level tasks (results of at least 300 characters; failed, interrupted and stopped tasks excluded). Each page is capped at 8 KB. 64 of the 300 pages are repeats of another task in the corpus ("run all tests", "update the README", "run update_models.py", the same reviewer prompt on different papers). |
|---|---|
| Probes | 55 questions, each with exactly one gold page. 15 were written by hand to be phrased differently from the original request ("how do I put the car through an emissions readiness drive cycle" for the BMW 330i task). 40 were written by claude-fable-5-1, which was shown a sampled page and asked for the question a user might ask weeks later, without copying distinctive phrases or identifiers. |
| Retrievers | vector: the memoryfield index with text-embedding-3-small. BM25: SQLite FTS5 with the Porter stemmer, query words OR-ed (the "just grep it" baseline). hashed: the same index with the offline feature-hashing embedder. hybrid: reciprocal-rank fusion of vector and BM25. |
| Metrics | Recall@k: the gold page is within the top k. MRR: the mean of 1/rank of the gold page. Family recall additionally counts a hit when a page whose request text is identical (after case and whitespace normalisation) to the gold page's request is in the top k; that is what a curated memory would contain after merging repeated tasks into one page per topic. |
| Retriever | R@1 | R@3 | R@5 | MRR | Family R@1 | Family R@3 | Family R@5 | Mean latency |
|---|---|---|---|---|---|---|---|---|
| vector (text-embedding-3-small) | 0.62 | 0.85 | 0.91 | 0.74 | 0.75 | 0.91 | 0.95 | 223 ms |
| hybrid (vector + BM25, RRF) | 0.69 | 0.85 | 0.87 | 0.77 | 0.78 | 0.93 | 0.93 | 224 ms |
| BM25 (SQLite FTS5) | 0.58 | 0.76 | 0.84 | 0.68 | 0.62 | 0.80 | 0.89 | 1 ms |
| hashed bag-of-words (offline) | 0.33 | 0.53 | 0.67 | 0.44 | 0.36 | 0.58 | 0.75 | 13 ms |
| Probes | Retriever | R@1 | R@3 | R@5 | MRR |
|---|---|---|---|---|---|
| 15 hand-written | vector | 0.80 | 1.00 | 1.00 | 0.90 |
| BM25 | 0.73 | 1.00 | 1.00 | 0.86 | |
| hybrid | 0.87 | 1.00 | 1.00 | 0.93 | |
| 40 LLM-paraphrased | vector | 0.55 | 0.80 | 0.88 | 0.68 |
| BM25 | 0.53 | 0.68 | 0.78 | 0.61 | |
| hybrid | 0.62 | 0.80 | 0.82 | 0.71 |
The LLM-written probes are harder on purpose: they ask about a detail of the outcome ("were there actually any containers running to stop?", "which failures were test bugs versus real code bugs?") rather than restating the request, and they were sampled uniformly, so they land on repeated tasks in proportion to how common repeats are in the corpus.
| Question asked | Page it had to find (original request) | vector | BM25 | hybrid |
|---|---|---|---|---|
| which memory architecture did we conclude is best for an AI agent | what is the simplest and most powerful memory system that I can use wi | 1 | 1 | 1 |
| how do I put the car through an emissions readiness drive cycle | how do I complete drive cycle on BMW 2017 330i in the easiest way? | 1 | 1 | 1 |
| nonstick cookware brand comparison for safety and durability | Compare SENSARTE with Carote, GreenPan, and Caraway on safety certific | 1 | 2 | 1 |
| is that Chinese cookware brand safe and where is it made | is SENSARTE an american brand? Where do they manufacture? Are they s | 1 | 1 | 1 |
| how to authenticate the google docs agent | authenticate gdocs | 1 | 1 | 1 |
| implement the whatsapp channel agent like the other messaging agents | can you implement ./src/kiss/agents/third_party_agents/whatsapp_agent. | 1 | 1 | 1 |
| add a green border around the settings panel | can you add a green border (using a theme color) around the settings p | 1 | 1 | 1 |
| fix race conditions, hangs and deadlocks across the codebase | Can you precisely and thoroughly find and fix all race conditions, han | 1 | 1 | 1 |
| why did the previous run stop responding and hang the web app | Why did the last task fail? It stopped responding and the remote weba | 2 | 1 | 2 |
| show images from tool results inline in the event panel | when an image is generated by you or is the result of a tool call, can | 1 | 2 | 1 |
| publish website changes to the github pages repo | Push the updated website content to the kisssorcar.github.io GitHub Pa | 1 | 1 | 1 |
| task classifier should use structured output and not be agentic | can you make the KISSAgent("Task Classifier") non-agentic and use stru | 2 | 1 | 1 |
| merge conflict help | Can you check the following message for a merge conflict and help me f | 1 | 3 | 1 |
| rename SEA methods to drop the get_ prefix | can you remove the get_ prefix from all methods in SEAs? also add the | 1 | 1 | 1 |
| MCP servers for brave search, notion, postgres, firecrawl and google workspace | Implement the brave-search, notion, postgres, firecrawl, github, and G | 2 | 2 | 2 |
Of the 21 probes where the vector retriever did not put the gold page first, seven had a page from the same repeated task on top (for example "What changed the last time we refreshed the model catalog?" matched a different run of update_models.py, and both "run all tests" probes matched a sibling run). No retriever can separate those from the question alone; the fix belongs in the memory, not the index: an agent following the protocol updates one "model catalog refresh" page instead of leaving four run logs. Three probes were missed by every retriever (no family page in the top 5): one about a git branch, phrased very differently from the request ("which older commit did I base that clean fallback branch on"), one asking what was outdated in a README sync when several README pages compete, and one asking about a code review's conclusion in vocabulary the review itself did not use. Keyword search ranked first in a few cases where the embedding did not (the install.sh walkthrough: rank 1 by BM25, rank 5 by vector), which is why the hybrid edges ahead at rank 1.
After the implementation and tests were complete, a second model (gpt-5.6-sol) did a strictly read-only review of the five source files and four test files, with instructions to report only verified problems. It reproduced 15 defects, none of which the 44 tests at that point had caught. All were fixed and each now has a regression test; the suite is 51 tests.
| Severity | Finding | Fix |
|---|---|---|
| High | with sqlite3.connect(...) as conn commits but does not close; every search leaked a file descriptor until garbage collection. | All connections wrapped in contextlib.closing; tests run with ResourceWarning as an error. |
| High | Opening the index ran an INSERT OR IGNORE, so even a search held a write transaction; sync() then held that lock across every embedding API call. | Schema setup commits immediately; sync() computes embeddings with no transaction open and writes all rows in one short transaction; 30 s busy timeout. |
| High | If the index filename was a symlink, sync wrote tables into whatever database it pointed at. | A symlinked index path is refused. |
| High | The evaluation reused a stale page directory when --limit or the database changed, and then crashed on a missing gold page. | The corpus is reconciled exactly on every run: missing pages added, extra pages removed, existing pages untouched. |
| Medium | Two model names that sanitise to the same filename (provider/a:b, provider/a/b) silently shared one index. | The stored meta.model_code is checked on every open and a mismatch raises. |
| Medium | A symlinked page inside the directory was listed, and deleting the alias deleted its target. | Symlinked entries are neither listed nor readable, writable or deletable. |
| Medium | memory_pull only enforced its 24,000-character cap from the second hit on; one large page produced a 60 KB tool result. | An oversized first hit is truncated with a pointer to memory_read. |
| Medium | Sync decoded invalid UTF-8 with replacement but reading used strict decoding, so such a page appeared in search and then crashed memory_pull. | One tolerant decoder is used everywhere. |
| Medium | A page edited while its embedding was being computed was stored with the old vector and the old hash. | The file is re-hashed after embedding; if it changed it is left for the next sync. |
| Medium | The cached LLM probes were reused regardless of probe count, seed, model or corpus. | The cache records those four values and is regenerated on any mismatch. |
| Medium | Task families were derived from the 48-character filename slug, so unrelated long requests with a shared prefix counted as duplicates. | Families come from the full normalised request text (64 repeated pages, down from 100 under the slug rule). |
| Low | Hybrid latency included the unrelated hashed search and excluded fusion. | Timed as vector + BM25 + fusion. |
| Low | Top-level tasks with NULL parent_task_id were skipped. | Uses Sorcar's own predicate, IS NULL OR = ''. |
| Low | Frontmatter supplied inside a page body could overwrite the stored uuid and created on update. | Identity keys are protected on update (still honoured on create, for imports). |
| Low | The read-only database URI broke on paths containing ? or #. | Built with Path.as_uri(). |
The review also confirmed what was right: every exported name exists, the JSON schemas generated for the six tools have correct required fields and integer typing for k, plain ../ traversal is rejected, and the sha256-based add/update/remove logic behaves as documented.
task_history columns; four live tests use text-embedding-3-small and claude-haiku-4-5, including a two-session agent test (store a fact, recall it by paraphrase in a fresh agent).uv run check --full (ruff, mypy, pyright) passes.uuid in frontmatter and query words give unrelated pages a tiny non-zero score, so exact-list assertions were replaced by rank assertions.The evaluation says the retrieval layer is good enough to build on: for a question that a user would plausibly ask about past work, the right page is in the top three about five times out of six, and in the top five 19 times out of 20 once repeated tasks are merged. It does not measure whether an agent writes good pages; the corpus here was generated mechanically from task logs, which are longer and noisier than the distilled notes the protocol asks for. Better pages should make retrieval easier, not harder.
Two limits are worth stating plainly. First, the corpus is 300 pages; the exhaustive scan scales linearly, so at tens of thousands of pages the 19 ms becomes a second and an approximate index (sqlite-vec's vec0, or pgvector) would be the next step. Second, search quality is bounded by the embedding model; the offline hashed embedder is a fallback for machines without an API key, not a substitute (R@3 0.53 versus 0.85).
uv run python -m kiss.core.memoryfield.evaluate --limit 300 --llm-probes 40
# offline variant, no API keys needed:
uv run python -m kiss.core.memoryfield.evaluate --limit 300 --llm-probes 0 --embedding-model hashed-bow-v1
uv run pytest src/kiss/tests/core/memoryfield -q