Metadata-Version: 2.4
Name: curldb
Version: 0.1.0
Summary: HTTP exchange datastore: raw requests and responses in, structured queries out. One SQLite file per session, zero daemon, zero dependencies.
Author-email: Ranger Chen <ranger@rangerfoxsolutions.com>
License-Expression: MIT
Project-URL: Repository, https://github.com/rangersui/curldb
Keywords: http,sqlite,archive,llm,front-matter,fts5
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Database
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Text Processing :: Indexing
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# curldb

HTTP exchange datastore. One SQLite file per session; the requests and responses of an AI conversation stored as-is, queried by envelope fields.

curl is the reference client, hence the name: every operation the server accepts is one curl line, and the stored record is byte for byte what curl sent.

[中文说明在后面](#中文)

## What it does

Every message in an LLM conversation is an HTTP envelope: what you type is a request, what the model answers is a response, a tool call is a request plus a response. curldb stores each envelope verbatim in SQLite, parses status / method / path / headers into an index, and full-text indexes the body.

```
what you type ---------> curldb wrap 'POST /chat' -H X-Topic:x ---+
model reply (HTTP/1.1 200 OK ...) --------------------------------+--> curldb add --> session.sqlite
tool call (POST /tool/Read ... / HTTP/1.1 200 OK ...) ------------+
                                                                        |
                                         curldb query 'status=409 header:X-Scope=design'
                                         curldb tags
```

raw is the source of truth. Every index is parsed from raw and can be rebuilt.

## Three principles

**One session, one file.** The file name is the session identity: `curldb --db 2026-09-11-design.sqlite` or `CURLDB_PATH`, default `curldb.sqlite` in the current directory. Searching across sessions is querying each file in turn.

**Headers are tags, not a schema.** Names and values need no agreement up front. `X-Verdict: shaky` is unreadable to a conventional program and readable to an LLM. The program stores, counts and lists; `curldb tags` prints every header name and value this file has seen, so you look first, then query. The vocabulary grows out of the data.

**The body is never touched.** Whoever adds headers (you, the model, an annotating model, a hook) adds headers only. Annotators sign with `Via:` so records tagged by different models stay distinguishable.

## How messages get in

### Model replies

Give the model a system prompt like [SYSTEM.md](SYSTEM.md) so its output carries its own envelope: technical output starts with `HTTP/1.1 200 OK` and headers such as `X-Verdict` or `X-Scope`; small talk starts with `PUBLISH topic/path`. The first line decides the pipe:

```
HTTP/     -> curldb add
PUBLISH   -> mosquitto_pub, or append to a log file; never stored
```

### What you type

Type as usual, no HTTP. An adapter wraps it:

```bash
echo 'what about tool calls' | curldb wrap 'POST /chat' -H X-Topic:tool-call | curldb add
```

`wrap` adds only the start line, a `Date:` and the headers you pass; the body is untouched. To tag your own words, tee a copy to a cheap annotating model that emits header lines only, merge them back into the envelope, store. You stay out of the loop.

### Tool calls

A tool call is already a request and its result already a response. The translation is mechanical, no model involved:

```
tool name     -> POST /tool/Read
args (JSON)   -> body, Content-Type: application/json
result        -> response body
success/error -> status 200 / 4xx / 5xx
```

It lives in the harness hook (Claude Code's PreToolUse / PostToolUse):

```bash
# PreToolUse
echo "$ARGS_JSON" | curldb wrap "POST /tool/$TOOL" -H Content-Type:application/json -H X-Tool:$TOOL | curldb add
# PostToolUse
echo "$RESULT" | curldb wrap 200 -H X-Tool:$TOOL | curldb add
```

A turn that mixes prose with several tool calls is split at protocol boundaries and each piece is stored on its own.

Envelopes are stored as UTF-8. Tool args produced by Python's `json.dumps` with default settings escape non-ASCII to `\u4e2d`; they are stored escaped and full-text search will not find the characters. Use `json.dumps(args, ensure_ascii=False)` in the hook.

### Notes and files

The `---` YAML front matter at the top of a markdown file is header + body too. `curldb add note.md` reads a local file: `kind=note`, the file path becomes `path`, raw is the whole file. The front matter is flattened into the headers index:

```
title: hello              ->  title: hello
tags: [rust, iot]         ->  tags: rust / tags: iot        (list = repeated name)
metadata:                 ->  metadata.type: feedback       (nesting = dotted name)
  type: feedback
desc: |                   ->  desc: first line second line  (block scalar = lines joined)
  first line
  second line
```

Supported is the common front matter subset: `key: value`, indentation nesting, `- item` and `[a, b]` lists, `|` / `>` blocks, quotes. Anchors and inline maps are stored as text. A file without front matter is `kind=raw`, the whole file is the body, and it is searchable all the same.

One file per call; a whole directory is a shell loop:

```bash
for f in vault/*.md; do curldb add "$f"; done
curldb tags                     # the tag panel of this vault
curldb query 'header:tags=iot'
```

### Network door

`curldb serve` opens an HTTP door onto the same file. It is a peer of the CLI, operation for operation:

```
POST /<anything>     stored as received (request line, all headers, body), 201 + Location: /<id>
GET  /<id>           the whole stored message, Content-Type: message/http
HEAD /<id>           standard HEAD
GET  /?q=<expr>      same as curldb query
GET  /tags[/<name>]  same as curldb tags
```

The received request is the envelope; nothing to wrap:

```bash
curldb serve                                       # 127.0.0.1:200, --db picks the file
curl -X POST localhost:200/chat -H 'X-Topic: fork' -d 'what about fork?'
curl localhost:200/42
curl 'localhost:200/?q=status=409'
```

The default port is 200. Ports below 1024 need root on Linux/macOS; `curldb serve 8200` avoids sudo.

An AI can curl straight in. A Codex review sent as `PUT /review` is stored as that PUT request with the review in the body; to store it as a response, use `curldb add` from the CLI.

Bound to localhost, no token. The trust model is the CLI's: whoever can run curl on this machine.

## Querying afterwards

```bash
curldb tags                                  # which headers exist in this file, with values
curldb tags X-Verdict                         # every value of one header

curldb query 'status=400'                     # the model said you were wrong
curldb query 'status=409'                     # conflicts with something established
curldb query 'header:X-Verdict=shaky'         # conclusions that did not hold
curldb query 'kind=request path=/chat'        # things you said
curldb query 'header:X-Tool status=500'       # failed tool calls
curldb query 'path~/tool/ body~timeout'       # tool calls that timed out
curldb query 'status=200 header:X-Scope=design body~protocol'

curldb get 42                                 # the raw envelope
curldb ls 50                                  # the latest 50
```

## Usage

```
curldb add [file]           store one envelope from stdin or a file: HTTP request/response, or markdown with front matter
curldb wrap START [-H N:V]  wrap the stdin body in an envelope; START is a status code (200) or a request line (POST /chat)
curldb get <id>             print the raw envelope by id
curldb headers <id>         print the headers of a record
curldb query '<expr>'       search
curldb tags [name]          header names and values, with counts
curldb ls [n]               the latest n records (default 20)
curldb stats                database status
curldb serve [port]         HTTP door on 127.0.0.1, default 200

--db PATH or CURLDB_PATH picks the file, default ./curldb.sqlite
```

### Query DSL

All conditions are ANDed, separated by spaces:

```
kind=request|response|note|raw
status=200              status code
status=200,201          status in set
method=POST             request method
path=/chat              request path equals
path~/tool/             request path contains
header:X-Verdict        header exists
header:X-Verdict=solid  header value equals
body~fork               body full-text search
body~"exact phrase"     body phrase search
anyword                 bare word, body search
```

Full-text search uses FTS5 with the trigram tokenizer (SQLite 3.34+), so CJK substrings match directly; older SQLite falls back to unicode61 + LIKE, and `curldb stats` shows which one is in use.

## Install

```bash
pip install curldb
```

Or copy the one file:

```bash
cp curldb.py ~/.local/bin/curldb
chmod +x ~/.local/bin/curldb
```

Zero dependencies. Python 3.10+, standard-library sqlite3.

## Development and verification

Run the standard-library tests from the repository root, no test dependencies needed:

```bash
python -m unittest discover -s tests -v
```

The tests cover raw round trips (UTF-8, LF / CRLF / CR), combined queries, CJK search and the tokenizer fallback, front matter, legacy layout migration, the CLI and the local HTTP door. Databases live in temporary directories; the HTTP tests use a system-assigned port.

Build and check a release:

```bash
python -m pip install build twine
python -m build
python -m twine check --strict dist/*
python scripts/check_dist.py dist
```

`dist` should hold exactly one wheel and one sdist from this build. The check script verifies license, version, zero runtime dependencies and sdist contents, then installs the wheel offline into a temporary virtual environment and exercises the installed command line. The sdist contains `SYSTEM.md`, the tests and the check script.

GitHub Actions runs the tests on Windows and Linux with Python 3.10 and 3.14, plus a build-and-install check of the release artifacts.

## Design choices

- **Verbatim storage** -- the raw envelope goes into a SQLite TEXT column as-is.
- **Rebuildable index** -- kind / status / method / path / headers / body_fts are all parsed from raw.
- **Lenient parsing** -- accepts `\n` and `\r\n`, ignores malformed headers, Content-Length optional. It is a document parser.
- **One session, one file** -- backup is `cp`, sync is rsync.
- **One CLI** -- in, out, exit. `serve` is a second door onto the same cabinet, opened when wanted.
- **Append only** -- no update, no delete.
- **Stores HTTP messages, queries HTTP messages** -- `GET /<id>` returns the stored message itself (`message/http`). A stored message is always data; it is never replayed as the server's own reply. To view HTML an AI wrote: `curldb get 42 > x.html` and open it locally.

---

# 中文

HTTP exchange 原生存储。一个 session 一个 SQLite 文件,AI 对话里的 request 和 response 原样存,按信封字段查。

curl 是参考客户端,名字由此而来:server 接受的每个操作都是一行 curl,存下来的记录就是 curl 发出去的那条消息,一个字节不差。

## 干嘛的

LLM 对话里的每条消息都是一个 HTTP 信封:你的话是 request,模型的回答是 response,工具调用是 request + response。curldb 把信封原样存进 SQLite,解析出 status / method / path / headers 建索引,body 做全文索引。

```
你的话 -----------> curldb wrap 'POST /chat' -H X-Topic:x ---+
模型的回答 (HTTP/1.1 200 OK ...) ---------------------------+--> curldb add --> session.sqlite
工具调用 (POST /tool/Read ... / HTTP/1.1 200 OK ...) --------+
                                                                   |
                                    curldb query 'status=409 header:X-Scope=design'
                                    curldb tags
```

raw 是源数据,所有索引从 raw 解析出来,丢了可以重建。

## 三条原则

**一个 session 一个文件。** session 的身份就是文件名,`curldb --db 2026-09-11-design.sqlite` 或 `CURLDB_PATH`,默认当前目录的 `curldb.sqlite`。跨 session 查就是对多个文件各查一遍。

**header 是 tag,不是 schema。** 名字和值都不用事先约定。`X-Verdict: shaky` 这种传统程序读不懂的东西,读它的是 LLM。程序只管存、数、列;`curldb tags` 把这个库里出现过的 header 名和值全列出来,看一眼再查。词表是从数据里长出来的。

**body 永远原样。** 加 header 的人(你、模型、标注模型、hook)只加 header,谁都不改 body。标注者用 `Via:` 留名,以后换模型重标,新旧记录能区分。

## 三种消息怎么进来

### 模型的回答

给模型配 [SYSTEM.md](SYSTEM.md) 那样的 system prompt,让输出自带信封:技术产出以 `HTTP/1.1 200 OK` 开头,带 `X-Verdict`、`X-Scope` 这类 header;闲聊以 `PUBLISH topic/path` 开头。第一行决定这条消息走哪条管道:

```
HTTP/     -> curldb add
PUBLISH   -> mosquitto_pub,或 append 到日志文件;不进库
```

### 你的话

你照常打字,不用写 HTTP。adapter 套信封:

```bash
echo '那 tool call 呢' | curldb wrap 'POST /chat' -H X-Topic:tool-call | curldb add
```

`wrap` 只加 start line、`Date:` 和你给的 header,body 一个字不动。要给你的话打 tag,tee 一份给一个便宜的标注模型,它只出 header 行,合回信封再入库。你不在环里。

### 工具调用

tool call 本来就是 request,结果本来就是 response,机械翻译,不需要模型:

```
tool name     -> POST /tool/Read
args (JSON)   -> body, Content-Type: application/json
result        -> response body
success/error -> status 200 / 4xx / 5xx
```

放在 harness 的 hook 里(Claude Code 的 PreToolUse / PostToolUse):

```bash
# PreToolUse
echo "$ARGS_JSON" | curldb wrap "POST /tool/$TOOL" -H Content-Type:application/json -H X-Tool:$TOOL | curldb add
# PostToolUse
echo "$RESULT" | curldb wrap 200 -H X-Tool:$TOOL | curldb add
```

一个 turn 里 prose 和多个 tool call 混着,按协议边界切开各存各的。

信封一律按 UTF-8 存。tool args 如果是用 Python 的 `json.dumps` 默认参数生成的,中文会变成 `\u4e2d` 这种转义,存进去就是转义,全文搜索搜不到中文;hook 里用 `json.dumps(args, ensure_ascii=False)`。

### 笔记和文件

markdown 顶上那段 `---` 夹着的 YAML front matter 也是 header + body。`curldb add note.md` 直接读本地文件,`kind=note`,文件路径当 `path`,raw 是整个文件原文。front matter 打平进 headers 索引:

```
title: hello              ->  title: hello
tags: [rust, iot]         ->  tags: rust / tags: iot        (列表 = 同名重复)
metadata:                 ->  metadata.type: feedback       (嵌套 = 点号)
  type: feedback
desc: |                   ->  desc: first line second line  (多行块按空格拼)
  first line
  second line
```

支持的是 front matter 常见子集:`key: value`、缩进嵌套、`- item` 和 `[a, b]` 列表、`|` / `>` 块、引号。锚点和行内 map 当文本存。没有 front matter 的文件 `kind=raw`,整个文件是 body,一样能搜。

一次一个文件,整个目录用 shell 循环:

```bash
for f in vault/*.md; do curldb add "$f"; done
curldb tags                     # 这个 vault 的 tag 面板
curldb query 'header:tags=iot'
```

### 网络入口

`curldb serve` 给同一个文件开一个 HTTP 门,和 CLI 平级,功能一一对应:

```
POST /<anything>     收到什么存什么(请求行、所有 header、body),201 + Location: /<id>
GET  /<id>           整条存的消息,Content-Type: message/http
HEAD /<id>           标准 HEAD
GET  /?q=<expr>      同 curldb query
GET  /tags[/<name>]  同 curldb tags
```

收到的请求本身就是信封,不用 wrap:

```bash
curldb serve                                       # 127.0.0.1:200,--db 选文件
curl -X POST localhost:200/chat -H 'X-Topic: fork' -d 'what about fork?'
curl localhost:200/42
curl 'localhost:200/?q=status=409'
```

端口默认 200。1024 以下在 Linux/macOS 要 root,不想 sudo 就 `curldb serve 8200`。

AI 可以直接 curl 进来。Codex 的 review 以 `PUT /review` 发过来,存的是这个 PUT 请求,review 原文在 body 里;要把它当一条 response 存,走 CLI 的 `curldb add`。

只绑 localhost,没有 token:信任模型和 CLI 一样,能在这台机器上跑 curl 的人。

## 事后查

```bash
curldb tags                                  # 这个库里有哪些 header,值是什么
curldb tags X-Verdict                         # 一个 header 的所有值

curldb query 'status=400'                     # 模型说你说错了
curldb query 'status=409'                     # 和已定的东西冲突
curldb query 'header:X-Verdict=shaky'         # 站不住的结论
curldb query 'kind=request path=/chat'        # 你说过的话
curldb query 'header:X-Tool status=500'       # 失败的工具调用
curldb query 'path~/tool/ body~timeout'       # 超时的工具调用
curldb query 'status=200 header:X-Scope=design body~协议'

curldb get 42                                 # 原始信封
curldb ls 50                                  # 最近 50 条
```

## 用法

```
curldb add [file]           从 stdin 或文件存一个信封:HTTP request/response,或带 front matter 的 markdown
curldb wrap START [-H N:V]  给 stdin 的 body 套信封;START 是 status code (200) 或 request line (POST /chat)
curldb get <id>             按 id 取原始信封
curldb headers <id>         看某条的 headers
curldb query '<expr>'       查询
curldb tags [name]          header 名和值,带计数
curldb ls [n]               最近 n 条(默认 20)
curldb stats                数据库状态
curldb serve [port]         HTTP 门,127.0.0.1,默认 200

--db PATH 或 CURLDB_PATH 选文件,默认 ./curldb.sqlite
```

### query DSL

所有条件 AND 连接,空格分隔:

```
kind=request|response|note|raw
status=200              status code
status=200,201          status in set
method=POST             request method
path=/chat              request path 等于
path~/tool/             request path 包含
header:X-Verdict        header 存在
header:X-Verdict=solid  header 值等于
body~fork               body 全文搜索
body~"exact phrase"     body 短语搜索
anyword                 裸词,body 搜索
```

全文索引用 FTS5 trigram(SQLite 3.34+),中文子串直接搜;老 SQLite 自动退到 unicode61 + LIKE,`curldb stats` 里能看到用的哪个。

## 安装

```bash
pip install curldb
```

或者就一个文件,复制走:

```bash
cp curldb.py ~/.local/bin/curldb
chmod +x ~/.local/bin/curldb
```

零依赖。Python 3.10+,标准库 sqlite3。

## 开发与验证

在仓库根目录运行标准库测试,不需要安装测试依赖:

```bash
python -m unittest discover -s tests -v
```

测试覆盖原文存取(UTF-8、LF / CRLF / CR)、组合查询、中文搜索及 tokenizer 回退、front matter、旧库迁移、CLI 和本地 HTTP 入口。数据库都放在临时目录,HTTP 测试使用系统分配的临时端口。

构建和检查发行包:

```bash
python -m pip install build twine
python -m build
python -m twine check --strict dist/*
python scripts/check_dist.py dist
```

`dist` 中应只有本次构建的一份 wheel 和一份源码包。检查脚本核对许可证、版本、零运行时依赖和源码包内容,再在临时虚拟环境中离线安装 wheel,验证实际安装后的命令行存取。源码包包含 `SYSTEM.md`、测试和检查脚本。

GitHub Actions 在 Windows / Linux 的 Python 3.10 / 3.14 上运行测试,另有发行包构建与安装检查。

## 设计选择

- **原文存储** -- raw 信封原样进 SQLite TEXT 字段。
- **索引可重建** -- kind / status / method / path / headers / body_fts 都从 raw 解析出来。
- **宽松解析** -- 接受 `\n` 和 `\r\n`,忽略畸形 header,Content-Length 可选。这是文档解析器。
- **一个 session 一个文件** -- 备份是 `cp`,同步是 rsync。
- **一个 CLI** -- 进出,用完退出。`serve` 是同一个档案柜的第二扇门,想开就开。
- **只追加** -- 没有 update,没有 delete。
- **存 HTTP 消息,查 HTTP 消息** -- `GET /<id>` 回的是存的那条消息本身(`message/http`),存的消息永远是数据,不当 server 自己的回复回放。想看 AI 写的 HTML:`curldb get 42 > x.html`,本地打开。
