---
tool: bin/chatlog_embed_sweeper.py
sha1: 35b62eed11b5
mtime_utc: 2026-08-07T23:53:51.770285+00:00
generated_utc: 2026-08-08T14:40:49.730088+00:00
private: false
---

# bin/chatlog_embed_sweeper.py

## Purpose

chatlog_embed_sweeper.py — lazy embed chat log rows missing embeddings.

Runs on a schedule (default every 30 min via install_schedules.py). Picks up
rows written with embed=False, embeds in batches using memory_core._embed_many,
and drains any spill-to-disk files from the async write queue.

---

## Entry points

- `async def main()` (line 289)
- `if __name__ == "__main__"` guard

---

## CLI flags / arguments

| Flag(s) | Help | Default | Default behavior | Type/Action | Impact when set |
|---|---|---|---|---|---|
| `--batch` | Batch size (default from config.embed_sweeper.batch_size) | None | Uses config batch size | int | Embeds N rows per batch iteration |
| `--max-per-run` | Max rows per run (default from CHATLOG_EMBED_MAX_PER_RUN env or 10000) | None | Embeds up to 10000 rows per run | int | Processes up to N rows total per invocation |
| `--deadline` | Soft wall-clock budget (seconds) for one run. The embed loop stops starting new batches once this elapses, so a large backlog is drained across several scheduled runs instead of monopolizing the GPU in one long run. Default from CHATLOG_EMBED_DEADLINE_S env or 60s; 0 disables (unbounded, the old behavior). max-per-run still caps total rows. | None |  | float |  |
| `--dry-run` | Query and log but don't embed | `False` | Queries unembed rows, embeds in batches, updates DB | store_true | Queries and logs row counts without embedding |
| `--drain-spill` | Process spill files before embedding | `False` | Skips spill drain (unless spill dir exists) | store_true | Always processes spill files before embedding |
| `--database` | SQLite database path. Env: M3_DATABASE. Default: memory/agent_memory.db. | None | Falls back to M3_DATABASE env then memory/agent_memory.db | str | Routes this run against PATH for all DB reads/writes |

---

## Environment variables read

- `M3_DATABASE`

---

## Calls INTO this repo (intra-repo imports)

- `_task_runtime (add_log_file_arg, setup_task_runtime)`
- `chatlog_config`
- `embed_sweep_lib (Counters)`
- `embed_sweep_lib (run_embed_loop)`
- `embedding_utils (pack)`
- `m3_sdk (add_database_arg)`
- `m3_sdk (getenv_compat)`
- `memory_core (_content_hash)`
- `memory_core (_embed_many)`
- `memory_core (_pack)`

---

## Calls OUT (external side-channels)

**sqlite**

- `sqlite3.connect()  → `db_path`` (line 121)
- `sqlite3.connect()  → `db_path`` (line 367)


---

## Notable external imports

_(only stdlib)_

---

## File dependencies (repo paths referenced)

_(none detected)_

---

## Re-validation

If the `sha1` above differs from the current file's sha1, the inventory is stale — re-read the tool, confirm flags/env vars/entry-points/calls still match, and regenerate via `python bin/gen_tool_inventory.py`.
