Metadata-Version: 2.4
Name: aleth-ingest
Version: 0.1.0
Summary: Open-source file ingestion companion: walk a folder of text files and store them verbatim in Aleth memory over the public wire, skipping unchanged files and superseding changed ones (ALB-3, go/no-go Step 1). Depends only on aleth-client.
Author: Aleth Authors
License: Apache-2.0
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: aleth-client>=1.1
Dynamic: license-file

# aleth-ingest

The file ingestion companion for Aleth memory. Point it at a folder of text
files and it stores them verbatim in your memory, then keeps them in sync:
unchanged files are skipped, changed files supersede their old version, removed
files are retracted. Your memory does not bloat with duplicates.

It only ever calls the public memory verbs (`remember`, `supersede`,
`retract`, `forget`). It has no special access to the engine, so what it can
do, you can do.

## What it does

- **Walks a directory** of text files (`.md`, `.txt`, `.rst` by default; add
  your own globs).
- **Stores content verbatim.** Small files become one memory; larger files are
  split on paragraph and heading boundaries, never rewritten. Because the
  content is stored exactly, you can delete the original file and rely on the
  copy in memory.
- **Skips unchanged files** using a size + modified-time fast path, so a re-run
  over a big folder is cheap and does not re-read files that did not move.
- **Supersedes changed files, by name.** The new version is stored first,
  then each old memory is retired *behind the memory that replaced it*. Ask
  your memory what a file used to say and it can answer, because the link
  from the old text to the new one is recorded rather than implied. Nothing
  is destroyed unless you ask for it.
- **Leaves unchanged parts alone.** Edit one section of a long document and
  only that section's memories are replaced. The rest keep the identity they
  already had, so a small edit does not churn the whole file.
- **Retracts what nothing replaced.** A deleted file, or a section dropped
  when a file shrank, has no successor to name, so it is retracted instead --
  marked no longer current, with its history kept.
- **Tracks everything in a local manifest** (`<root>/.aleth-ingest/manifest.json`),
  so runs are idempotent and an interrupted run resumes correctly.

## Install

```
pip install aleth-ingest
```

Requires a running memory engine reachable on the same machine (the companion
finds it through its discovery file, the same way the memory client does).

## Use

```
aleth-ingest plan    ./notes     # dry run: show what would change (no engine needed)
aleth-ingest run     ./notes     # ingest: apply the plan
aleth-ingest status  ./notes     # manifest summary against what is on disk
aleth-ingest history ./notes     # what replaced what, per file
```

`history` prints one line per replacement, old memory to new:

```
  pricing.md
    2026-08-22T22:04:00+00:00
      da2b0a62-24b5-48c2-bef7-830e160580b7  ->  ac331252-fd06-4137-a04d-4158342d5551
```

Both ids are real memories you can look up. The old one is still there and
still readable; ask for it and your memory will tell you it is no longer
current and which memory took its place.

There is no "undo this replacement" command, because a memory that could be
un-replaced would not be much of a record. The way back is the way forward:
put the earlier text back in the file and run again. The old wording returns
as the current memory, and today's is retired behind it.

Common options:

```
--include GLOB     add an include glob (repeatable; default *.md,*.txt,*.rst)
--exclude GLOB     add an exclude glob (repeatable)
--manifest PATH    use a specific manifest file
--max-files N      cap the walk (default 1000); a truncated walk never removes
--max-supersessions N
                   cap replacements in ONE run (run; default 1000). Past the
                   cap a changed memory is retracted rather than replaced --
                   it still stops being current, it just does not name a
                   successor -- and the run tells you the cap bound. It is
                   there so a sweeping mechanical edit across a whole folder
                   is something you see rather than something that happens.
--origin TAG       origin recorded on each memory (run; default "user")
--hard-remove      permanently forget memories of removed files (run;
                   default is retract, which keeps history)
--discovery-dir D  where the engine's discovery file lives (run)
--timeout SECONDS  wire timeout (run; default 30)
```

`run` exits non-zero if any write fails.

## How re-runs stay clean

Every stored memory carries a one-line context describing its source file
(path, content hash, size, modified time, and which chunk it is). The companion
keeps the authoritative file-to-memory mapping in the local manifest. On the
next run it compares each file against the manifest by hash and only touches
what changed. Even without the manifest (say it was deleted), a re-ingest is
safe: the engine drops exact-duplicate content on its own.

That last property is also why an edit stays local. Because identical text
stores as the same memory, re-reading a document you barely touched hands back
the same memories for every unchanged part, so the companion recognises them
as still current and leaves them alone, and only the part you actually edited
is replaced.

## What is out of scope

This is a verbatim text-file tool by design. PDFs, office documents, images,
audio, and video are not handled here, and it does not watch folders or run as
a daemon. It ingests once when you run it, and again when you run it next.

## License

Apache-2.0.
