Metadata-Version: 2.4
Name: ghostjournal
Version: 0.1.1
Summary: A local-first, append-only reflective journal substrate for LLM agents.
Author: Shelleyguitar
License-Expression: MIT
Project-URL: Homepage, https://pypi.org/project/ghostjournal/
Keywords: agents,journal,memory,embeddings,local-first,llm
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: filelock<4,>=3.16
Provides-Extra: nn
Requires-Dist: numpy>=1.24; extra == "nn"
Requires-Dist: sentence-transformers>=3.0; extra == "nn"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: build>=1.2; extra == "dev"
Dynamic: license-file

# ghostjournal

**0.1.1 hardening release.** Stop all 0.1.0 writers before upgrading. The lock
protocol changed from age-based lockfiles to OS-held advisory locks. Do not mix
old and new writers on one root. Existing valid entry JSON and existing manifest
bytes are preserved; new roots receive a journal UUID. Read CHANGELOG.md.

`ghostjournal` is boring, local-first infrastructure for agents that need a durable reflective journal rather than a chat-log dump.

Each journal entry is immutable JSON. SQLite, FTS, and optional embedding vectors are **derived state**: delete `index/`, run `ghostjournal reindex`, and the searchable journal is rebuilt from `entries/` without rewriting history.

## Install

```bash
pip install -e .
```

For local sentence embeddings:

```bash
pip install -e '.[nn]'
```

The NN extra uses `sentence-transformers` with
`sentence-transformers/all-MiniLM-L6-v2` by default. The base runtime dependency is
`filelock`; SQLite/FTS5 supplies lexical search. NN is now explicitly opt-in even
when the extra is installed. Ordinary encoder operations use CPU and
`local_files_only=True`, with `trust_remote_code=False`. Download the selected
model only with this explicit setup command:

```sh
ghostjournal --root ./journal download-model
```

That command requires the NN extra and network access. Later `--nn` operations
use the cached model and do not intentionally fetch model files. In strictly
offline deployments also set `HF_HUB_OFFLINE=1` and `TRANSFORMERS_OFFLINE=1` and
enforce network policy at the OS boundary. No GPU or API key is required for the
default public model. Actual model download/inference was not exercised in this
hardening build; the lexical path and adapter configuration were tested.

## API

```python
from ghostjournal import Journal

journal = Journal("./journal", enable_nn=False)

entry = journal.append({
    "kind": "pulse",
    "agent": "motoko",
    "voice": "I keep using practical light to explain where work happens.",
    "theme": "light as labor",
    "tags": ["light", "places"],
    "signals": {"focus": "light", "novelty": 0.5, "continuity_refs": []},
    "client_key": "motoko:pulse:2026-09-05T16",
    "meta": {"source_routine": "pulse-4h"},
})

print(journal.get(entry.id).to_dict())
print(journal.list(kind="pulse", limit=20))
print(journal.search("what did I care about in lighting?", k=8))
print(journal.relate(entry.id, k=8))
print(journal.digest())
print(journal.prompt_context("what production habits keep returning?", k=6))
```

`append()` supplies `id`, timezone-aware UTC `ts`, top-level `schema_version`, `tags`, and `meta.schema_version` when omitted. All supplied fields are validated strictly. Unknown top-level fields are rejected.

`client_key` is optional. When reused, `append()` returns the previously stored
entry, allowing cron retries without duplicate pulses. For compatibility, the
original payload wins even if a retry supplies different prose. This differs
from deeprem's strict retry-key conflict rule. All generated IDs are UUIDs.
Invalid/non-finite JSON values, invalid timestamps, and unsafe IDs are rejected.

## CLI

```bash
ghostjournal --root ./journal init

echo '{
  "kind": "pulse",
  "agent": "motoko",
  "voice": "The crane silhouette is becoming a landmark.",
  "theme": "recurring landmarks",
  "tags": ["places", "continuity"],
  "meta": {"source_routine": "pulse-4h"}
}' | ghostjournal --root ./journal append

ghostjournal --root ./journal list --kind pulse --limit 20
ghostjournal --root ./journal search "recurring places" -k 8
ghostjournal --root ./journal relate ENTRY_ID -k 8
ghostjournal --root ./journal digest
ghostjournal --root ./journal prompt-context "what did I care about last week?" -k 6
ghostjournal --root ./journal reindex
```

Add `--nn` before the subcommand to enable semantic embeddings when `ghostjournal[nn]` is installed:

```bash
ghostjournal --root ./journal --nn search "what identity am I developing?"
```

## On-disk layout

```text
journal/
  manifest.json
  entries/
    YYYY/MM/DD/<uuid>.json
  index/
    journal.sqlite3
  models/
  .write.lock
```

The JSON entry files are canonical. SQLite contains metadata, FTS content, idempotency keys, and—when enabled—float32 embedding blobs. Keeping vectors in SQLite avoids an additional vector database and makes the derived index transactional and simple to rebuild. `models/` is reserved for encoder/cache integrations; model caching itself follows the sentence-transformers/Hugging Face cache configuration.

Reads, writes, and reindex are serialized with an OS-held advisory lock. No live
lock is stolen based on age. Entry files are flushed/fsynced and published with
an atomic no-clobber hard link; newly created parent directories are synced on
POSIX. SQLite connections are explicitly closed. The journal's canonical files,
not a stale index, determine ID uniqueness and retry keys. An exact retry after
a crash between file publication and indexing repairs the index. Digests remain
computed views unless the caller explicitly appends a new `kind="digest"` entry.

Use a local filesystem supporting OS locks, hard links and atomic replacement.
Network filesystems and hostile same-account processes are not a supported
security boundary. This package is not encrypted or cryptographically signed;
use deeprem for evidence seals and review decisions, and encrypted storage for
source-journal confidentiality. Symlink entry paths and path/glob syntax in IDs
are rejected. `prompt_context()` now emits bounded, escaped historical JSON data,
not a privileged instruction block. It is not a complete prompt-injection defense.

## Entry schema v1

The packaged JSON Schema lives at `ghostjournal/schemas/entry-v1.schema.json`.

Core fields:

- `schema_version`: `1`
- `id`: UUID by default; custom IDs must match `[A-Za-z0-9][A-Za-z0-9_-]{0,127}`
- `ts`: timezone-aware ISO8601 timestamp
- `kind`: `pulse | evening | note | digest`
- `agent`: agent identity
- `voice`: reflective prose
- `theme`: short theme label
- `tags`: unique strings
- `mood`: optional short string
- `anchors`: optional `{type, ref, label?}` external references; ghostjournal never fetches them
- `signals`: optional `focus`, `novelty`, and `continuity_refs`
- `embedding`: optional numeric vector; normally embeddings are stored separately in the index
- `client_key`: optional idempotency key
- `meta`: extensible metadata object with required `schema_version` and optional `source_routine`

## Why structured JSON helps a “ghost” develop

Raw markdown preserves prose but forces every later agent run to rediscover what the prose means. A ghostjournal entry preserves both levels at once: `voice` keeps the subjective record, while stable machine fields expose theme, focus, novelty, provenance, tags, and explicit continuity links.

That gives retrieval more than a transcript. An agent can ask for semantically similar past thoughts, restrict by entry kind or time, aggregate recurring themes, and carry compact “past-you” context into its next reflective prompt. The library does not claim to create identity and does not call an LLM; it makes the agent’s self-observations durable, addressable, and comparable over time.

## Recovery contract

The `entries/` tree is the source of truth. With writers stopped, remove the entire `index/` directory and run:

```bash
ghostjournal --root ./journal reindex
```

All searchable metadata and optional vectors are recreated solely from immutable entry JSON. Historical JSON files are not modified. Opening an existing journal with a missing index also rebuilds it automatically. NN reindex requires the selected encoder to already be cached.

## Development

```bash
python -m pytest
python examples/motoko_sim.py
python -m build
```

## License

MIT
