Metadata-Version: 2.5
Name: basalt-vault
Version: 1.0.0
Summary: Python implementation of the Basalt Protocol v1 — a portable knowledge vault in any object store
Project-URL: Homepage, https://github.com/CarterCole/python-basalt
Project-URL: Repository, https://github.com/CarterCole/python-basalt
Project-URL: Issues, https://github.com/CarterCole/python-basalt/issues
Project-URL: Specification, https://github.com/CarterCole/basalt/blob/main/protocol/PROTOCOL.md
Author: Carter Cole
License-Expression: MIT
License-File: LICENSE
Keywords: acl,basalt,knowledge-base,markdown,s3,vault,wikilink
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: System :: Archiving
Classifier: Topic :: Text Processing :: Markup :: Markdown
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: regex>=2024.5.15
Requires-Dist: ruamel-yaml>=0.18
Provides-Extra: s3
Requires-Dist: boto3>=1.34; extra == 's3'
Provides-Extra: topics
Requires-Dist: gensim>=4.4.0; extra == 'topics'
Description-Content-Type: text/markdown

# basalt-vault — the Basalt Protocol v1 in Python

An independent implementation of [the Basalt Protocol](https://github.com/CarterCole/basalt/blob/main/protocol/PROTOCOL.md),
held to the same language-neutral fixture corpus as the
[TypeScript reference](https://github.com/CarterCole/ts-basalt) in
[`conformance/`](https://github.com/CarterCole/basalt/tree/main/conformance).

**Conformance: L0–L4** (§14.1) — Transport, Content, Index, Security,
Gatekeeper. L5 (history) is not implemented.

The load-bearing claim is in §0 and §13: two independent implementations given
the same content MUST emit byte-identical index objects. This package
reproduces every file in `conformance/vaults/minimal/expected/` — generated by
the TypeScript implementation — byte for byte, and the reverse holds too: the
TypeScript `check` command confirms that indexes written by this package
reproduce exactly.

```
$ basalt conformance-regen --vault ../conformance/vaults/minimal/vault --out /tmp/regen
$ diff -r /tmp/regen ../conformance/vaults/minimal/expected/   # no output
```

## Install

```bash
pip install basalt-vault
```

This installs the `basalt` package and the `basalt` command. `boto3` is an
optional extra, so the core protocol — indexing, link resolution, ACL
evaluation — runs with no cloud dependency installed:

```bash
pip install 'basalt-vault[s3]'
```

[Topic search](#topic-search-optional-not-protocol) is a second extra, because
gensim brings numpy and scipy with it:

```bash
pip install 'basalt-vault[topics]'
```

From a checkout, `uv sync` (with `--extra s3` or `--extra topics`) sets up the
same thing for development.

Dependencies are deliberately few: `ruamel.yaml`, because §13.4 requires
comments be preserved when an implementation edits YAML it did not author, and
`regex`, because §9.6's inline-tag grammar and §10.5's tokenizer are written
in Unicode property classes (`\p{L}`, `\p{N}`) that the standard `re` module
does not implement. The gatekeeper is `http.server`.

## The self-check, §14.3

§14.3 makes this command required rather than optional, "because without it
the answer is unknowable":

```bash
basalt check --vault ../conformance/vaults/minimal/vault \
             --principal u:reader --path "Notes/Columns.md"
```

```
vault 01JBQ7X8Z4N3VYK9M2T5HRWC6P Minimal Conformance Vault
audience=gated bypass_traverse=false

integrity  0 errors, 1 warning(s)
  warn  UNRESOLVED_LINK Notes/Hexes.md "Does Not Exist"

index      fresh
rebuild    reproduces stored indexes byte for byte

access     u:reader on Notes/Columns.md
  deny  read            by deny:g:staff
  allow list            by allow:everyone
  allow traverse        by allow:everyone
  …
```

It reports all four things §14.3 asks for: integrity (§10.7), index freshness
(§10.6), whether a rebuild reproduces the stored indexes byte for byte (§13),
and the ACL evaluation of a named principal against a named path (§8.7) —
naming the ACE that decided each right, which is what makes a band-ordering
bug visible.

## Commands

```
basalt check       self-check (§14.3)
basalt index       rebuild every index from content alone; --write to persist
basalt mint        mint node ids for unbound objects (§4.3)
basalt add         store a note, image or file as a NEW object (§12.7)
basalt gist        export a folder to a GitHub gist, or import a gist into a vault
basalt serve       run the gatekeeper (§12)
basalt effective   effective rights with the deciding ACE per right (§8.7)
basalt tree        the visible tree for a principal
basalt digest      the content digest and index freshness (§10.6)
basalt find        find notes by topic (optional, not protocol; needs the topics extra)
```

Every command takes `--vault DIR`, `--git DIR` (a folder in a git working
tree: each write is one commit, `--remote NAME` to pull and push), `--gist ID`
(a vault inside a GitHub gist: text only, at most 300 files), or an
S3-compatible store:

```bash
basalt check --bucket my-vault --endpoint https://….r2.cloudflarestorage.com --flavor r2
basalt check --bucket my-vault --endpoint http://localhost:9000 --flavor minio
```

The flavors exist because the three stores disagree in ways §1 pushes into the
adapter: R2 requires region `auto` and rejects the integrity headers newer
SDKs send by default, MinIO needs path-style addressing, and no two of them
compute an etag for a multipart upload the same way — which is exactly why
§10.1 records `content_hash` separately and treats *it* as authoritative.

## Layout

```
src/basalt/
  order.py        §13.3  the one comparator all ordering goes through
  keys.py         §1.1, §2  key validation, path algebra, collisions
  canonical.py    §13.1  canonical JSON; json.dumps is wrong four ways
  yamlio.py       §13.4  YAML in, YAML out, comments intact
  frontmatter.py  §5     parsing with error recovery, kinds
  links.py        §9     extraction, exclusion zones, the six resolution steps
  acl.py          §7, §8 principals, roles, the four-band evaluation
  security.py     §8.5, §8.6  descriptor chains and the traverse composition
  indexer.py      §10    every index, plus the merkle digest
  vault.py        §3     the facade: load, scan, reindex, read, mint
  store.py        §12.7  creating objects: notes, images, any file
  gatekeeper.py   §12    the HTTP read surface
  cli.py          §14.3  the self-check and friends
  topics.py       (none) topic search; optional and not protocol, see below
  gist.py         (none) `basalt gist export` and `import`
  transport/      §1     the five operations: local, memory, git, gist, S3
tests/
  test_conformance.py  the L0-L2 corpus
  test_acl.py          the L3 corpus
  test_expected.py     byte-exact index reproduction
  test_gatekeeper.py   L4, including the §12.5 cache-leak test
  test_store_conformance.py  the §12.7 corpus, shared with ts/ and rust/
  test_git.py          the git store, against repositories on disk
  test_gist.py         the gist name corpus, export, import and the gist store,
                       against a local stand-in for the gist API
  …
```

## Topic search (optional, not protocol)

```bash
basalt find "volcanic rock" --vault ../conformance/vaults/demo/vault
basalt find --like "Geology/Basalt.md" --vault ../conformance/vaults/demo/vault
```

```
1.00  Basalt  Geology/Basalt.md
0.88  Columnar Jointing  Geology/Columnar Jointing.md
0.71  Columns  Start Here.md
```

`search.json` matches words: a query for "volcanic stone" finds only the notes
that contain them. `basalt find` fits a latent semantic model (gensim: TF-IDF,
then a truncated SVD) and ranks notes by how close they are in topic, so a note
about hexagonal basalt columns answers a query about volcanic stone without
sharing a word with it. From Python it is `basalt.topics.TopicSearch`, whose
`find` and `related` take the principal to ask as.

It is not part of the protocol, and these are the places that shows:

- **Python only.** The TypeScript and Rust implementations have no equivalent
  and the gatekeeper has no route for it; §12.3 is the protocol's read surface
  and this is not on it.
- **Nothing is written to the vault.** A fitted model is floating point that
  moves with library versions, so it is not an index (§0) and cannot be held to
  byte-identical output (§13). It is fitted on demand and kept in memory.
- **A model never learns from a note its requester cannot read.** It is fitted
  per principal over only the notes that principal may read, because a model's
  vocabulary and loadings carry the text it was fitted to (§8.8), and cached by
  *which notes those are* rather than by who asked (§12.5). The cost is one fit
  per distinct visibility set; 300 notes fit in about a fifth of a second.
- **It needs a few notes to generalise from**: at least three readable ones,
  and `basalt find` says so rather than returning nothing. The number of topics
  defaults to about the square root of the note count, since a model with as
  many dimensions as notes is a keyword match; `--topics` overrides it.
- **Scores below 0.3 are dropped**, a cutoff chosen on a 300-article corpus
  where the lower scores were mostly noise. `min_score` changes it from Python.

## Tests

```bash
uv run pytest
```

312 tests, of which 15 need the `topics` extra and are skipped without it. The
157 that come from `conformance/` define nothing themselves —
every expectation is loaded from the corpus, because §14.2 is explicit that no
implementation defines its own answers for behavior the spec covers. The rest
cover the conditions §4–§12 describe in prose that no single fixture exercises
end to end: duplicate-id repair, orphaned sidecars, protected folders, the
traverse chain, inline ACLs replacing folder entries, §8.8's search leak rule
in both directions, and §12.3's 404/403 split. The topic tests are about the
same thing for a model: that its vocabulary never holds a private note's words,
and that a model warmed by the owner is not served to the reader.

## Python-specific notes

Three places where the obvious Python is the wrong Python:

- **`json.dumps` cannot produce §13.1.** `sort_keys` sorts by code point
  rather than by the UTF-8 byte sequence §13.3 specifies, `ensure_ascii`
  escapes exactly what §13.1 forbids escaping, and U+2028/U+2029 are left
  literal. The serializer in `canonical.py` is written out.
- **`bool` is a subclass of `int`.** An `isinstance(value, int)` test in a
  JSON writer renders `True` as `1`. Checked in that order, with a test.
- **UTF-8 is order-preserving**, so Python's own `str` ordering happens to
  agree with §13.3 — the ordering that differs is UTF-16 code unit order,
  which a JavaScript implementation has to correct for. This package still
  sorts on `s.encode("utf-8")`, because §13.3 specifies bytes and sorting the
  bytes cannot drift.

See the implementation report for the spec ambiguities found while writing
this.
