Metadata-Version: 2.4
Name: casket-store
Version: 0.1.2
Summary: simple CAS program
Author-email: "Pierre A. Sauriol" <pantosaur@gmail.com>
License-Expression: GPL-3.0-or-later
Project-URL: Homepage, https://codeberg.org/pantosaur/casket
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: POSIX
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# casket

A simple content-addressed store for large files, kept out of git as plain
files and linked into projects with symlinks.

## Goals

- No service to run: stores and remotes are plain directories, local or
  reached over ssh.
- No location tracking: remotes are asked what they have when needed, nothing
  records where objects live.
- Clear control over storage: objects are plain read-only files, links show
  the key they point to, the config is a text file.

## Model

A central store is the source of truth. Its tree of symlinks is the catalog,
organized by hand. Projects consume it: they hold symlinks to assets and fetch
the objects behind them into their own `.casket`.

    casket init                                  # create .casket here
    casket remote add central /mnt/nas/assets/.casket
    casket put rock.png draft.png                # store files, replacing them by links
    casket publish rock.png textures/            # add to central as textures/rock.png
    casket publish assets/ lib/                  # add every link under assets to central's lib
    casket -c central list -l textures/          # browse central's catalog, with sizes and keys
    casket -c central mv textures/ lib/textures  # reorganize central's catalog
    casket add textures/rock.png assets/         # link a central asset here, fetching it
    casket add textures/ assets/textures         # link every asset under it; again to get new ones
    casket fetch                                 # get the objects of the links under .
    casket get rock.png                          # replace a link by a writable copy
    casket drop assets/                          # free local copies central has, keep the links
    casket gc                                    # remove unreferenced objects central has
    casket verify                                # check every object against its key

The commands pair up, each moving one thing in one direction:

| between                   | in            | out           |
|---------------------------|---------------|---------------|
| files and the local store | `put`         | `get`         |
| local store and a remote  | `fetch`       | `drop`        |
| project and central trees | `add`         | `publish`     |

Commands that move things between two stores (`fetch`, `drop`, `add`,
`publish`, `gc`) take the remote with `-r`. Every other command works on one
store: the one owning its target, or the one given with `-c`/`--cas`, before
or after the command. `-c` takes a path, an ssh url or the name of a remote,
and runs the command in that store's root, so paths are relative to it; over
ssh the command runs on the server.

`publish` only sends the objects central lacks, and only along with a link in
its tree: nothing reaches central without a place in the catalog.

In a git project, commit the links and ignore the store: add `/.casket` to
`.gitignore`, without a trailing slash so that it also matches a `.casket`
symlink to a shared cache.

The central store does not know who links to it. Cleaning it (`gc --force` on
central) may break the history of projects that linked what was removed; that
is a deliberate decision of whoever runs it.

## Remotes

A remote is another store's `.casket` directory (a store is always a
directory named `.casket`, which `init` and `remote add` enforce), given as:

- a path: `/mnt/nas/assets/.casket`, or relative to the project root;
- over ssh: `nas:/srv/assets/.casket`, `me@nas:assets/.casket` (relative to the
  home directory), `ssh://me@nas:2222/srv/assets/.casket`.

An ssh remote uses the system `ssh`, so keys, agents and `~/.ssh/config` work
as for git. On the server it needs `python3` (3.9 or later) and nothing
installed: casket sends its own code over the connection and runs it there,
so a store over ssh follows exactly the rules of a local one. Each command
uses one connection; fetched objects are checked locally, uploads are checked
on the server before they land.

Change where a remote is with `casket remote set-url NAME URL`, forget one
with `casket remote remove NAME`.

Remotes added with `--global` (`-g`) go in `~/.config/casket/config` (under
`$XDG_CONFIG_HOME`) and are known to every store, and to `-c NAME` anywhere:

    casket remote add -g nas nas:/srv/assets/.casket
    casket -c nas list

A store's own remote of the same name wins over a global one. `remote list`
marks the global remotes; `set-url` and `remove` take `-g` too. A global
remote's path must be absolute: there is no project root to be relative to.

Create a store on a server the same way as a local one:
`casket --cas nas:/srv/assets/.casket init`.

Set `CASKET_SSH` to change the ssh command, e.g.
`CASKET_SSH="ssh -i ~/.ssh/assets_key"`.

## Compatibility contract

Projects commit the symlinks to git, so what a symlink contains must stay
valid forever. Everything else may change.

1. **Key format.** Keys are `SHA256E-s{size}--{sha256}{extension}`, e.g.
   `SHA256E-s123456--abcdef….png`. New key formats may be added, but every key
   format ever written must keep being recognized.
2. **Link targets.** A link is a relative symlink to
   `.casket/objects/{ab}/{cd}/{KEY}/{KEY}`, where `abcd` are the first 4 hex
   characters of the MD5 of the key string. The `.casket` and `objects` names and
   this layout never change.
3. **Scope.** A directory holding an entry named `.casket` is the root of a
   scope: its tree, minus `.casket` itself, ignored directories (`.git`)
   and the trees of nested scopes. Any entry named `.casket` makes a scope,
   whatever it is: a store, a symlink to a shared cache (missing or not), or
   something that is not a store at all. The scope of a path is the nearest
   one at or above it, never one further up.
4. **Link recognition.** A symlink belongs to a scope's store if it lies in
   the scope and its target's final name parses as a key.
5. **Store marker.** A store's `config` holds `[casket]` `format = 1`.
   casket refuses to work in a scope whose `.casket` is not a store of a
   format it handles: a directory without the marker, a dangling symlink, a
   future format. It never falls back to a store further up, which would
   claim and rewrite links that are not its own. A future layout gets a new
   format number, which older versions refuse.

`.casket` may itself be a directory or a symlink to a shared cache: links go
through the `.casket` name either way.

## Shared cache

Projects can share one store by making their `.casket` a symlink to it:

    casket --cas ~/.cache/casket/.casket init
    ln -s ~/.cache/casket/.casket .casket

Each project still scopes its own tree, but the objects and the config,
remotes included, are shared. `gc` only sees the links of the project it runs
in, so in a shared cache it removes objects other projects use: `gc` makes
them fetch again, `gc --force` can lose their local-only objects.

## License

casket is free software: you can redistribute it and/or modify it under the
terms of the GNU General Public License as published by the Free Software
Foundation, either version 3 of the License, or (at your option) any later
version. See [LICENSE](LICENSE).
