Metadata-Version: 2.4
Name: covalence
Version: 0.1.0.dev3
License-File: LICENSE-APACHE
License-File: LICENSE-CC0
License-File: LICENSE-MIT
Summary: Python bindings for Covalence
License-Expression: CC0-1.0 OR MIT OR Apache-2.0
Requires-Python: >=3.11
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Documentation, https://github.com/hiloopai/covalence/blob/main/crates/python/README.md
Project-URL: Issues, https://github.com/hiloopai/covalence/issues
Project-URL: Repository, https://github.com/hiloopai/covalence

# Covalence for Python

Content identifiers (CIDs), DASL values, and content-addressed and key-value
stores, from Python.

```pycon
>>> from covalence import Cid, Value
>>> cid = Cid.of(b"payload", "sha256")
>>> Cid(str(cid)) == Cid(bytes(cid)) == cid
True
>>> cid.hash, cid.codec
(Hash('sha256'), Codec('raw'))
>>> value = Value({"a": cid})
>>> Value.decode(value.encode()) == value
True
>>> value.links() == [cid]
True

```

## Install

```sh
pip install covalence==0.1.0.dev3
```

- Pre-releases need an exact pin like this one, or `pip install --pre`.
- Wheels cover CPython 3.11 and later on Linux (x86_64 and aarch64, glibc
  2.28 or later) and macOS (arm64).
- Elsewhere pip builds the sdist, which needs Rust 1.98 or later.
- Each wheel and the sdist include `THIRD-PARTY-LICENSES.md`, the licenses of
  the Rust crates compiled into the package.

## Terms

- A **CID** ([content identifier](https://github.com/multiformats/cid)) names
  bytes by their hash. It holds a codec, which says how the bytes are
  encoded, a hash function, and the digest.
- A **multicodec** code is a number from the
  [multicodec table](https://github.com/multiformats/multicodec) that names a
  codec or a hash function.
- [**DASL**](https://dasl.ing) is a small set of specifications for CIDs and
  data. [**DRISL**](https://dasl.ing/drisl.html) is its deterministic CBOR
  encoding, which CIDs label with the `dag-cbor` codec.
- A **CAS** (content-addressed store) keeps bytes and finds them by their
  CIDs.

## CIDs, hashes and codecs

- `Cid` is any CIDv1 with a digest of at most 64 bytes. `Cid(x)` takes text
  or binary; `Cid.from_text` and `Cid.from_bytes` take only one.
- Text is `b` and lowercase base32 only. Invalid CIDs raise `ValueError`.
- `Hash` and `Codec` wrap multicodec codes. `from_code` takes an int, and
  the constructors take names:
  - hashes: the multicodec names `"sha1"`, `"sha2-256"`, `"sha2-512"` and
    `"blake3"`, or the `hashlib` names `"sha256"` and `"sha512"`;
  - codecs: `"raw"`, `"dag-cbor"` (or `"drisl"`) and `"git-raw"`.
- `Cid.of` hashes with `"blake3"` and `"raw"` by default.
- `Cid.of_file(path)` is the raw BLAKE3 `Cid` of a file, as `Cas.put_file`
  returns it, without storing it or reading the file into memory.

## Values

- `Value` is a DASL value. `Value(...)` takes None, bool, int, str, bytes,
  `Cid`, `Value`, lists, tuples and dicts with str keys. It checks the same
  rules as `decode`:
  - links must be DASL CIDs;
  - integers outside i64, floats and nesting deeper than 126 raise
    `ValueError`.

  Other types raise `TypeError`.
- `Value.to_json` and `Value.from_json` use the ATProto JSON form of
  `covalence-lib-dasl`.
- `Value.cid()` is the DRISL BLAKE3 CID of the value's encoding.

## Key-value stores

- `Kv` is a byte key-value store:
  - `Kv.memory()` makes one in memory;
  - `Kv.s3("s3://bucket/prefix/")` (or `Kv.s3("bucket", "prefix")`) makes
    one in S3.
- S3 is configured as the AWS SDKs are:
  - the region is `region=`, else from `AWS_REGION`, the profile or EC2
    instance metadata, which can take a moment off EC2;
  - the endpoint is from `AWS_ENDPOINT_URL` or the profile's `endpoint_url`,
    else AWS's for the region, so other S3 services need `AWS_ENDPOINT_URL`;
  - credentials come from the SDKs' default chain: environment variables,
    profile files, web identity, ECS or EKS Pod Identity, or EC2 instance
    roles. They are fetched at the first request; missing or failed
    credentials raise `StoreError` then.
  - `profile=` picks the profile instead of `AWS_PROFILE` or `default`; its
    credentials win over environment variables.
- `get` returns `bytes` or `None`.
- `put` replaces any value under the key.
- `exists` says whether there is a value without reading it (a HEAD on S3).
- `Kv.s3` raises `ValueError` for an invalid location, a URL with a prefix
  argument too, or no region.
- Without `s3:ListBucket`, S3 answers 403 for a missing key, so reading one
  raises `StoreError` instead of returning `None` or `False`, here and
  through `Cas.kv`.
- S3 keys must be UTF-8 with non-empty `/`-separated segments other than `.`
  and `..`; other keys raise `StoreError`.
- On S3, values over 8 MiB go up as multipart uploads, and values come down
  in ranged GETs of 8 MiB, 16 parts or ranges at a time. Covalence sets no
  size limit; S3's is 5 TiB.

## Content-addressed stores

```pycon
>>> from covalence import Cas
>>> cas = Cas.memory()
>>> cid = cas.put(b"hello")
>>> cas.size(cid)
5
>>> cas.get_range(cid, 1, 3), cas.get_range(cid, 4, 10)
(b'ell', b'o')

```

- `Cas` is a CAS. Make one with:
  - `Cas.memory()`;
  - `Cas.sqlite(path)`, in memory without a path;
  - `Cas.kv(kv, prefix)`, keeping blocks in a `Kv` under
    `prefix/blake3/<digest>/bytes`.
- `put` returns the raw BLAKE3 `Cid`, after checking that the store kept the
  data under it.
- `Cas.kv` never replaces a block its `Kv` already holds: on `Kv.s3` each
  write is conditional (`If-None-Match: *`), so concurrent writers of one
  block race safely and the first one's bytes stay.
- `put_value(value)` stores the DRISL encoding of a `Value` (or what
  `Value(...)` takes) and returns its DRISL CID, which is `value.cid()`.
- `get` returns the bytes and `get_fact` a `BlobFact`, both checked against
  the CID; a failed check raises `IntegrityError`.
- `has` and `missing(cids)` say which CIDs the store holds bytes for,
  without checking their integrity:
  - `Cas.sqlite` checks its index, without reading the bytes;
  - `Cas.kv` asks its `Kv` whether the key exists (a HEAD on S3);
  - `Cas.memory` looks the bytes up in memory.
- `info` and `stats` return native values.
- `size(cid)` and `get_range(cid, start, length)` return a byte count, or up
  to `length` bytes from `start`, as the store vouches for them.
  `size_unchecked` and `get_range_unchecked` skip any checking. No store
  checks sizes or ranges yet.
  - Missing values return `None`, even for a zero `length`.
  - Ranges are clipped at the end.
  - Stores without native ranges fetch the whole value and slice it.

### Files and streams

- `Cas.put_file(path)` stores a file and returns its raw BLAKE3 `Cid`, as
  `put` does.
- `Cas.put_from(cid, chunks, length)` stores a block from an iterable of
  bytes, such as a request body, and returns its raw BLAKE3 `Cid`. Before a
  new block is stored, its bytes must match `length` and the BLAKE3 `cid`,
  and `chunks` must end there. A store may return an existing block without
  consuming all the input; unread bytes and source errors are not checked.
  While consuming input, a hash mismatch (`IntegrityError`), too few or too
  many bytes (`ValueError`), or a source exception publish no new block.
  Source exceptions propagate unchanged unless cleanup also fails, in which
  case `StoreError` reports that failure with the source as its `__cause__`.
- `Cas.get_file(cid, path)` writes the block to a file and returns its
  length, or `None` if missing.
- `put_file`, `put_from` and `get_file` stream on `Cas.kv` over `Kv.s3`,
  holding up to 16 parts of 8 MiB (128 MiB) at a time, or as many as
  `Kv.s3(..., concurrency=n)` says; blocks over 78 GiB go up in larger parts. Other stores, including `Cas.kv(Kv.memory())`, hold
  the block in memory.
- `put_file` reads the file twice, to hash it and to store it, and stores the
  bytes it hashed. If they changed in between it raises `IntegrityError`,
  and if the file got shorter `OSError`, storing nothing either way. Bytes
  appended in between are not stored.
- `get_file` checks the bytes against the CID in a private temporary file
  beside `path`. That file replaces `path` only once the check passes, so a
  missing block or a failed check (`IntegrityError`) leaves `path` as it
  was. The new file has the permissions of the file it replaces, or those
  the umask gives a new file.

## Blobs and facts

- `Blob` is bytes read whole or by range:
  - `Blob(data)` holds a snapshot of `data` in memory, and `Blob(blob)`
    shares `blob`'s source;
  - `size()`, `read_range(start, length)` and `read_all()` are not checked,
    and ranges are clipped at the end;
  - a `read_range` returning more than `length` bytes raises `StoreError`;
  - `slice(start, length)` is a blob over the same bytes;
  - copies keep their class and share the source.
- `Blob` can be subclassed, but its methods are not virtual: an override in
  Python changes only what Python callers see, while `fact` and `slice` read
  the Rust blob underneath.
- `Blob.fact(cid)` reads the blob once and returns a `BlobFact` for exactly
  `cid`, checked, or `None` if the bytes do not hash to it:
  - it checks SHA-1, SHA-256 and BLAKE3 CIDs;
  - a digest over 32 bytes, such as SHA-512's, raises `ValueError`;
  - another hash, or a digest length the hash does not give, raises
    `UnsupportedError`.
- `Fact` is an interface for bytes and the `Cid` they claim to hash to: `cid`
  and `value()`, which decodes the bytes as the CID's codec says. `BlobFact`
  is checked; `isinstance(x, Fact)` alone proves nothing.
- `BlobFact` is a `Fact` and a `Blob` of its bytes, with `data` too. It
  compares and hashes by fact, and cannot be subclassed.
- `BlobFact.of(data)` hashes `data` with BLAKE3. A fact asked for its own
  hash and digest under another codec is relabelled, not hashed again.

## Errors

- Bad input raises `ValueError` or `TypeError`, or `OverflowError` for an int
  below 0 or of 2**64 or more where a size, offset or code is expected.
- Other failures raise a subclass of `covalence.Error`:
  - `IntegrityError` (also a `ValueError`) when data does not match its CID;
  - `StoreError` (also an `OSError`) when a store's storage fails;
  - `UnsupportedError` (also a `NotImplementedError`) for an unsupported
    operation, such as hashing with an unknown function.
- A local file that cannot be read or written raises `OSError`.

## Developing

- `glu build` builds the `covalence` wheel into `target/glu/wheels/`.
- `glu test` installs it in a fresh venv and runs `tests/`.
- Classes are named after the Rust types they wrap.

## Documentation

The compiled module is the private `covalence._native`; users import
everything from `covalence`. PyO3's typing guide recommends a hand-written
stub; this package writes its stub by hand and checks it against `__doc__`:

- Rust `///` comments are the source. PyO3 makes them `__doc__`, with
  signatures from the Rust arguments. Python code in `covalence/` keeps its
  own docstrings.
- `_native.pyi` repeats each docstring word for word, since editors read the
  stub and cannot see a compiled module's `__doc__`. `tests/test_docs.py`
  fails if the two differ, or if the stub lacks a public member.
- Classes and exceptions set their module to `covalence` (`#[pyclass(module
  = "covalence")]`), so they show as `covalence.Name`, never
  `covalence._native.Name`. `_native` itself is not documented.
- Every class, method, property and exception in `covalence.__all__` has a
  docstring. Examples are `>>>` doctests. `tests/test_docs.py` runs them,
  and this README's, with `doctest.DocTestSuite` and `DocFileSuite`.
- [`docs/python`](https://github.com/hiloopai/covalence/tree/main/docs/python)
  builds the docstrings into HTML with Sphinx, for Read the Docs.

## Prebuilt wheels

`.github/workflows/python-wheels.yml` builds abi3 wheels (CPython 3.11 and
later) for manylinux_2_28 x86_64 and aarch64 and macOS arm64, and an sdist,
and runs `tests/` against each wheel. Pull requests and pushes keep them as
workflow artifacts. A release tag must match the workspace version, such as
`0.1.0-dev.2`; the files carry its PEP 440 form, `0.1.0.dev2`:

- `v<version>` attaches them to that GitHub release. In the public
  `hiloopai/covalence` repository it also publishes them to PyPI, and
  `.github/workflows/crates.yml` publishes the crates to crates.io.
- `python-v<version>` only attaches them to the GitHub release.

To publish `0.1.0-dev.3`:

```sh
git tag v0.1.0-dev.3 && git push origin v0.1.0-dev.3
```

