Metadata-Version: 2.4
Name: meshlogd
Version: 0.5.0
Summary: Event-sourced, peer-to-peer data store that converges over a partitioning Reticulum/LoRa mesh. No servers, no internet.
Project-URL: Homepage, https://meshlogd.org
Project-URL: Source, https://github.com/meshlogd/meshlogd
Project-URL: Issues, https://github.com/meshlogd/meshlogd/issues
Author-email: Kevin Hoffman <alothien+meshlogd@gmail.com>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: anti-entropy,crdt,event-sourcing,eventually-consistent,gossip,lora,mesh,offline-first,radio,reticulum
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Communications
Classifier: Topic :: System :: Networking
Requires-Python: >=3.9
Provides-Extra: dev
Requires-Dist: build; extra == 'dev'
Requires-Dist: twine; extra == 'dev'
Provides-Extra: radio
Requires-Dist: rns>=0.7; extra == 'radio'
Provides-Extra: service
Description-Content-Type: text/markdown

# Meshlogd
An **event-sourced, eventually-consistent data store** that converges over a frequently-partitioning, LoRa radio mesh. 

Built for the "trusted
collaborators, no internet, private radio" case: a base camp and field teams
(humanitarian assessment, SAR, expeditions) collecting a shared dataset with no
server and no connectivity, reconciling whenever radios come into range.

The transport is [Reticulum](https://reticulum.network) on real hardware
(LoRa/packet radio/anything), but the engine is transport-agnostic and ships
with an in-memory transport so you can run, test, and watch it converge with no
radios at all.

## Install

```bash
pip install meshlogd            # core engine (stdlib-only)
pip install "meshlogd[radio]"   # + Reticulum, for the real LoRa/packet-radio transport
```

> **Naming:** the distribution and daemon command are **`meshlogd`**; the importable
> library is **`meshlog`** (`import meshlog`) — the trailing `d` follows the Unix
> daemon convention (cf. `ssh`/`sshd`), like how `pyyaml` installs but imports as `yaml`.

## The idea in one breath

KoboToolbox already solves offline *data entry* but syncs to a central server
over the internet. meshlog replaces that central sync with **peer-to-peer
anti-entropy gossip over a radio mesh**, so the shared dataset converges with no
server and no internet. Kobo (or any form UI) owns the forms; meshlog owns the
replication layer Kobo doesn't have.

## Why event sourcing makes this easy

The whole database is a **grow-only set of immutable, per-author events**. That
single decision hands you the hard distributed-systems properties almost for
free:

- **Merging is set union.** Union is commutative, associative, and idempotent,
  so the dataset is a CRDT: it converges regardless of the order or grouping in
  which events arrive across a partitioning network.
- **Per-author contiguous feeds** (the Secure-Scuttlebutt trick) collapse "what
  do you have?" down to one integer per author — a **high-water-mark frontier**.
  That digest is tiny enough to ride inside a Reticulum *announce*.
- **Hash-chained feeds** make history tamper-evident (we're trusted-collaborator,
  so we want integrity — just not anonymity).
- **The materialized view is a pure fold** over the event set, so two nodes with
  the same events compute byte-identical state. Conflicts on mutable scalar
  fields resolve by **Last-Writer-Wins keyed on a Hybrid Logical Clock**, not
  wall time and not arrival order — so concurrent edits from two islands pick the
  same winner everywhere.

## Anti-entropy protocol (the entire thing)

```
when two peers can talk:
    each sends its FRONTIER  = {author: high_water_seq}
    on receiving a peer's frontier:
        push the slice of every feed above the peer's marks   (chunked)
    ingest is idempotent + gap-buffering, so overlaps/reorders are harmless
```

Multi-hop convergence is automatic: a node relays *every* author's events it
holds, not just its own, so A's events reach C through B even if A and C never
meet. That's the store-and-forward property, achieved without special-casing it.

## Layout

```
meshlog/
  hlc.py            Hybrid Logical Clock
  event.py          immutable, hash-chained, content-addressed Event
  feedstore.py      append-only per-author log; frontier + delta computation
  reducer.py        deterministic fold -> materialized records (Kobo-style)
  transport.py      Transport interface + in-memory SimTransport (partitions)
  node.py           Node + the anti-entropy protocol
  rns_transport.py  real Reticulum transport (Link + Resource + announce)
demo_sim.py         base camp + 4 teams converge across staged partitions
test_convergence.py 200 randomized partition trials + fold-determinism check
meshlog/cli.py      CLI to run a node over a live Reticulum mesh (the `meshlogd` command)
```

## Run it (no radios needed)

```bash
python -m demo_sim          # scripted scenario, prints the converged dataset
python -m test_convergence  # 200 randomized partition/merge trials
```

The demo islands three groups, has each collect independently, stages a
conflicting concurrent edit, then heals the network in stages and verifies every
node ends with identical frontiers, identical views, and the same LWW winner.

## Run it for real (over Reticulum)

```bash
pip install "meshlogd[radio]"          # pulls in Reticulum (rns)
# Machine A:
meshlogd --name base   --data base.jsonl   --create R-100 site_assessment
# Machine B (same LAN to start; add a LoRa interface in ~/.reticulum/config later):
meshlogd --name team_a --data team_a.jsonl --create R-200 water_point
# type `view` at either prompt to watch the dataset converge
```

## Verification status

- `feedstore`, `reducer`, `hlc`, `node`, `transport`, the demo, and the 200
  randomized trials **run and pass** as committed.
- `rns_transport.py` / `meshlog/cli.py` are written against the Reticulum 1.3.x API
  (Identity, Destination, Link, Resource, Packet, announce handlers) and are
  faithful to the docs/examples, but have **not** been exercised on a live mesh
  here. Stand up two instances to validate end-to-end before field use.

## Deliberately left as extension points

- **Attachments / blobs.** Events carry a content hash + size; the bytes are not
  inlined. Add a content-addressed blob store and transfer blobs lazily over RNS
  Resources, *after* structured events, so photos never starve assessment data on
  a slow link.
- **Signatures.** `Event.is_authentic()` currently checks the content hash. With
  RNS you already hold each author's Ed25519 key — sign the event id and verify
  on ingest to close the trusted-but-verify loop.
- **Wire codec.** Prototype uses JSON for readability. Swap `_encode/_decode` for
  MessagePack or CBOR to cut bytes on the radio.
- **Compaction / snapshots.** Grow-only logs grow. Add periodic signed snapshots
  + feed truncation once a quorum has acknowledged a watermark.
- **Kobo bridge.** Poll the KoboToolbox/ODK submission API (or read KoboCollect
  instances) and translate each submission into `record.create` + `field.set`
  events. That's the actual on-ramp to the HHI use case.

## Reading list (steal from these)

- **Secure Scuttlebutt** — append-only per-author feeds + gossip replication for
  intermittent offline networks. This design is a close cousin.
- **CRDTs** — why union-based / LWW-register merges converge.
- **Kulkarni et al. 2014, Hybrid Logical Clocks** — the clock used here.
- **Reticulum manual**, Understanding + API Reference chapters — Link, Resource,
  and the announce mechanism.
