PVDM
Serverless Data Mesh
Vaquar Pattern · GitHub Pages
Website PVDM paper arXiv Python 3.12+ PyPI version PyPI downloads/month PyPI downloads/week Docker License AWS Lambda Iceberg

Docs, demo walkthrough & developer path

Pipelines are created from YAML + compiler, not by drawing in the UI. The control UI runs locally (Python stdlib HTTP) to observe a compiled mesh, then you package and deploy with Terraform.

Greenfield E2E — captioned demo (XYZ mesh)

Scripted captions: greenfield developer → YAML/JSON metadata for domain xyz → apply creates bronze/silver/gold → local control UI observe → readers → Terraform → Step Functions / PVDM end-to-end. Not a draw-on-canvas designer — see caption script.

Final Control UI — create mesh (region · accounts · VPC)

Final tour: Create data mesh (region, accounts, VPC none/existing/create) → domains → pipelines → PVDM paper. Direct file: control-ui-final.mp4.

Design Studio — create mesh (region · accounts · VPC)

Create data mesh: pick region, single or three accounts, and VPC — default (no VPC) · use existing · create new. If the player fails, use the direct MP4 link. Paper: arXiv:2608.14643.

Control UI tour (recorded)

Tab walkthrough: Overview → Pipelines → Trust → PVDM → Durable → Tutorial → Run PVDM demo.

1. YAML→ 2. apply / compile→ 3. local UI→ 4. readers.py→ 5. Terraform→ 6. AWS run

Local control UI (not Jetty — Python ThreadingHTTPServer)

pip install "serverless-data-mesh"
# from a clone of this repo:
serverless-data-mesh apply ^
  --contract examples/medallion-e2e/northstar.mesh.yaml ^
  --output examples/medallion-e2e/generated

serverless-data-mesh ui --path examples/medallion-e2e/generated --open
# → http://127.0.0.1:8765/
# → http://127.0.0.1:8765/walkthrough

On Mac/Linux use \ instead of ^. Full developer instructions: developer-ui-and-deploy.md

Interactive walkthrough

Auto-play GIF demo: install → YAML → apply → UI → deploy → observe.

GitHub Pages

Greenfield E2E video (MP4)

Captioned story: metadata → all layers → UI → Terraform (~65s).

Video

UI tour video (MP4)

Download the recorded control-center tab tour (~21s, H.264).

Video

Developer UI → Terraform

How the UI starts, create workflow, package Lambda, terraform apply, first run.

Primary guide

Terraform guide

Prod dual clocks, LMI, sync vs async_callback Step Functions modes.

Deploy

Metadata / medallion

One YAML → bronze / silver / gold. Control UI observes; it does not design.

Compiler

GitHub repository

Source, examples, Terraform modules, PyPI package.

Code