PVDM
Serverless Data Mesh
Vaquar Pattern · GitHub Pages
Website PVDM paper arXiv Python 3.12+ PyPI version PyPI downloads/month PyPI downloads/week Docker License AWS Lambda Iceberg

Docs, demo walkthrough & developer path

Create a mesh in the Design Studio (region · accounts · VPC) or from YAML, then apply compiles bronze/silver/gold pipelines. The local control UI observes the generated mesh; package and deploy with Terraform.

Greenfield E2E — captioned demo (XYZ mesh)

Scripted captions: greenfield developer → YAML/JSON metadata for domain xyz → apply creates bronze/silver/gold → local control UI → readers → Terraform → Step Functions / PVDM end-to-end. Caption script.

Control UI — create mesh (final)

Live Design Studio: region · accounts · VPC (none / existing / create) → bronze/silver/gold domains → pipelines → PVDM paper. Rebuild: python scripts/build_final_ui_video.py.

1. Design / YAML→ 2. apply / compile→ 3. local UI→ 4. readers.py→ 5. Terraform→ 6. AWS run

Local control UI (not Jetty — Python ThreadingHTTPServer)

pip install "serverless-data-mesh"
# from a clone of this repo:
serverless-data-mesh apply ^
  --contract examples/medallion-e2e/northstar.mesh.yaml ^
  --output examples/medallion-e2e/generated

serverless-data-mesh ui --path examples/medallion-e2e/generated --open
# → http://127.0.0.1:8765/
# → http://127.0.0.1:8765/walkthrough

On Mac/Linux use \ instead of ^. Full developer instructions: developer-ui-and-deploy.md

Interactive walkthrough

Auto-play GIF demo: install → YAML → apply → UI → deploy → observe.

GitHub Pages

Greenfield E2E video (MP4)

Captioned story: metadata → all layers → UI → Terraform.

Video

Control UI final (MP4)

Create mesh: region · accounts · VPC → domains → pipelines.

Video

Developer UI → Terraform

How the UI starts, create workflow, package Lambda, terraform apply, first run.

Primary guide

Terraform guide

Prod dual clocks, LMI, sync vs async_callback Step Functions modes.

Deploy

Metadata / medallion

Design Studio or YAML → bronze / silver / gold via apply.

Compiler

GitHub repository

Source, examples, Terraform modules, PyPI package.

Code