Open source · MIT · Made by data engineers, for data engineers

The Databricks API,
running on your laptop.

MiniLake is a Databricks API emulator. SQL runs on DuckDB, Delta tables are real files, and jobs run spark-submit in a container. Point databricks-sdk, the Databricks CLI, Terraform or an MCP agent at http://localhost:8000; any token is accepted.

Open source today, and it will stay that way.

$ pip install minilake && minilake --port 8000 PyPI

$ docker run -p 8000:8000 ghcr.io/dmux/minilake:latest GHCR

Get started See it in code

Then open http://localhost:8000/ui/ for the built-in SQL workspace. The image downloads nothing at runtime — it works air-gapped.

What it does

The API is emulated; the engines behind it are not.

SQL executes on DuckDB, Delta tables are written to disk, and jobs run spark-submit. Everything else in the API is documented below with its status.

Unity Catalog

Catalogs, schemas, tables and volumes with full CRUD. Every catalog is its own DuckDB database file, so catalog.schema.table addressing is native — not a regex rewrite.

SQL on DuckDB

Statement execution on real DuckDB: INLINE and EXTERNAL_LINKS dispositions, JSON_ARRAY / ARROW_STREAM / CSV formats, and a result manifest that carries true column types.

Delta Lake

EXTERNAL Delta tables are genuine Parquet plus a _delta_log/. INSERT, UPDATE and DELETE are routed through a real Spark job; reads go through DuckDB's delta_scan().

Jobs

spark_python_task runs spark-submit in a sibling container and returns actual stdout. Real DAG scheduling honours depends_on and run_if, with independent branches in parallel.

Terraform & Asset Bundles

Point the databricks provider at localhost and terraform apply creates real catalogs. Verified on every test run: the real provider is driven through apply, plan and destroy, and a second plan has to come back empty. databricks bundle deploy and bundle run work end to end — jobs execute for real.

Designed for CI

One POST /_minilake/reset gives every test a clean workspace without restarting the container. /health, /ready and /services round out the control plane.

Zero code changes

Change the host. That's the migration.

MiniLake speaks the Databricks REST API on the wire, so every client you already use keeps working — the same calls, the same response shapes, the same typed errors.

Any token is accepted. No sign-up, no PAT, no OAuth dance to develop against.
Types round-trip. STRING, LONG, DECIMAL(10,2), ARRAY<T>, MAP and STRUCT map to DuckDB and come back as the SDK's ColumnInfo.
Physical truth wins. Declared column types that disagree with the files lose, so mismatches are visible instead of believed.
Persistence when you want it. MINILAKE_PERSIST=1 snapshots state on shutdown and restores it on start.

quickstart.py

from databricks.sdk import WorkspaceClient
from databricks.sdk.service.catalog import ColumnInfo, TableType

# The only line that differs from production
w = WorkspaceClient(host="http://localhost:8000", token="dev")

w.catalogs.create(name="sales")
w.schemas.create(name="store", catalog_name="sales")
w.tables.create(
    name="orders", catalog_name="sales", schema_name="store",
    table_type=TableType.MANAGED,
    data_source_format="DELTA", storage_location="/data/sales/store/orders",
    columns=[
        ColumnInfo(name="id", type_text="BIGINT"),
        ColumnInfo(name="customer", type_text="STRING"),
        ColumnInfo(name="amount", type_text="DECIMAL(10,2)"),
    ],
)

wh = w.warehouses.create(name="dev", cluster_size="Small")
w.statement_execution.execute_statement(
    warehouse_id=wh.id,
    statement="INSERT INTO sales.store.orders VALUES (1, 'Ana', 459.90)",
)

result = w.statement_execution.execute_statement(
    warehouse_id=wh.id,
    statement="SELECT customer, amount FROM sales.store.orders",
)
print(result.result.data_array)   # [['Ana', '459.90']] — really stored

from pyspark.sql import SparkSession

UC = "http://localhost:8000"

spark = (
    SparkSession.builder
    .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
    # Unity Catalog connector — three-part names resolve against MiniLake
    .config("spark.sql.catalog.spark_catalog", "io.unitycatalog.spark.UCSingleCatalog")
    .config("spark.sql.catalog.spark_catalog.uri", UC)
    .config("spark.sql.catalog.sales", "io.unitycatalog.spark.UCSingleCatalog")
    .config("spark.sql.catalog.sales.uri", UC)
    .getOrCreate()
)

# Real Delta files: parquet + _delta_log/, written by real Spark
spark.createDataFrame([(1, "a"), (2, "b")], ["id", "name"]) \
     .write.format("delta").mode("overwrite").save("/data/delta/sales/store/events")

spark.sql("INSERT INTO sales.store.events VALUES (3, 'c')")
print(spark.table("sales.store.events").count())   # 3

# ...and the same bytes are readable through MiniLake's SQL API:
#   SELECT e.name, o.customer
#   FROM sales.store.events e JOIN sales.store.orders o ON o.id = e.id

provider "databricks" {
  host  = "http://localhost:8000"
  token = "dev"
}

resource "databricks_catalog" "sandbox" {
  name    = "sandbox"
  comment = "Local sandbox catalog"
}

resource "databricks_schema" "things" {
  catalog_name = databricks_catalog.sandbox.name
  name         = "things"
}

resource "databricks_sql_endpoint" "compute" {
  name         = "local-compute"
  cluster_size = "Small"
}

# terraform apply  -> creates real catalogs and real DuckDB databases
# terraform destroy -> removes them
# Resources backed by APIs MiniLake does not emulate fail loudly with 501,
# instead of silently pretending to succeed.

bundle:
  name: my_pipeline

targets:
  dev:
    mode: development
    workspace:
      host: http://localhost:8000

resources:
  jobs:
    daily_load:
      name: daily_load
      tasks:
        - task_key: main
          spark_python_task:
            python_file: ../src/load.py

# $ databricks bundle deploy -t dev
# $ databricks bundle run daily_load -t dev
#
# The CLI syncs files through the Workspace API, the provider creates the job,
# and `bundle run` executes it on real Spark in a sibling container —
# returning the job's actual output.

# Is it up, and which services answered?
curl http://localhost:8000/_minilake/health

# Create a catalog through the Databricks API surface
curl -X POST http://localhost:8000/api/2.1/unity-catalog/catalogs \
  -H 'Content-Type: application/json' \
  -d '{"name": "sales"}'

# Run SQL and get real rows back
curl -X POST http://localhost:8000/api/2.0/sql/statements \
  -H 'Content-Type: application/json' \
  -d '{"warehouse_id": "dev", "statement": "SELECT 42 AS answer"}'

# Give the next test a clean workspace — no restart needed
curl -X POST http://localhost:8000/_minilake/reset

How it works

One process. One port. Real engines underneath.

A FastAPI application serves the Databricks REST surface, the web UI and the MCP endpoint together. Behind it sit DuckDB, real Delta files on disk, and Docker for Spark.

For LLM agents

An MCP server for agents.

Optional, at /mcp. Several tools collapse multi-call sequences into one, and a resource describes the DuckDB SQL dialect so agents stop writing Spark SQL. Tools go through MiniLake's own ASGI stack, so an agent sees the same errors an SDK client would.

setup_fixture()catalog + schema + warehouse in one idempotent call
seed_table()table + rows, exactly the rows you asked for
describe_catalog_tree()the whole hierarchy instead of dozens of calls
run_python_script()stage, submit to real Spark, poll, return stdout

MCP documentation All tools

# One line to register it with Claude Code
claude mcp add --transport http minilake http://localhost:8000/mcp

# Then just ask:
#   "Use minilake: create a sales catalog with a seeded
#    orders table, and tell me the total."

# The agent does this — three calls, no glue code:
#   setup_fixture({"catalog": "sales", "schema": "store"})
#   seed_table({"full_name": "sales.store.orders", ...})
#   run_sql({"statement": "SELECT SUM(amount) FROM sales.store.orders"})

# Enable it on the server side:
MINILAKE_MCP=1 minilake --port 8000

Coverage

What's emulated.

Anything out of scope answers 501 NOT_IMPLEMENTED instead of faking a success. The full per-service list is in FEATURES.md.

Service	Status	What that means
Unity Catalog	Real	Catalogs, schemas, tables, volumes, functions — each catalog is its own DuckDB database, with native three-part addressing
SQL statements	Real	Executed on DuckDB; all three formats and both dispositions, with a typed result manifest
EXTERNAL Delta tables	Real	Genuine Delta files; writes go through a generated Spark job, reads through `delta_scan()`
Jobs	Real	`spark-submit` in sibling containers (or a subprocess fallback), real DAG scheduling, real logs, one-shot `runs/submit`
Workspace, DBFS & Files	Real	File-backed storage with chunked upload; raw-bytes sync is what makes `bundle deploy` work
Clusters	State machine	CRUD plus timed lifecycle transitions — deliberately no Spark compute; that lives in Jobs
Permissions	Allow-all CRUD	Real create/read/update, but nothing is ever enforced — this is a single-user tool
Identity & SCIM	Unenforced CRUD	Users, groups and service principals, with SCIM filtering and `PATCH` — records, not credentials; any token is still accepted
DLT, Model Registry, Vector Search, Lakeview, Repos	Not implemented	Returns `501 NOT_IMPLEMENTED` — loudly, so you never mistake it for working

Limits

What MiniLake is not.

These are deliberate design decisions, not a backlog. MiniLake targets one developer running it locally — not a shared, multi-tenant server.

No authentication

Any token is accepted and there is exactly one user. Never expose it beyond localhost.

No access control

The Permissions API is real CRUD that always allows everything — a passing test here says nothing about grants in a real workspace.

DuckDB, not Spark SQL

Statements run on DuckDB's dialect. The MCP server ships a dialect resource for exactly this reason; the docs carry the translation table.

No Spark on clusters

The cluster lifecycle is a state machine. Real Spark happens through Jobs, in sibling containers.

Single process, no HA

DuckDB is single-writer, so concurrent load serializes on a lock. Fine for dev and CI; not a deployment target.

MANAGED tables are invisible to Spark

They are DuckDB tables with no files. Use EXTERNAL Delta when Spark has to see the data — that trade-off is what keeps SQL fast.

Open source

MIT licensed, developed in the open, and it stays that way.

MiniLake is written by data engineers for data engineers. The scope is what pipeline and CI work needs, and the limits above are listed because we would want them listed.

MIT license

Use it, fork it, ship it.

Developed on GitHub

Source, CI, changelog and issues are public at dmux/minilake.

Contributions welcome

Report the SDK call that fails in an issue, or add an API group following CONTRIBUTING.md.

Everyone who has committed to the project. See all contributors.

The Databricks API,running on your laptop.