Metadata-Version: 2.5
Name: gulliver-mcp
Version: 0.1.0
Summary: Test on tiny data before you run on giant data. A dress rehearsal for data agents: estimate the cost, rehearse on 1% -> 10%, predict 100%, and only go full scale within budget.
Project-URL: Homepage, https://github.com/laaielichter/gulliver
Project-URL: Issues, https://github.com/laaielichter/gulliver/issues
Author: laaielichter
License-Expression: MIT
License-File: LICENSE
Keywords: ai-agents,bigquery,cost,data-engineering,duckdb,llm,mcp,postgres,snowflake,sql
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Database
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.10
Requires-Dist: duckdb>=1.1
Requires-Dist: mcp>=1.10
Requires-Dist: sqlglot>=25.0
Requires-Dist: tomli>=2.0; python_version < '3.11'
Provides-Extra: all
Requires-Dist: google-cloud-bigquery>=3.20; extra == 'all'
Requires-Dist: psycopg[binary]>=3.1; extra == 'all'
Requires-Dist: snowflake-connector-python>=3.10; extra == 'all'
Provides-Extra: bigquery
Requires-Dist: google-cloud-bigquery>=3.20; extra == 'bigquery'
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == 'dev'
Provides-Extra: postgres
Requires-Dist: psycopg[binary]>=3.1; extra == 'postgres'
Provides-Extra: snowflake
Requires-Dist: snowflake-connector-python>=3.10; extra == 'snowflake'
Description-Content-Type: text/markdown

# Gulliver

<!-- mcp-name: io.github.laaielichter/gulliver -->

**Test on tiny data before you run on giant data.**

Gulliver gives AI agents a dress rehearsal for SQL: estimate the cost, rehearse on 1% → 3% → 10% of the data, predict what 100% will do, and only run the full query inside a budget. Local-first, open source, MCP + CLI for any agent and any LLM.

[![ci](https://github.com/laaielichter/gulliver/actions/workflows/ci.yml/badge.svg)](https://github.com/laaielichter/gulliver/actions/workflows/ci.yml)
![license](https://img.shields.io/badge/license-MIT-blue)
![python](https://img.shields.io/badge/python-3.10%2B-blue)

> In *Gulliver's Travels*, Gulliver visits Lilliput, a land of tiny people, and Brobdingnag, a land of giants.
> Gulliver does the same with your data: it tries your query in Lilliput first, and only walks into Brobdingnag
> when it knows what will happen there.

## Why

Agents write SQL that works on a small table and breaks on a big one. They do not know that `events` has
5 billion rows, that a join multiplies rows, or that a forgotten partition filter scans every day since 2019.
Benchmarks score agents on *is the answer right*, never on *what did it cost* or *does it still work at 100x*.
One unlucky query can burn a monthly budget.

Gulliver sits between the agent and the warehouse:

| Step | Tool | What happens |
|---|---|---|
| 1 | `gulliver_inspect` | Which tables are big? Rows, bytes, partitions, columns. Metadata only, free. |
| 2 | `gulliver_preflight` | Bytes, rows and money **before** running, plus **scale lint** (12 rules) and the budget gate. |
| 3 | `gulliver_rehearse` | Runs the query on 1%, 3% and 10% samples, fits a growth curve and **predicts the full run** with a confidence range. Catches joins that multiply rows. |
| 4 | `gulliver_run` | Runs the full query only inside the budget. Otherwise answers `needs_rehearsal` or `needs_approval`. |
| 5 | `gulliver_receipt` | Markdown proof for the human: what was estimated, rehearsed, approved and run, and predicted vs actual. |

Every answer has a `status` and a `next_step`, so even small models know what to do next.

And for the pipeline around the query (details in [Pipeline tools](#pipeline-tools)):

| Tool | Problem it solves |
|---|---|
| `gulliver_impact` | "If I change or drop this column, what breaks?" Column-level lineage over SQL models or a dbt manifest. |
| `gulliver_drift` | Schema changed under you: columns removed, retyped or added since the last snapshot, plus the models they break. |
| `gulliver_diff` | "Does my refactored / migrated / incremental model return the same rows?" Key-based data diff inside the engine, also across engines. |
| `gulliver_translate` | SQL between DuckDB, Postgres, BigQuery and Snowflake, with warnings and an optional preflight on the target. |
| `gulliver_incremental` | A model rebuilds everything every run. Is incremental safe, what is the SQL / dbt config, and how many rows does it skip? |
| `gulliver_check` | Data quality in one cheap scan: unique keys, not null, accepted values, ranges, freshness, row-count jumps. |

## Quick start (2 minutes, no warehouse needed)

```bash
pip install git+https://github.com/laaielichter/gulliver
gulliver demo                       # creates gulliver_demo.duckdb with 5M events
export GULLIVER_DSN=gulliver_demo.duckdb

gulliver inspect
gulliver preflight "select country, count(*) from events e join users u on e.user_id = u.user_id group by 1"
gulliver lint "select * from events e, users u"           # GV003: join without a condition -> blocked
gulliver rehearse "select a.event_id, b.event_id from events a join events b on a.user_id = b.user_id"
```

The last one is the classic agent mistake: a self-join on a non-unique key. The rehearsal sees the output grow
with exponent ~1.8 instead of ~1 and warns that the full result has more than 100 million rows, before you run it.

```
| Sample | Runtime  | Output rows |
|--------|----------|-------------|
| 1%     | 0.0202s  | 41,706      |
| 3%     | 0.0484s  | 237,422     |
| 10%    | 0.3909s  | 2,438,395   |

Prediction for 100%: output rows 134,525,654 (range 67,262,827 - 269,051,308)
Growth exponent output_rows: 1.77 -> grows faster than the data
```

Add `--json` to any command for machine-readable output. Exit codes: `0` ok, `1` error, `2` blocked,
`3` needs rehearsal or approval, `4` a data check failed, so it also works as a CI check.

## Use it from your agent (MCP)

Gulliver is an MCP server (stdio or Streamable HTTP). Until it is on PyPI, run it straight from GitHub with
[uv](https://docs.astral.sh/uv/):

```bash
uvx --from git+https://github.com/laaielichter/gulliver gulliver-mcp --dsn path/to/warehouse.duckdb
```

**Claude Code**

```bash
claude mcp add gulliver -- uvx --from git+https://github.com/laaielichter/gulliver gulliver-mcp --dsn /abs/path/data.duckdb
```

Or install it as a plugin (MCP server + skill):

```
/plugin marketplace add laaielichter/gulliver
/plugin install gulliver@gulliver
```

**Claude Desktop, Cursor, Windsurf, VS Code and other MCP clients** (JSON config):

```json
{
  "mcpServers": {
    "gulliver": {
      "command": "uvx",
      "args": ["--from", "git+https://github.com/laaielichter/gulliver", "gulliver-mcp", "--config", "/abs/path/gulliver.toml"]
    }
  }
}
```

**Codex CLI** (`~/.codex/config.toml`):

```toml
[mcp_servers.gulliver]
command = "uvx"
args = ["--from", "git+https://github.com/laaielichter/gulliver", "gulliver-mcp", "--config", "/abs/path/gulliver.toml"]
```

**Gemini CLI** (`~/.gemini/settings.json`): same `mcpServers` block as above.

**Any HTTP client or remote agent:** `gulliver-mcp --http --port 8765` serves `http://127.0.0.1:8765/mcp`.

**No MCP?** Every tool is also a CLI command with `--json`, and `gulliver tools --format openai` or
`--format gemini` prints ready-made function-calling definitions for the OpenAI or Gemini APIs.

## Engines

| Engine | Estimate before running | Sampling | Money model | Status |
|---|---|---|---|---|
| DuckDB (files, Parquet, CSV, in-memory) | EXPLAIN + metadata (heuristic) | `TABLESAMPLE` system/bernoulli, seeded | free (time is the budget) | tested in CI |
| BigQuery | dry run: **exact** bytes | `TABLESAMPLE SYSTEM` | on-demand $/TiB, plus a hard `maximum_bytes_billed` cap | beta, tested with fakes |
| Snowflake | `EXPLAIN USING JSON`: bytes and partitions (upper bound) | `SAMPLE SYSTEM ... SEED` | warehouse credits x runtime | beta, tested with fakes |
| PostgreSQL | `EXPLAIN (FORMAT JSON)` (heuristic), partition-aware sizes | `TABLESAMPLE ... REPEATABLE` | self-hosted (time and bytes are the budget) | tested in CI against a live Postgres 16 |

Install the drivers you need: `pip install "gulliver-mcp[bigquery]"`, `[snowflake]`, `[postgres]` or `[all]`.
Every estimate says how much to trust it: `exact`, `upper_bound` or `heuristic`. A heuristic is never called exact.

## Configuration

Copy [`gulliver.example.toml`](gulliver.example.toml) to `gulliver.toml`:

```toml
default_connection = "local"

[connections.local]
engine = "duckdb"
path = "warehouse.duckdb"

[connections.bq]
dsn = "bigquery://my-project?location=EU"
dataset = "analytics"
price_per_tib_usd = 6.25

[budget]
max_query_usd = 1.00        # hard stop for one query
approval_over_usd = 0.25    # above this a human runs `gulliver approve <receipt_id>`
rehearse_over_usd = 0.10    # above this a rehearsal is required first
day_usd = 10.00             # total per day (local ledger)
max_runtime_s = 900
max_output_rows = 200       # rows handed back to the agent
```

Or skip the file: `GULLIVER_DSN`, `GULLIVER_MAX_QUERY_USD`, `GULLIVER_DAY_USD` and friends.
Credentials never go in the config: Gulliver uses each engine's own environment variables and credential chains
(Google ADC, `SNOWFLAKE_PASSWORD`, `PGPASSWORD`, ...). The agent only ever sees connection *names*.

## Safety model

- **Read-first.** Only `SELECT` (and `WITH ... SELECT`) runs by default. `DELETE`, `UPDATE`, `DROP`, `TRUNCATE`,
  `ALTER`, `MERGE`, several statements at once and unparsable SQL are refused by a parser, not a regex.
- **Writes need two keys.** `CREATE TABLE ... AS SELECT` and `INSERT ... SELECT` run only when the call says
  `allow_write` **and** a human set `allow_write = true` on that connection. Rehearsals never write.
- **Budgets fail closed.** Over the per-query, per-day, bytes or runtime limit means `block`. Money above the
  approval limit needs `gulliver approve <receipt_id>`, a CLI command for humans that is not exposed as an MCP tool.
- **Engine-side caps** where they exist: BigQuery `maximum_bytes_billed`, Snowflake and Postgres statement timeouts,
  a DuckDB interrupt timer.
- **Local state only.** Receipts, the spend ledger and approvals live in `./.gulliver/` as JSON. No telemetry.

## How the prediction works

Each rung measures runtime, output rows and (where the engine reports it) bytes. Gulliver fits
`metric = a * fraction^k` in log-log space and reads off the value at 100%.

- `k ≈ 1`: grows with the data (scans, filters, key joins against a full table).
- `k ≈ 0`: flat (aggregations). Group counts saturate, so grouped results are capped by what was observed and the planner estimate.
- `k > 1.3`: grows faster than the data: a join that multiplies rows, a big sort, or both join sides sampled. Flagged.

Runtime first subtracts the fixed per-query overhead (measured with `SELECT 1`). Confidence is `high`, `medium`
or `low`, with a range of x0.7-1.5, x0.5-2 or x0.25-4. After a real run, the receipt shows predicted vs actual.

By default only the **largest** table is sampled and the others stay complete, which keeps key joins honest.
`--sample all` samples every table. For BigQuery the money prediction uses the exact dry-run bytes of the full query.

**Limits, honestly:** block (system) sampling is cheap but can be skewed on clustered data; use `--method bernoulli`
when that matters. Extrapolating 10x always has uncertainty; Gulliver shows it instead of hiding it. Databricks,
ClickHouse and Spark are not supported yet.

## Pipeline tools

These reuse the same connections, budget gate and receipts. Every query they run is generated, parsed,
estimated and gated like your own queries.

### Impact: what breaks if a column changes

```bash
gulliver impact raw.events.amount --models models/        # or --manifest target/manifest.json
```
```
Changing raw.events.amount affects 4 model(s):
  stg_events: amount  <- raw.events.amount
    fct_revenue: revenue  <- stg_events.amount
    user_ltv: ltv  <- stg_events.amount
      country_report: revenue  <- fct_revenue.revenue
```

Plain `*.sql` files and dbt models (`{{ ref() }}`, `{{ source() }}`, `{{ config() }}`, `{% if %}`) both work.
Columns used only in filters or joins are reported too, and `SELECT *` models are flagged because their shape
follows the source. `gulliver lineage --models models/` prints the whole graph.

### Drift: schema changes since last time

```bash
gulliver snapshot                          # remember tables, columns and row counts
gulliver drift --models models/            # later: what changed, and which models it breaks
gulliver drift --save                      # accept the new schema
```

Removed and retyped columns are `warn` and are mapped to impacted models; row counts that move more than 20%
are listed too.

### Diff: prove two queries return the same rows

```bash
gulliver diff --left "select * from fct_orders" --right "select * from fct_orders_v2" --key order_id
gulliver diff --left orders --right orders --key id -c postgres_prod --right-connection snowflake_new
```

Each side is reduced inside its engine to 256 buckets of (row count, sum of MD5 row hashes) keyed by the key's
hash, so it scales to billions of rows and costs about one scan per side. Only buckets that differ are opened
(with a finer second pass when they are big) to list keys that are missing, extra or changed. MD5 is the same on
every engine, so you can diff across engines; values are compared as text, so cast or round in your queries where
engines format numbers or timestamps differently. Use it after a refactor, a migration or an incremental switch.

### Translate: SQL between dialects

```bash
gulliver translate --from snowflake --to bigquery -f model.sql
gulliver translate --from duckdb --to postgres --check -c pg "select ..."   # also preflight on the target
```

Functions the translator does not know are copied as-is and listed as warnings, so nothing is silently wrong.

### Incremental: stop rebuilding everything

```bash
gulliver incremental -f models/user_ltv.sql --time-column ts --unique-key user_id --period 1d
```

| Strategy | When | What it generates |
|---|---|---|
| `append` | no aggregation | read only rows newer than the target's max (merge on `unique_key` for late updates) |
| `partition_overwrite` | grouped by a time grain | dbt `microbatch` config: recompute the recent periods (+ lookback) |
| `merge_recompute` | grouped (or windowed) by an entity key | recompute only the keys that got new rows, merge on the key |
| `not_safe` | LIMIT, totals over everything, windows without PARTITION BY | keeps the full rebuild and says why |

It also measures the saving on the real source ("each run would read about 172,800 of 5,000,000 rows, 96.5% fewer")
and, on billed engines, compares dry-run bytes, telling you when you need to partition or cluster the source to
turn fewer rows into less money. Prove the result with `gulliver diff`.

### Check: data quality in one scan

```bash
gulliver check events --unique event_id --not-null user_id,amount \
  --accepted "event_type=view|click|cart|buy" --range "amount=0:10000" --fresh ts=24h --max-change 30
```

All checks become one aggregate query (one scan), budget-gated like any other. Status `fail` (exit code 4)
when a check fails, so it drops into CI or an orchestrator. Row-count history is kept per target, so
`--max-change` catches a load that suddenly lost half its rows.

## Scale lint rules

| Rule | Name | Severity |
|---|---|---|
| GV001 | select-star | warn |
| GV002 | unbounded-output | warn |
| GV003 | cartesian-join | error (no condition) / warn (CROSS JOIN) |
| GV004 | missing-partition-filter | error |
| GV005 | function-on-filter-column | warn |
| GV006 | order-without-limit | warn |
| GV007 | exact-count-distinct | info |
| GV008 | non-equi-join | warn |
| GV009 | union-distinct | info |
| GV010 | scalar-subquery-in-select | info |
| GV011 | window-without-partition | warn |
| GV012 | not-in-subquery | info |

`gulliver lint` works without any database.

## CLI reference

```
gulliver status                       connections, budget, spend today
gulliver inspect [TABLE]              biggest tables, or one table's columns
gulliver lint SQL [--dialect D]       scale lint only
gulliver preflight SQL                estimate + lint + gate, no execution
gulliver rehearse SQL [--fractions 0.01,0.03,0.1] [--sample largest|all] [--method auto|system|bernoulli]
gulliver run SQL [--max-rows N] [--allow-write]
gulliver receipt [ID|latest] [--format markdown|json]
gulliver approve ID [--note TEXT]     (human only)
gulliver demo [PATH]                  create a demo DuckDB database
gulliver lineage --models DIR | --manifest FILE
gulliver impact TABLE[.COLUMN] --models DIR | --manifest FILE
gulliver snapshot | drift [--models DIR] [--save]
gulliver diff --left SQL --right SQL --key k1,k2 [--right-connection NAME] [--columns a,b]
gulliver translate SQL --to D [--from D] [--check]
gulliver incremental SQL [--time-column C] [--unique-key K] [--period 1d] [--lookback 1]
gulliver check TARGET [--where W] [--unique K] [--not-null C] [--accepted C=a|b] [--range C=min:max] [--fresh C=24h] [--min-rows N] [--max-change PCT]
gulliver tools --format mcp|openai|gemini
gulliver mcp [--http]                 start the MCP server
```

Exit codes: `0` ok, `1` error, `2` blocked, `3` needs rehearsal or approval, `4` a data check failed.

SQL can be an argument, `-f file.sql`, or `-` for stdin. Global flags: `--dsn`, `--config`, `-c NAME`, `--json`.

## Development

```bash
git clone https://github.com/laaielichter/gulliver && cd gulliver
pip install -e ".[dev]"
pytest
```

See [AGENTS.md](AGENTS.md) for the agent-facing guide and [SECURITY.md](SECURITY.md) to report a vulnerability.

## License

MIT, see [LICENSE](LICENSE).
