Schema-first synthetic data for engineering teams

Synthetic enterprise data from schemas, SQL DDL, and generation plans

Great Generator helps data teams create realistic non-production datasets for development, QA, ETL testing, Spark pipelines, lakehouse demos, CDC, anomalies, relational models, and deterministic advisor-reviewed generation.

Great Generator creates synthetic data. It does not anonymize, mask, de-identify, or transform production records.

Get started View GitHub Read docs Install from PyPI

Schema-first Pandas and Spark SQL DDL CDC and anomalies Optional AI advisor Deterministic generation

Schema SQL DDL

from great_generator import generate_from_schema

schema = """
customer_id int,
customer_name string,
email string,
signup_date date,
account_status string,
balance decimal(12,2)
"""

df = generate_from_schema(schema=schema, rows=1000, seed=42)
print(df.head())

from great_generator import parse_ddl, generate_from_schema

ddl = """
CREATE TABLE sales.customers (
  customer_id BIGINT PRIMARY KEY,
  customer_name STRING NOT NULL,
  email VARCHAR(120) UNIQUE,
  signup_date DATE,
  balance DECIMAL(12,2)
)
"""

contract = parse_ddl(ddl, dialect="databricks")
df = generate_from_schema(contract, rows=1000, seed=42)

Install

Simple install. Optional extras when you need Spark, Delta, or advisors.

Install with a hyphen. Import with an underscore.

pip install great-generator
pip install "great-generator[spark]"
pip install "great-generator[delta]"
pip install "great-generator[ai]"
pip install "great-generator[anthropic]"
pip install "great-generator[ollama]"

import great_generator

from great_generator import generate_from_schema

df = generate_from_schema("id int, name string", rows=100)

What is new

The project has grown beyond simple schema samples.

Recent work adds query-aware generation, SQL DDL contracts, advisor artifacts, and stronger launch documentation while keeping the base package lightweight.

Current

Query-aware generation

Generate synthetic data that contains required filter values, partition dates, selectivity targets, and relational join paths for SQL and pipeline tests.

Contracts

SQL DDL ingestion

Parse the documented ANSI, Spark, and Databricks `CREATE TABLE` subset into canonical contracts with stable fingerprints and parser diagnostics.

Planning

AI advisor layer

Use optional design-time advisors for schema understanding, column tagging, and realism review. Advisors produce JSON artifacts; they do not generate row data.

Repeatability

Generation plans

Inspect, edit, save, and reuse `GenerationPlan` and `ColumnTags`. The same schema, plan, seed, and arguments are designed to produce repeatable output.

Data systems

Relational and lakehouse data

Generate parent-child tables, CDC records, anomalies, SCD2 history, dimensional models, Data Vault-style examples, and Spark/Delta outputs.

Core workflows

Pick the API that matches your task.

1. Generate from schema

df = generate_from_schema(schema, rows=1000)

Lower-environment data
QA and ETL testing
Analytics prototypes
API contract testing

2. Generate related tables

data = generate_relational(
    tables={
        "customers": {"schema": "customer_id int primary key, customer_name string", "rows": 1000},
        "orders": {"schema": "order_id int primary key, customer_id int references customers.customer_id", "rows": 5000},
    }
)

Joins and referential integrity
Warehouse and lakehouse demos
Dimensional modeling
Integration tests

3. Parse SQL DDL

contract = parse_ddl(ddl, dialect="databricks")
print(contract.fingerprint())

Database contracts
Warehouse schemas
Lakehouse table definitions
Repeatable schema parsing

4. Use an optional advisor

plan = infer_generation_plan(
    "customer_id int, customer_name string, email string",
    advisor="none"
)

df = generate_from_schema(
    "customer_id int, customer_name string, email string",
    rows=100,
    plan=plan,
    seed=42,
)

Design-time review
Column tagging
Realism review
Auditable generation plans

`advisor="none"` is the default and does not call a model.

Feature set

Practical capabilities for test data, demos, and engineering workflows.

Schema-first generation

Generate synthetic DataFrames from mappings, DDL strings, DataFrames, Spark schemas, `TableSchema`, and domain schemas.

SQL DDL parsing

Use `parse_ddl(...)` for documented SQL `CREATE TABLE` contracts with stable fingerprints.

Semantic field detection

Recognize names, IDs, contacts, dates, amounts, statuses, quantities, and lifecycle fields from column names.

Query-aware generation

Include required filter values, partition dates, selectivity targets, and join paths when your tests need them.

Relational generation

Create parent-child tables with primary keys, foreign keys, and valid references by default.

CDC simulation

Generate insert, update, and delete style records with event timestamps and ingestion timestamps.

Anomaly injection

Add opt-in nulls, duplicates, orphan keys, late records, outliers, invalid statuses, and negative amounts for data quality tests.

SCD2 history

Create slowly changing dimension history tables for supported Pandas domain workflows.

Dimensional models

Generate facts and dimensions for analytics engineering, BI examples, and warehouse modeling demos.

Data Vault models

Create hubs, links, and satellites for architecture examples and modeling experiments.

Pandas and Spark engines

Use Pandas locally or Spark in runtimes such as Databricks, Fabric Spark, EMR, Glue, and Synapse Spark.

Exports

Use convenience exports for CSV, JSON, Parquet, and Delta, or write returned DataFrames with native APIs.

Recipes and CLI

Package repeatable generation scenarios as JSON, TOML, or simple YAML recipes and run them from the command line.

Advisor artifacts

Create optional `GenerationPlan`, `ColumnTags`, and realism review artifacts before deterministic generation.

Manifest metadata

Record generation parameters, tables, validation checks, schema fingerprints, and optional advisor contribution.

Schema inputs

Start from the contract you already have.

Input type	Example	Best for	Status
Plain Python mapping	`{"name": "string", "age": "int"}`	Fast schema-based generation	Supported
Rich mapping and custom rules	`custom_rules={"age": {"min": 18}}`	Business ranges, categories, patterns, and null rates	Supported through `custom_rules`
Pandas dtype mapping	`df.dtypes.to_dict()`	Pandas and notebook workflows	Supported
Pandas DataFrame schema	Empty or populated `DataFrame`	Preserve Pandas column names and dtypes	Supported
Compact DDL string	`"id int, name string"`	SQL-like quick starts	Supported
Full SQL `CREATE TABLE` DDL	`parse_ddl(ddl, dialect="databricks")`	Contracts from databases, warehouses, Spark, and lakehouse tables	Supported for documented subset
PySpark `StructType`	`StructType([...])`	Spark notebook schemas	Supported
PySpark DataFrame	Empty or populated Spark DataFrame	Use an existing Spark schema	Supported
`TableSchema`	`TableSchema(name="customers", ...)`	Library-native schema metadata	Supported
`DomainSchema`	Multi-table schema metadata	Relational and domain-shaped datasets	Supported
JSON Schema	`schema.json`	Application contracts	Planned
YAML schema profile	`schema.yml`	Reusable schema configs	Planned
Pydantic model	`BaseModel`	API model contracts	Planned
Dataclass	`@dataclass`	Python application models	Planned

Query-aware generation

Generate data that your SQL query can actually match.

Query-aware generation creates synthetic data that contains the required values, partition dates, and join paths your SQL queries expect.

All query-aware options are optional. Existing behavior is unchanged unless users provide `required_values`, `partition_by`, `target_selectivity`, `ensure_join_coverage`, or `query_profile`.

Query-aware generation helps synthetic data match expected query values, partition dates, and join paths. It does not guarantee identical production performance because file layout, table statistics, clustering, caching, concurrency, warehouse size, and query engine configuration also affect runtime.

Current query-aware shaping is implemented for Pandas generation paths. Spark-native query-aware generation is planned; Spark users can continue using existing Spark generation paths and normal Spark writers.

df = generate_from_schema(
    schema=schema,
    rows=100000,
    required_values={
        "region": ["SOUTH"],
        "product_type": ["CHECKING", "SAVINGS"],
    },
    partition_by={
        "column": "business_date",
        "values": ["2026-01-01", "2026-01-02"],
        "distribution": "balanced",
    },
    target_selectivity={"region": {"SOUTH": 0.25}},
)

SQL DDL and contracts

Contract-first schema generation from documented SQL DDL.

Use `parse_ddl(...)` to parse documented SQL `CREATE TABLE` DDL into canonical contracts with stable fingerprints, table names, column order, normalized types, keys, relationships, constraints, comments, defaults, and selected Spark/Databricks metadata where supported.

The parser supports a documented subset of ANSI, Spark, and Databricks DDL. Unsupported syntax should produce clear diagnostics rather than silent assumptions.

from great_generator import parse_ddl, generate_from_schema

contract = parse_ddl(ddl, dialect="databricks", strict=True)
print(contract.fingerprint())

df = generate_from_schema(contract, rows=1000, seed=42)

Optional AI advisor layer

Design-time advice, deterministic row generation.

The advisor layer can propose a generation plan, tag columns, and review a sample for realism. Advisors do not generate row data. Generation remains deterministic because the generator consumes inspectable JSON artifacts, not model output at row-generation time.

schemaadvisorplanreviewgenerationdata

Advisor	Status	Network	Notes
none	default	none	No API key, no model call.
Anthropic	optional	online	Requires advisor extra and API key.
Ollama	optional	local	Supports offline local model use.
OpenAI	stub/planned	online	Reserved interface; not listed as active advisor support.
llama.cpp	stub/planned	local	Reserved interface; not listed as active advisor support.

Relational and lakehouse workflows

Generate connected tables, history, CDC, and modeling examples.

Great Generator supports parent-child tables, primary keys and foreign keys, fact and dimension examples, CDC simulation, anomaly injection, SCD2 history, dimensional models, Data Vault-style examples, and Spark/Delta output.

Returned DataFrames stay in your control, so you can write to files, catalogs, databases, or cloud storage through native Pandas and Spark APIs.

data = generate_relational(tables=tables, seed=42)

customers = data["customers"]
orders = data["orders"]

data = generate_domain("banking", history="scd2")
cdc = generate_cdc("banking", table="customers", rows=1000)

Spark, Delta, and cloud paths

Use the same package locally and in Spark notebooks.

Generate Pandas or Spark DataFrames and write through normal runtime APIs. Pandas can write local CSV, JSON, and Parquet. Spark can write to DBFS, ADLS, S3, GCS, HDFS, mounted paths, Parquet, Delta, and configured database connectors.

df.to_parquet("customers.parquet", index=False)

spark_df.write.mode("overwrite").parquet("s3://bucket/demo/customers")
spark_df.write.format("delta").mode("overwrite").save("dbfs:/tmp/demo_delta")

Trust, safety, and determinism

Repeatable generated data without production-record transformation.

Great Generator separates design-time advice from row generation. Advisor outputs are saved as JSON artifacts. The generator consumes those artifacts deterministically. This keeps test data repeatable and makes plan review possible before data is generated.

Deterministic path

Same schema, plan, seed, and arguments should produce the same output for repeatable tests and demos.

Offline default

`advisor="none"` is the default. It does not call a model, read API keys, or require network access.

JSON artifacts

Plans, tags, reports, and manifests can be reviewed, edited, saved, and committed with your project.

Synthetic boundary

The library creates synthetic data. It does not anonymize, mask, de-identify, or transform production records.

Documentation

Guides for quick starts, implementation, demos, and releases.

QuickstartInstall, import, generate, inspect, and export. Schema InputsCurrent support matrix, examples, and limitations. SQL DDLCanonical contracts, parser diagnostics, and DDL examples. Query-Aware GenerationRequired values, partitions, selectivity, join coverage, and coverage reports. API ReferenceMain public APIs and usage patterns. AI AdvisorsOptional schema planning, column tagging, caching, and offline Ollama usage. Generation Plans`GenerationPlan`, `ColumnTags`, and realism report artifacts. CDCChange data capture style event records. AnomaliesOpt-in data problems for quality testing. Spark and DeltaCluster usage and lakehouse outputs. Cloud DeploymentAWS, Azure, GCP, Databricks, and storage guidance. Recipes and CLIReusable generation parameters and command-line workflows. Domain PacksSupported industry domains and table layouts. Demo GuideSpeaker-friendly story, examples, and demo flow.

Release highlights

Recent project direction.

Version	Focus	Highlights
0.1.7	SQL DDL ingestion and query-aware generation	`parse_ddl(...)`, canonical contracts, parser diagnostics, documented ANSI/Spark/Databricks subset, required values, partitions, selectivity, and join coverage.
0.1.6	AI advisor planning layer	Advisors, `GenerationPlan`, `ColumnTags`, cached advisor calls, manifest metadata, and `plan=` support.
0.1.5	Schema-first docs and Spark writes	Schema input matrix, Databricks examples, Snowflake writes, Azure SQL writes, and documentation site updates.
0.1.1	Advanced APIs	Anomaly labels, SCD2 history, recipes, CLI, dimensional models, and Data Vault models.
0.1.0	Initial release	Domain packs, Pandas/Spark engines, exports, CDC, anomalies, schema generation, and relational generation.

Roadmap

Planned areas are labeled clearly.

Infer query profiles from SQL text JSON Schema support YAML schema profiles Pydantic and dataclass schema adapters Spark-native query-aware generation More Spark and lakehouse examples Quality-tool examples for Great Expectations and Pandera More domain packs and reference values

Project

Created and maintained by Ravi Kiran Pagidi.

Great Generator is MIT licensed and maintained by Ravi Kiran Pagidi.

GitHub repository PyPI package GitHub Wiki Issues License