Metadata-Version: 2.4
Name: gendantic
Version: 0.1.0
Summary: Generate synthetic data using Pydantic models and LLMs
Author-email: Benjamin Rowell <ben.rowell@ai.gov.uk>
License: MIT License
        
        Copyright (c) 2026 Benjamin Rowell
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://github.com/benjaminr/gendantic
Project-URL: Repository, https://github.com/benjaminr/gendantic
Project-URL: Documentation, https://github.com/benjaminr/gendantic#readme
Project-URL: Changelog, https://github.com/benjaminr/gendantic/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/benjaminr/gendantic/issues
Keywords: synthetic-data,pydantic,llm,statistics,distributions,copula,data-generation
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Software Development :: Testing
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pydantic[email]>=2.5.0
Requires-Dist: litellm>=1.40.0
Requires-Dist: python-dotenv>=1.1.1
Requires-Dist: numpy>=1.24.0
Requires-Dist: scipy>=1.10.0
Provides-Extra: notebooks
Requires-Dist: jupyter>=1.0.0; extra == "notebooks"
Requires-Dist: ipykernel>=6.0.0; extra == "notebooks"
Requires-Dist: matplotlib>=3.7.0; extra == "notebooks"
Provides-Extra: pandas
Requires-Dist: pandas>=2.0.0; extra == "pandas"
Provides-Extra: db
Requires-Dist: sqlalchemy>=2.0.0; extra == "db"
Requires-Dist: psycopg[binary]>=3.1.0; extra == "db"
Dynamic: license-file

# Gendantic

Intelligent synthetic data generation using Pydantic models and LLMs, with statistical distribution support.

## Quick Start

```python
import asyncio
from typing import Annotated
from pydantic import BaseModel, Field
from gendantic import generate_synthetic_data, Normal, Uniform, Categorical

class Employee(BaseModel):
    name: str  # LLM generates realistic names
    age: Annotated[int, Uniform(min=22, max=65)]  # Sampled from uniform distribution
    salary: Annotated[float, Normal(mean=75000, std=20000)]  # Sampled from normal distribution
    department: Annotated[str, Categorical(weights={
        "Engineering": 0.4,
        "Sales": 0.3,
        "HR": 0.15,
        "Marketing": 0.15
    })]

async def main():
    employees = await generate_synthetic_data(Employee, count=100, seed=42)
    for emp in employees[:5]:
        print(f"{emp.name} ({emp.age}) - {emp.department}: £{emp.salary:,.0f}")

asyncio.run(main())
```

### Synchronous usage

If you're not managing an event loop (scripts, REPLs, notebooks outside async
cells), use the `*_sync` wrappers:

```python
from gendantic import generate_synthetic_data_sync

employees = generate_synthetic_data_sync(Employee, count=100, seed=42)
```

### Exporting to a DataFrame

```python
from gendantic import generate_synthetic_data_sync, to_dataframe

employees = generate_synthetic_data_sync(Employee, count=100, seed=42)
df = to_dataframe(employees)  # requires the `pandas` extra
```

## Features

- **Statistical Distributions**: Use `Annotated` types with numpy-backed distributions for guaranteed statistical compliance
- **Correlated Fields**: Model realistic relationships between fields using copulas (Gaussian, Student's t, Clayton, Gumbel, Frank)
- **Conditional Distributions & Constraints**: Switch a field's distribution on another field's value with `Conditional` (categorical or numeric-threshold discriminators), and enforce cross-field ordering with `Constraints`/`Ordering`
- **Relational Generation**: Generate multiple related models with referential integrity using `PrimaryKey`/`ForeignKey` and `generate_dataset()`, including composite keys and join tables
- **Database Binding**: Reflect an existing database schema into models, generate synthetic data, and load it back with `gendantic.db` (`reflect_schema`, `load_dataset`, `infer_distributions`)
- **LLM-Driven Semantic Fields**: Names, descriptions, and text are generated by LLMs with context awareness
- **Dynamic Model Generation**: Let the LLM design your Pydantic model from a natural language description
- **Smart Distribution & Correlation Suggestions**: Use `extend_model_with_distributions()` and `extend_model_with_correlations()` to let the LLM suggest appropriate distributions, correlations, and copula types
- **Fidelity Validation**: Statistically verify that generated data matches its declared spec with `fidelity_report()` (KS / chi-square goodness-of-fit and correlation checks)
- **Reproducible**: Set a seed for deterministic distribution sampling
- **Async-First (with sync wrappers)**: High-performance async API with batch generation, plus `*_sync` helpers for scripts and REPLs
- **DataFrame Export**: Turn generated records into a pandas DataFrame with `to_dataframe()`
- **Provider-Agnostic**: Connects through a [LiteLLM](https://docs.litellm.ai/) proxy, giving access to any model LiteLLM supports (OpenAI, Anthropic, Azure, Bedrock, local models, and more)

## Installation

```bash
pip install gendantic
```

Or with uv:

```bash
uv add gendantic
```

For DataFrame export, install the optional `pandas` extra:

```bash
pip install 'gendantic[pandas]'
```

For the database binding (schema reflection and loading), install the `db` extra:

```bash
pip install 'gendantic[db]'
```

## Statistical Distributions

Gendantic separates concerns: **numpy samples numeric/categorical fields**, while **LLMs generate semantic content**. This gives you statistical guarantees where you need them and realistic text where it matters.

### Available Distributions

```python
from typing import Annotated
from gendantic import (
    Normal,      # Bell curve: salaries, scores, measurements
    Uniform,     # Even spread: ages, random IDs
    Categorical, # Weighted categories: departments, statuses
    LogNormal,   # Right-skewed: incomes, file sizes
    Exponential, # Wait times, decay processes
    Poisson,     # Count data: number of events
    Beta,        # Probabilities, percentages (0-1)
    Binomial,    # Success counts from n trials
)

class SalesData(BaseModel):
    # Numeric distributions
    revenue: Annotated[float, LogNormal(mean=10, sigma=1)]
    deals_closed: Annotated[int, Poisson(lam=5)]
    conversion_rate: Annotated[float, Beta(alpha=2, beta=5)]

    # Categorical distribution
    region: Annotated[str, Categorical(weights={
        "North": 0.3, "South": 0.25, "East": 0.25, "West": 0.2
    })]

    # LLM-generated (no distribution annotation)
    sales_rep_name: str
    deal_notes: str
```

### Combining Distributions with Field Constraints

Distributions work alongside Pydantic's `Field()` constraints:

```python
class BoundedEmployee(BaseModel):
    # Normal distribution, restricted to a valid range
    salary: Annotated[int, Normal(mean=50000, std=15000)] = Field(ge=30000, le=200000)

    # Uniform within specific bounds
    age: Annotated[int, Uniform(min=18, max=65)] = Field(ge=18, le=100)
```

`ge` / `le` / `gt` / `lt` bounds **truncate** the distribution: values are drawn from the conditional distribution *on the allowed interval* (inverse-CDF truncation), not by clamping out-of-range draws onto the boundary. Clamping would pile a spike of probability mass on the bound — e.g. a `Normal(mean=0)` with `ge=0` would land ~half its draws exactly on `0` — and make the field disagree with its declared shape. Truncation keeps the boundary at its true (near-zero) density, so the field remains a genuine truncated Normal. `fidelity_report()` is truncation-aware and compares against the truncated distribution, so bounded fields still pass. For integer fields the strict/inclusive distinction is snapped onto the integer support (`gt=a` → `≥ a+1`, `ge=a` → `≥ a`; `lt=b` → `≤ b-1`, `le=b` → `≤ b`).

### Reproducibility with Seeds

```python
# Same seed = same distribution samples
batch1 = await generate_synthetic_data(Employee, count=100, seed=42)
batch2 = await generate_synthetic_data(Employee, count=100, seed=42)

# Distribution-sampled fields will be identical
assert batch1[0].salary == batch2[0].salary
assert batch1[0].age == batch2[0].age
```

## Correlated Fields

Real-world data has correlations: older employees tend to have more experience, higher performers often get larger bonuses. Gendantic uses **copulas** to model these relationships while preserving each field's marginal distribution.

### Basic Correlations

```python
from gendantic import generate_synthetic_data, Correlations, Normal, Uniform

class Employee(BaseModel):
    age: Annotated[int, Uniform(min=22, max=65)]
    years_experience: Annotated[int, Uniform(min=0, max=40)]
    salary: Annotated[float, Normal(mean=75000, std=20000)]
    name: str

    # Define correlations between distribution fields
    __correlations__ = Correlations(
        ("age", "years_experience", 0.85),  # Strong positive: older = more experience
        ("years_experience", "salary", 0.6),  # Moderate: experience → higher salary
        ("age", "salary", 0.4),  # Weaker direct correlation
    )

employees = await generate_synthetic_data(Employee, count=1000, seed=42)

# Correlations are preserved in the generated data
import numpy as np
ages = [e.age for e in employees]
experience = [e.years_experience for e in employees]
print(f"Age-Experience correlation: {np.corrcoef(ages, experience)[0,1]:.2f}")
# Output: Age-Experience correlation: 0.84
```

### Copula Types

Different copula types model different dependency patterns:

```python
from gendantic import Correlations, CopulaType

class FinancialData(BaseModel):
    stock_return: Annotated[float, Normal(mean=0.07, std=0.15)]
    bond_return: Annotated[float, Normal(mean=0.03, std=0.05)]
    performance_score: Annotated[float, Beta(alpha=5, beta=2)]
    bonus_pct: Annotated[float, Normal(mean=10, std=5)]
    risk_score: Annotated[float, Beta(alpha=2, beta=5)]
    loss_severity: Annotated[float, LogNormal(mean=8, sigma=1)]

    __correlations__ = Correlations(
        # Student's t: heavy tails, extreme values occur together (financial crises)
        ("stock_return", "bond_return", 0.5, "student_t"),

        # Gumbel: upper tail dependence - things boom together
        ("performance_score", "bonus_pct", 0.7, "gumbel"),

        # Clayton: lower tail dependence - things crash together
        ("risk_score", "loss_severity", 0.6, "clayton"),

        # Frank: symmetric, no tail dependence (weak correlations, sign allowed)
        ("stock_return", "risk_score", -0.2, "frank"),
    )
```

**Copula Types:**

| Copula | Tail Dependence | `corr` means | Sign | Use Case |
|--------|-----------------|--------------|------|----------|
| `gaussian` | None | latent correlation | ±  | Standard correlations, most business data |
| `student_t` | Both tails | latent correlation | ±  | Financial data, extreme events together |
| `clayton` | Lower tail | Kendall's τ | + only | Risk modelling, crashes happen together |
| `gumbel` | Upper tail | Kendall's τ | + only | Success metrics, booms happen together |
| `frank` | None | Kendall's τ | ±  | Weak to moderate symmetric correlations |

For the Gaussian and Student-t families `corr` is the latent (Pearson-of-the-copula) correlation; for the Archimedean families (Clayton, Gumbel, Frank) it is the target **Kendall's τ**. Clayton and Gumbel model positive dependence only — a negative `corr` for either raises `ValueError` (use `gaussian` or `frank` for negative relationships).

**How per-pair copulas compose.** Each pair keeps its own family and strength — mixed families no longer collapse to a single joint copula. Internally the pairs form a [vine](https://en.wikipedia.org/wiki/Vine_copula) (a 1-truncated R-vine, equivalent to a Markov tree): the specified pairs are the tree edges, and two fields not joined by an edge are conditionally independent given the path between them. This means:

- The pairs must form a **forest** — each pair must connect two fields not already linked. A set of pairs that closes a cycle (e.g. `a–b`, `b–c`, `a–c`) raises `ValueError`, because a 1-truncated vine cannot place the third pair without conditional-copula parameters the spec does not provide.
- A pair may be specified **at most once**; a repeated pair (in either order) raises `ValueError`.
- Homogeneous all-Gaussian or all-Student-t specs still use an exact full correlation matrix (no forest restriction beyond a valid matrix); the vine path is used whenever families are mixed or any Archimedean family is present.

### LLM-Suggested Distributions with `extend_model_with_distributions()`

Have a plain Pydantic model without distributions? Let the LLM add them:

```python
from gendantic import extend_model_with_distributions

# Start with a basic model
class Employee(BaseModel):
    """Employee record for HR analytics."""
    name: str
    age: int
    salary: float
    department: str

# LLM analyses field semantics and suggests appropriate distributions
DistributedEmployee, code = await extend_model_with_distributions(Employee)

print(code)
# class Employee(BaseModel):
#     """Employee record for HR analytics."""
#     name: str
#     age: Annotated[int, Uniform(min=22, max=65)]
#     salary: Annotated[float, Normal(mean=75000, std=20000)]
#     department: Annotated[str, Categorical(weights={"Engineering": 0.4, "Sales": 0.3, "HR": 0.3})]

# Now generate with statistical guarantees
employees = await generate_synthetic_data(DistributedEmployee, count=100)
```

### LLM-Suggested Correlations with `extend_model_with_correlations()`

Let the LLM suggest appropriate correlations and copulas for your model:

```python
from gendantic import extend_model_with_correlations, Normal, Uniform, Beta

class Employee(BaseModel):
    """Employee record for HR analytics."""
    age: Annotated[int, Uniform(min=22, max=65)]
    years_experience: Annotated[int, Uniform(min=0, max=40)]
    salary: Annotated[float, Normal(mean=75000, std=20000)]
    performance_score: Annotated[float, Beta(alpha=5, beta=2)]
    name: str

# LLM analyses field semantics and suggests correlations
ExtendedEmployee, code = await extend_model_with_correlations(Employee)

print(code)
# class Employee(BaseModel):
#     ...
#     __correlations__ = Correlations(
#         ("age", "years_experience", 0.85),
#         ("years_experience", "salary", 0.6),
#         ("performance_score", "salary", 0.5, "gumbel"),
#     )

# Now generate with realistic correlations
employees = await generate_synthetic_data(ExtendedEmployee, count=100)
```

## Conditional Distributions and Constraints

Correlations capture linear-ish co-movement, but some relationships are conditional: a field's *distribution* depends on the value of another field, or several fields must always come out in a fixed order. gendantic expresses both without any LLM call.

### Conditional distributions

Annotate a field with `Conditional` to switch its distribution based on another field's sampled value. The discriminator (`on`) must be another distribution-sampled field. Any value not matched by a case falls through to `default`.

```python
from typing import Annotated
from pydantic import BaseModel
from gendantic import (
    generate_synthetic_data, Conditional, Categorical, Normal,
)

class Employee(BaseModel):
    department: Annotated[str, Categorical({"Eng": 0.5, "Sales": 0.3, "HR": 0.2})]
    salary: Annotated[
        float,
        Conditional(
            on="department",
            cases={
                "Eng": Normal(mean=90000, std=15000),
                "Sales": Normal(mean=70000, std=20000),
            },
            default=Normal(mean=50000, std=10000),  # HR and anything else
        ),
    ]

# Eng salaries centre on 90k, Sales on 70k, everyone else on 50k
employees = await generate_synthetic_data(Employee, count=100, seed=42)
```

Discriminators can also be **numeric thresholds** using `Range(min, max)` (half-open `[min, max)`; either bound may be omitted for open-ended bins):

```python
from gendantic import Conditional, Range, Uniform, Normal

class Customer(BaseModel):
    age: Annotated[int, Uniform(min=18, max=80)]
    annual_spend: Annotated[
        float,
        Conditional(
            on="age",
            cases={
                Range(max=30): Normal(mean=3000, std=500),      # age < 30
                Range(30, 50): Normal(mean=6000, std=800),      # 30 <= age < 50
                Range(min=50): Normal(mean=9000, std=1000),     # age >= 50
            },
            default=Normal(mean=5000, std=1000),
        ),
    ]
```

Numeric bins are matched on the *converted* value the record exposes (after int rounding and constraint clipping), so a bin boundary like `30` behaves exactly as it reads. Conditionals may depend on other conditional fields — gendantic resolves them in dependency order and raises on cycles or self-references.

### Cross-field ordering constraints

Declare an `Ordering` inside a `__constraints__` attribute to guarantee two or more fields always come out sorted ascending per record (ties allowed). This is enforced by sorting each row's values across the constrained fields after sampling.

Because the sort reassigns which field receives which value, the *combined* pool of values is preserved but each constrained field's individual marginal shifts to an order statistic — the first field becomes the row-wise minimum, the last the row-wise maximum. This is the deliberate trade-off for a hard ordering guarantee: use it when the invariant matters more than the exact per-field marginals (e.g. `start <= end` dates).

```python
from gendantic import Constraints, Ordering, Uniform

class Booking(BaseModel):
    check_in: Annotated[float, Uniform(min=0, max=365)]
    check_out: Annotated[float, Uniform(min=0, max=365)]

    __constraints__ = Constraints(
        Ordering("check_in", "check_out"),  # check_in <= check_out, always
    )
```

`Ordering` accepts more than two fields (`Ordering("start", "review", "end")`) and each field must be distribution-sampled.

#### Preserving marginals with `method="resample"`

The default `method="sort"` always succeeds but reshapes marginals into order statistics (above). When your fields have **different, already-mostly-separated marginals** — e.g. `birth < hire < termination` — and you want each field to keep its own distribution, use `method="resample"`. It keeps each field's sampled value and redraws only the records that violate the order, repeating until the whole batch complies:

```python
from gendantic import Constraints, Ordering, Uniform

class Career(BaseModel):
    birth: Annotated[float, Uniform(min=0, max=30)]
    hire: Annotated[float, Uniform(min=30, max=60)]
    termination: Annotated[float, Uniform(min=60, max=100)]

    __constraints__ = Constraints(
        Ordering("birth", "hire", "termination", method="resample"),
    )
```

Trade-offs and limits:

- A hard order and *identical* marginals are mathematically incompatible (if `a <= b` always and both share a distribution, then `a = b`), so resample only preserves marginals when the marginals are compatible with the order. For fully overlapping marginals it degenerates to the same distortion as sorting.
- If the marginals overlap so heavily that the rejection budget is exhausted, generation **raises** rather than return silently distorted data — separate the marginals or switch to `method="sort"`.
- Resample fields must be **independent plain distributions** — a field that is also correlated (`__correlations__`) or conditional (`Conditional`) raises, since an independent redraw would break that structure.
- The order is checked on the *converted* values (after int rounding / clipping), so the guarantee holds for the values the record actually exposes.

## Fidelity Validation

gendantic samples distribution-annotated fields to match their spec. `fidelity_report()` lets you *verify* that promise: it compares a batch of generated records against the model's declared distributions and correlations and returns a structured, printable report. It never raises — you inspect the result and decide what to do.

```python
from gendantic import fidelity_report

# records can be model instances or plain dicts
report = fidelity_report(records, Employee)

print(report)            # human-readable table
assert report.passed     # True when every field and correlation passed
```

```
Fidelity report (2000 records) - PASS

Fields:
  [ok ] salary (normal): ks=0.0137 p=0.8408 mean obs=5.013e+04/exp=5e+04
  [ok ] age (uniform): ks=0.0215 p=0.3092 mean obs=41.99/exp=41.5
  [ok ] dept (categorical): chi2=0.7729 p=0.6795
  [ok ] errors (poisson): chi2=14.6974 p=0.0996 mean obs=2.938/exp=3.001

Correlations:
  [ok ] age~salary: target=+0.60 spearman=+0.60 pearson=+0.60 err=0.00
```

**What gets checked** (only distribution-annotated fields and declared `__correlations__`; free-text LLM fields are ignored):

- **Continuous** (`Normal`, `Uniform`, `LogNormal`, `Exponential`, `Beta`) — Kolmogorov-Smirnov test against the theoretical CDF.
- **Discrete counts** (`Poisson`, `Binomial`) — chi-square goodness-of-fit on the count histogram.
- **Categorical** — chi-square goodness-of-fit on category frequencies.
- **Conditional** — checked *per case branch*: records are grouped by which case matched (on the discriminator value stored in the record) and each group is tested against its own case spec, yielding one result per branch (labelled e.g. `salary | department='Eng'`).
- **Correlations** — the observed rank statistic the copula family targets vs. the declared target: Kendall's τ for the Archimedean families (Clayton, Gumbel, Frank), Spearman's ρ for Gaussian/Student-t. Both rank statistics plus Pearson are reported alongside (`corr.basis` says which one drives the verdict).

**Tuning the verdict:**

```python
report = fidelity_report(
    records,
    Employee,
    alpha=0.05,                  # significance level: field passes when p-value >= alpha
    correlation_tolerance=0.15,  # max |observed rank stat - target| for a pair to pass
)

for field in report.fields:
    print(field.field, field.test, field.p_value, field.passed)
for corr in report.correlations:
    print(corr.field1, corr.field2, corr.error, corr.passed)
```

Because `DistributionSampler` produces spec-compliant data deterministically from a seed, fidelity checks are LLM-free and reproducible — useful in tests and CI to catch regressions in the sampling machinery.

## Dynamic Model Generation

Don't want to define a model? Let the LLM create one from a description:

```python
from gendantic import generate_model_from_description, generate_synthetic_data

async def main():
    # Step 1: Generate the model from a description
    Model, source_code = await generate_model_from_description(
        "A customer support ticket with priority, category, "
        "customer sentiment score, and a description of the issue"
    )

    # See what the LLM created
    print(source_code)
    # class SupportTicket(BaseModel):
    #     """A customer support ticket."""
    #     priority: Annotated[str, Categorical(weights={"High": 0.2, "Medium": 0.5, "Low": 0.3})]
    #     category: Annotated[str, Categorical(weights={"Billing": 0.3, "Technical": 0.4, "General": 0.3})]
    #     sentiment_score: Annotated[float, Uniform(min=0.0, max=1.0)]
    #     description: str

    # Step 2: Generate data using the model
    tickets = await generate_synthetic_data(Model, count=50)

asyncio.run(main())
```

The LLM intelligently decides which fields should use statistical distributions (numeric, categorical) and which should be LLM-generated (text, descriptions).

### Security

Generated code is validated via AST parsing before execution. The following are blocked:
- Import statements
- `exec`, `eval`, `compile`, `open`
- `globals()`, `locals()`, `__builtins__`
- Dunder attribute access (`__class__`, `__bases__`, etc.)

## Context-Aware Generation

Provide business context for more realistic LLM-generated fields:

```python
# Different contexts produce different realistic patterns
startup_employees = await generate_synthetic_data(
    Employee,
    count=20,
    context="Fast-growing Silicon Valley AI startup"
)

bank_employees = await generate_synthetic_data(
    Employee,
    count=20,
    context="Traditional London investment bank"
)
```

## Batch Generation

Generate data for multiple contexts concurrently:

```python
from gendantic import generate_synthetic_data_batch

contexts = [
    "Tech startup in San Francisco",
    "Manufacturing company in Detroit",
    "Consulting firm in New York"
]

batches = await generate_synthetic_data_batch(Employee, contexts, count=10, seed=42)
# Returns 3 lists of 10 employees each, with consistent distribution sampling
```

## Relational Generation

Generate several related models together with **referential integrity**: mark
primary keys with `PrimaryKey` and references with `ForeignKey`, then call
`generate_dataset`. Models are generated parent-first (topologically sorted by
their foreign keys), primary keys are unique, and every foreign key points at a
real row of the referenced model.

```python
from typing import Annotated
from pydantic import BaseModel
from gendantic import PrimaryKey, ForeignKey, Normal, Categorical, generate_dataset

class Customer(BaseModel):
    id: Annotated[int, PrimaryKey()]                 # unique, engine-generated
    name: str                                         # LLM-generated
    tier: Annotated[str, Categorical(weights={"free": 0.6, "pro": 0.3, "vip": 0.1})]

class Order(BaseModel):
    id: Annotated[str, PrimaryKey(strategy="uuid")]
    customer_id: Annotated[int, ForeignKey(Customer)] # a real Customer.id
    amount: Annotated[float, Normal(mean=200, std=80)]

dataset = await generate_dataset({Customer: 100, Order: 500}, seed=42)

customers = dataset[Customer]   # 100 rows, unique ids
orders = dataset[Order]         # 500 rows, each customer_id ∈ customers' ids
assert all(o.customer_id in {c.id for c in customers} for o in orders)
```

- **Primary keys**: `PrimaryKey(strategy=...)` — `"auto"` (default: sequential
  ints for `int` fields, UUID hex for `str`), `"sequential"`, or `"uuid"`.
- **Foreign keys**: `ForeignKey(Model, field=None, nullable=False, null_probability=0.1)`
  — each row is assigned a random primary key from the referenced model. `Model`
  may be a class or its name as a string.
- **Self-references** (e.g. `manager_id` → same model) use a string forward
  reference, since the class isn't defined inside its own body:
  `manager_id: Annotated[int | None, ForeignKey("Employee", nullable=True)] = None`.
- **Composite keys**: declare a multi-column primary key with a
  `__primary_key__ = ("col_a", "col_b")` class attribute, and multi-column
  foreign keys with `ForeignKeySpec` in a `__foreign_keys__` list. This covers
  join tables whose primary key *is* its foreign keys — gendantic generates
  distinct combinations so the composite key stays unique:
  ```python
  class OrderItem(BaseModel):
      order_id: int
      product_id: int
      quantity: Annotated[int, Normal(mean=2, std=1)]
      __primary_key__ = ("order_id", "product_id")
      __foreign_keys__ = [
          ForeignKeySpec(columns="order_id", model="Order"),
          ForeignKeySpec(columns="product_id", model="Product"),
      ]
  ```
- **Relational context**: when a child's foreign key points at a parent, the
  parent row's attributes are passed to the LLM as context so generated text is
  consistent with the referenced row (not just an opaque key).
- Use `generate_dataset_sync(...)` outside an event loop, and
  `dataset.to_dataframes()` to get a `{model_name: DataFrame}` mapping.

A fully runnable, no-proxy example (all non-key fields use distributions, so no
LLM call is made) lives in [`examples/relational_quickstart.py`](examples/relational_quickstart.py):

```bash
uv run python examples/relational_quickstart.py
```

For a version that actually calls an LLM for the semantic fields (customer
names, product names, review text) while keeping referential integrity, see
[`examples/relational_llm.py`](examples/relational_llm.py). It needs a LiteLLM
proxy — set the standard environment variables first:

```bash
export LITELLM_API_BASE="https://your-litellm-proxy/v1"
export LITELLM_API_KEY="your-proxy-key"   # if the proxy requires auth
export LITELLM_MODEL="openai/gpt-4o-mini"
uv run python examples/relational_llm.py
```

### More examples

The [`examples/`](examples/) directory has runnable scripts for the main
patterns:

| Example | LLM? | Shows |
|---------|------|-------|
| [`overview.py`](examples/overview.py) | yes | End-to-end tour: distributions, LLM-generated semantic fields, dynamic model generation from a description, seeded reproducibility, and context-aware generation |
| [`llm_single_model.py`](examples/llm_single_model.py) | yes | Single model: a minimal text-only case, plus distributions correlated with a copula |
| [`conditional.py`](examples/conditional.py) | no | Conditional distributions (categorical & numeric `Range` bins), chained dependencies, and cross-field `Ordering` constraints |
| [`relational_quickstart.py`](examples/relational_quickstart.py) | no | Basic three-table relational generation with referential integrity |
| [`relational_hierarchy.py`](examples/relational_hierarchy.py) | no | Self-references (string forward ref), nullable foreign keys, `to_dataframes()` |
| [`relational_llm.py`](examples/relational_llm.py) | yes | Relational generation where the LLM writes semantic fields coherent with related rows |
| [`db_binding.py`](examples/db_binding.py) | yes | Reflect a live database schema, generate synthetic rows, and load them back |

## Database Binding

Point gendantic at an existing database and it will reflect the schema into
annotated Pydantic models, generate synthetic rows that respect the schema's
keys and constraints, and load them back — a full round-trip. Requires the `db`
extra:

```bash
pip install 'gendantic[db]'          # SQLAlchemy + psycopg
```

```python
from gendantic.db import reflect_schema, load_dataset
from gendantic import generate_dataset_sync

# 1. Reflect: {table_name: PydanticModel}, with primary/foreign keys, enums,
#    nullability and column lengths mapped automatically.
models = reflect_schema("postgresql+psycopg://user:pw@host/shop", schema="public")

# 2. Generate with the normal engine (referential integrity guaranteed).
dataset = generate_dataset_sync(
    {models["customers"]: 100, models["orders"]: 500, models["order_items"]: 2000},
    seed=42,
)

# 3. Load back, parent-first, resetting identity sequences on PostgreSQL.
load_dataset(dataset, "postgresql+psycopg://user:pw@host/shop", schema="public")
```

What the reflection maps:

- Single-column primary keys → `PrimaryKey` (serial/identity → `"sequential"`,
  UUID → `"uuid"`); composite primary keys → `__primary_key__`.
- Foreign keys → `ForeignKey` (single-column) or `ForeignKeySpec` in
  `__foreign_keys__` (composite), including **join tables**.
- Enum columns → `Categorical`; `VARCHAR(n)` → a `max_length` constraint;
  nullable columns → `Optional`. Remaining text columns are generated by the LLM.

**Match production statistics.** If the database already holds data, fit
distributions to it so the synthetic data resembles the real thing:

```python
from gendantic.db import infer_distributions

specs = infer_distributions("postgresql+psycopg://user:pw@host/shop", "orders")
# {column: DistributionSpec}
# numeric columns -> fitted Normal; low-cardinality columns -> weighted Categorical
```

Built on SQLAlchemy reflection, so the same code works against PostgreSQL,
MySQL, SQLite and other supported engines (PostgreSQL is the primary target).
Identity-sequence resetting after load is PostgreSQL-specific.

## Configuration

Gendantic uses [LiteLLM](https://docs.litellm.ai/) to connect to a LiteLLM proxy, which provides unified access to multiple LLM providers.

### Setting up LiteLLM Proxy

1. Start a LiteLLM proxy server (see [LiteLLM docs](https://docs.litellm.ai/docs/proxy/quick_start)):
   ```bash
   litellm --model gpt-4o-mini
   ```

2. Configure the environment variables:
   ```bash
   export LITELLM_API_BASE="http://localhost:4000"
   export LITELLM_MODEL="gpt-4o-mini"
   export LITELLM_API_KEY=""  # If your proxy requires authentication
   ```

### Environment Variables

| Variable | Description | Default |
|----------|-------------|---------|
| `LITELLM_API_BASE` | Base URL for the LiteLLM proxy | (required) |
| `LITELLM_MODEL` | Model identifier to use | `gpt-4o-mini` |
| `LITELLM_API_KEY` | API key for proxy authentication | (empty) |
| `GENDANTIC_MAX_CONCURRENCY` | Max concurrent LLM field-generation calls per request (overridden by the `max_concurrency` argument) | `8` |

### Using Different Models

Change the model by setting `LITELLM_MODEL`:
```bash
# Use Claude
export LITELLM_MODEL="claude-3-5-sonnet-20241022"

# Use GPT-4
export LITELLM_MODEL="gpt-4o"
```

## API Reference

### `generate_synthetic_data(model_class, count, *, context, seed, max_concurrency)`

Generate synthetic data records for a Pydantic model.

- `model_class`: Your Pydantic BaseModel class
- `count`: Number of records to generate (default: 10)
- `context`: Business context for more realistic generation
- `seed`: Random seed for reproducible distribution sampling
- `max_concurrency`: Max concurrent LLM field-generation calls (default: `GENDANTIC_MAX_CONCURRENCY`, else 8)

### `generate_synthetic_data_batch(model_class, contexts, count, *, seed, max_concurrency)`

Generate multiple batches of synthetic data for different contexts concurrently. `max_concurrency` is a single budget shared across all contexts.

### `generate_dataset(counts, *, seed, context)`

Generate several related models together with referential integrity.

- `counts`: `dict[type[BaseModel], int]` mapping each model class to its row count
- `seed`: Optional seed for reproducible keys and sampling
- `context`: Optional generation context passed to each model

Returns: `Dataset` — a mapping keyed by model class (`dataset[Model] -> list[Model]`), with a `.to_dataframes()` helper. A synchronous `generate_dataset_sync(...)` wrapper is also available.

### `generate_model_from_description(description, *, model_name)`

Generate a Pydantic model class from a natural language description.

- `description`: Natural language description of the data model
- `model_name`: Optional name for the generated class

Returns: `tuple[type[BaseModel], str]` - The model class and its source code

### `extend_model_with_distributions(model_class)`

Extend a basic model with LLM-suggested statistical distributions.

- `model_class`: A Pydantic BaseModel with plain type annotations

Returns: `tuple[type[BaseModel], str]` - Extended model with distribution annotations and source code

### `extend_model_with_correlations(model_class)`

Extend an existing model with LLM-suggested correlations and copula types.

- `model_class`: A Pydantic BaseModel with at least 2 distribution-annotated fields

Returns: `tuple[type[BaseModel], str]` - Extended model with `__correlations__` and source code

### `Correlations(*specs, default_copula)`

Define correlations between distribution-sampled fields.

```python
__correlations__ = Correlations(
    ("field1", "field2", 0.5),                    # Uses default copula (gaussian)
    ("field3", "field4", 0.7, "gumbel"),          # Explicit copula type
    default_copula="gaussian",                     # Default copula for all
)
```

Each spec is a tuple: `(field1, field2, correlation)` or `(field1, field2, correlation, copula_type)`

### `Conditional(on, cases, default)`

Field annotation that selects a distribution based on another field's value.

- `on`: Name of the discriminator field (must itself be distribution-sampled)
- `cases`: Mapping of match keys to `DistributionSpec`. Keys are either exact values (e.g. `"Eng"`) or `Range` bins — not a mix of both.
- `default`: `DistributionSpec` used when no case matches

### `Range(min=None, max=None)`

Half-open interval `[min, max)` used as a `Conditional` case key. At least one bound is required; omit `min` or `max` for an open-ended bin.

### `Constraints(*orderings)`

Class attribute (`__constraints__`) holding cross-field constraints.

### `Ordering(*fields, method="sort")`

Guarantees `fields` come out sorted ascending (ties allowed) per record. Requires at least two distinct, distribution-sampled field names.

- `method="sort"` (default) — sort each row's values across the fields. Always succeeds; reshapes each field's marginal into an order statistic.
- `method="resample"` — keep each field's own value and redraw only violating records. Preserves marginals when they're compatible with the order; raises if the rejection budget is exhausted; fields must be independent (not correlated or conditional).

```python
__constraints__ = Constraints(Ordering("check_in", "check_out"))
__constraints__ = Constraints(
    Ordering("birth", "hire", "termination", method="resample")
)
```

### `fidelity_report(records, model_class, *, alpha=0.05, correlation_tolerance=0.15)`

Statistically check how well generated `records` match `model_class`'s declared distributions and correlations. Never raises.

- `records`: Generated model instances or plain dicts
- `model_class`: The model the records were generated for
- `alpha`: Significance level; a field passes when its goodness-of-fit p-value is >= `alpha`
- `correlation_tolerance`: Max absolute difference between the observed rank statistic (Kendall's τ for Archimedean copula families, Spearman's ρ otherwise) and the declared target for a pair to pass

Returns: `FidelityReport` with `.passed`, `.fields` (list of `FieldFidelity`), `.correlations` (list of `CorrelationFidelity`), and `.summary()`.

### Distribution Classes

| Distribution | Parameters | Use Case |
|-------------|------------|----------|
| `Normal(mean, std)` | mean, standard deviation | Salaries, scores, measurements |
| `Uniform(min, max)` | minimum, maximum | Ages, random IDs, dates |
| `Categorical(weights)` | dict of category: probability | Departments, statuses, types |
| `LogNormal(mean, sigma)` | log-mean, log-std | Incomes, file sizes, prices |
| `Exponential(scale)` | scale (1/rate) | Wait times, decay processes |
| `Poisson(lam)` | lambda (rate) | Event counts, arrivals |
| `Beta(alpha, beta)` | shape parameters | Probabilities, percentages |
| `Binomial(n, p)` | trials, probability | Success counts |

## Notebooks

The `notebooks/` directory contains interactive tutorials:

1. **01_getting_started.ipynb** - Basic usage and core concepts
2. **02_distributions.ipynb** - All available statistical distributions
3. **03_correlations.ipynb** - Correlated fields and copula types
4. **04_dynamic_models.ipynb** - Generating models from descriptions
5. **05_model_extension.ipynb** - Extending models with distributions and correlations
6. **06_relational.ipynb** - Multi-model datasets with primary/foreign keys and referential integrity
7. **07_fidelity.ipynb** - Statistically validating generated data against its spec with `fidelity_report()`
8. **08_conditional.ipynb** - Conditional distributions (`Conditional`, `Range`) and cross-field ordering constraints (`Constraints`, `Ordering`)

Install notebook dependencies:
```bash
uv sync --extra notebooks
```

## Development

```bash
git clone https://github.com/benjaminr/gendantic
cd gendantic
# The 'dev' dependency group is installed by default. The db and pandas
# extras are needed for the full test suite (test_db.py and the DataFrame
# export tests); without them those tests are skipped.
uv sync --extra db --extra pandas

# Run tests
uv run pytest

# Type checking
uv run mypy src/

# Format and lint
uv run ruff format
uv run ruff check
```

## License

MIT
