Metadata-Version: 2.4
Name: rand-engine
Version: 0.7.0rc1
Summary: Sink-agnostic synthetic data at NumPy speed from declarative specs, with related tables through pk/fk key columns; Faker composes on top.
License: MIT
License-File: LICENSE
Keywords: synthetic-data,random-data,mock-data,test-data,numpy,data-engineering
Author: marcoaureliomenezes
Author-email: marcoaurelioreislima@gmail.com
Requires-Python: >=3.10,<4.0
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: 3.15
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Software Development :: Testing
Provides-Extra: faker
Requires-Dist: faker (>=28.4.1,<29.0.0) ; extra == "faker"
Requires-Dist: numpy (>=2.1.1,<3.0.0)
Requires-Dist: pandas (>=2.2.2,<3.0.0)
Requires-Dist: pyarrow (>=23.0.1,<24.0.0)
Project-URL: Changelog, https://github.com/marcoaureliomenezes/rand_engine/blob/master/CHANGELOG.md
Project-URL: Documentation, https://github.com/marcoaureliomenezes/rand_engine/tree/master/docs
Project-URL: Homepage, https://github.com/marcoaureliomenezes/rand_engine
Project-URL: Issues, https://github.com/marcoaureliomenezes/rand_engine/issues
Project-URL: Repository, https://github.com/marcoaureliomenezes/rand_engine
Description-Content-Type: text/markdown

# rand-engine

Synthetic data, very fast, from a declarative spec — for batch and streaming, on any platform.

- **Sink-agnostic:** feed Kafka and other queues, data lakes and object storage, files, databases or Spark.
- **NumPy-fast:** the core is Python + NumPy (`DataGenerator`); Spark is supported (`SparkGenerator`), never required.
- **Faker composes on top:** build a small realistic pool with Faker, sample millions of rows from it at NumPy speed.
- **Related tables:** `pk` and `fk` columns keep keys consistent across tables, batches, streams and files.

## Install

<!-- derived-from: public-api sha256:889c68fd0f20 -->

```bash
pip install rand-engine
```

## Quickstart

<!-- derived-from: data-generator sha256:8af2e4c55309 -->
<!-- derived-from: rand-spec-grammar sha256:77ed429317a3 -->

A spec maps each column to a `method` and its `kwargs`.

```python
from rand_engine import DataGenerator

spec = {
    "user_id": {"method": "int_zfilled", "kwargs": {"length": 8}},
    "age": {"method": "integers", "kwargs": {"min": 18, "max": 80}},
    "plan": {"method": "distincts_prop", "kwargs": {"distincts": {"free": 8, "pro": 2}}},
    "signup": {"method": "dates", "kwargs": {"start": "2024-01-01", "end": "2024-12-31"}},
}

df = DataGenerator(spec, seed=42).size(1_000).get_df()
assert df.shape == (1_000, 4) and df["age"].between(18, 80).all()
```

## Seeded reproducibility

<!-- derived-from: data-generator sha256:8af2e4c55309 -->

The seed is per generator: each `DataGenerator` owns its own NumPy `Generator` (PCG64), so the
caller's NumPy global state is never read or reset. The same spec, seed and size give the same
frame. Values differ from 0.6.x, which drew from the global NumPy seed.

```python
import numpy as np

np.random.seed(0)
state = np.random.get_state()[1].copy()
a = DataGenerator(spec, seed=7).size(100).get_df()
b = DataGenerator(spec, seed=7).size(100).get_df()
assert a.equals(b)
assert (np.random.get_state()[1] == state).all()
```

## Faker pools at scale

<!-- derived-from: generation-methods sha256:739e81e44532 -->

Install the optional Faker dependency with `pip install faker` before running this example.
Faker is slow per value; draw a pool once and let rand-engine sample it.

```python
from faker import Faker

fake = Faker()
Faker.seed(0)
names = [fake.name() for _ in range(200)]

people = DataGenerator({"name": {"method": "distincts", "kwargs": {"distincts": names}}}, seed=1).size(10_000).get_df()
assert people["name"].isin(names).all()
```

## Related tables

<!-- derived-from: pk-fk-constraints sha256:87a6776da0b0 -->

Give the child's `fk` the parent's `pk` spec and size: every child key exists in the parent.

```python
customer_pk = {"method": "pk", "kwargs": {"start": 1, "format": "C{:05d}"}}

customers = DataGenerator({"customer_id": customer_pk}, seed=1).size(500).get_df()
orders = DataGenerator({
    "order_id": {"method": "pk", "kwargs": {"start": 1}},
    "customer_id": {"method": "fk", "kwargs": {"parent": customer_pk, "parent_size": 500}},
}, seed=2).size(2_000).get_df()

assert customers["customer_id"].is_unique
assert set(orders["customer_id"]) <= set(customers["customer_id"])
```

## Streams and files

<!-- derived-from: data-generator sha256:8af2e4c55309 -->
<!-- derived-from: writers-and-streaming sha256:21b53d28967c -->

`stream_dict` yields records forever at a bounded rate; `write` writes csv, json or parquet.

```python
from itertools import islice
import pandas as pd

events = DataGenerator({"event_id": {"method": "pk", "kwargs": {"start": 1}}}, seed=1).size(10)
records = list(islice(events.stream_dict(min_throughput=500, max_throughput=1000), 5))
assert [r["event_id"] for r in records] == [1, 2, 3, 4, 5]

orders_gen = DataGenerator({"order_id": {"method": "pk", "kwargs": {"start": 1}}}, seed=1).size(1_000)
orders_gen.write.format("parquet").save("out/orders.parquet")
assert len(pd.read_parquet("out/orders.parquet")) == 1_000
```

## Learn more

<!-- derived-from: public-api sha256:889c68fd0f20 -->
<!-- derived-from: benchmarks sha256:005825e85875 -->

- [DataGenerator](https://github.com/marcoaureliomenezes/rand_engine/blob/master/docs/1_DATA_GENERATOR.md): methods, seeds, streams.
- [SparkGenerator](https://github.com/marcoaureliomenezes/rand_engine/blob/master/docs/2_SPARK_GENERATOR.md): the same spec as a Spark DataFrame.
- [Writing files](https://github.com/marcoaureliomenezes/rand_engine/blob/master/docs/3_WRITING_FILES.md): `write` and `writeStream`, formats and options.
- [Keys](https://github.com/marcoaureliomenezes/rand_engine/blob/master/docs/4_CONSTRAINTS.md): `pk` and `fk` in depth.
- [Recipes](https://github.com/marcoaureliomenezes/rand_engine/blob/master/docs/5_RECIPES.md): Faker pools, queues, parquet lakes, related tables.
- [Benchmarks](https://github.com/marcoaureliomenezes/rand_engine/blob/master/docs/BENCHMARKS.md): rows/µs per method, measured in CI.
- [Changelog](https://github.com/marcoaureliomenezes/rand_engine/blob/master/CHANGELOG.md): the 0.7.0 breaking changes.
- [License](https://github.com/marcoaureliomenezes/rand_engine/blob/master/LICENSE): MIT.

