Metadata-Version: 2.3
Name: z-temper
Version: 0.1.0
Summary: Autonomous Dataset Tempering Engine for Synthetic Data Scaling
Author: Zeo
Author-email: Zeo <justzeo18@gmail.com>
Requires-Dist: dataflux-core>=0.1.0
Requires-Dist: google-genai>=2.12.1
Requires-Dist: pydantic>=2.0.0
Requires-Dist: python-dotenv>=1.2.2
Requires-Dist: rich>=15.0.0
Requires-Dist: tenacity>=8.0.0
Requires-Dist: tqdm>=4.66.0
Requires-Python: >=3.14
Description-Content-Type: text/markdown

# 🔥 Z-Temper

[![PyPI version](https://img.shields.io/pypi/v/z-temper.svg)](https://pypi.org/project/z-temper/)
[![Python versions](https://img.shields.io/pypi/pyversions/z-temper.svg)](https://pypi.org/project/z-temper/)
[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](https://opensource.org/licenses/MIT)

**Autonomous dataset tempering engine for synthetic data scaling.**

Z-Temper closes the loop between diagnostics and training data. It reads diagnostic reports and blueprints produced by [**Project Peony**](https://pypi.org/project/peony-core/) (`peony-core`), pulls seed exemplars through [**DataFlux**](https://pypi.org/project/dataflux-core/) (`dataflux-core`), and drives Google Gemini through a generate → critique → filter loop to synthesize clean, ready-to-train datasets in ShareGPT or Alpaca format.

---

## How it fits together

```
Peony diagnostic report / blueprint
            │
            ▼
   BlueprintParser  ──►  target capabilities, data recipes, edge cases
            │
            ▼
  SeedDatasetLoader  ◄── dataflux-core / local .json / .jsonl
            │
            ▼
      BatchGovernor
            │
   ┌────────┴────────┐
   ▼                 ▼
LoopGenerator    QualityCritic
(Gemini calls,   (structural checks,
 retry/backoff)   min length, dedup)
   │                 │
   └────────┬────────┘
            ▼
  ready_to_train.jsonl  (ShareGPT / Alpaca)
```

- **`BlueprintParser`** — validates and parses a Peony `orchid_dataset_blueprint.json` (or a full diagnostic report containing one) into target capabilities, synthesis recipes, and edge cases to cover.
- **`SeedDatasetLoader`** — loads seed exemplars via `dataflux-core` when available, falling back to local `.json`/`.jsonl` files.
- **`LoopGenerator`** — calls the Gemini API with a structured Pydantic response schema (`ShareGPTBatch` / `AlpacaBatch`) and retries transient API errors with exponential backoff.
- **`QualityCritic`** — rejects malformed samples, enforces a minimum character length, and suppresses duplicates via content hashing.
- **`BatchGovernor`** — orchestrates the loop end-to-end: generate a batch, filter it, stream valid samples to disk immediately, repeat until the target count is hit. Shows a live progress bar via `rich`.
- **`TemperEngine`** — the top-level facade tying all of the above together; this is what you'd typically import.

---

## Installation

Requires **Python ≥ 3.14**.

```bash
pip install z-temper
```

### From source (development)

Dependency management is via [`uv`](https://docs.astral.sh/uv/).

```bash
git clone https://github.com/JustZeo/Z-Temper.git
cd Z-Temper
uv sync
uv pip install -e .
```

### Configuration

Z-Temper needs a Gemini API key. Either export it in your shell:

```bash
export GEMINI_API_KEY="your-api-key-here"
```

or drop it in a `.env` file in your working directory (loaded automatically):

```
GEMINI_API_KEY=your-api-key-here
```

You can also pass `api_key=` explicitly when constructing `TemperEngine`, which takes priority over both.

---

## Quick Start

### Python API

```python
from z_temper import TemperEngine

engine = TemperEngine(model="gemini-2.5-flash")

engine.synthesize_dataset(
    blueprint_path="path/to/orchid_dataset_blueprint.json",
    output_file="output/ready_to_train.jsonl",
    target_count=100,
    batch_size=5,
    seed_path="path/to/seed_data.jsonl",  # optional
    format_type="sharegpt",               # or "alpaca"
)
```

### CLI

```bash
ztemper \
  --blueprint path/to/orchid_dataset_blueprint.json \
  --output output/tempered_dataset.jsonl \
  --count 500 \
  --batch-size 10 \
  --format sharegpt \
  --model gemini-2.5-flash
```

| Flag | Short | Default | Description |
|---|---|---|---|
| `--blueprint` | `-b` | *required* | Path to a Peony `orchid_dataset_blueprint.json`. |
| `--output` | `-o` | `tempered_dataset.jsonl` | Destination path for the generated dataset. |
| `--count` | `-c` | `100` | Target number of valid samples to produce. |
| `--batch-size` | | `5` | Samples requested per generation call. |
| `--seed` | `-s` | `None` | Optional seed dataset path or identifier. |
| `--format` | `-f` | `sharegpt` | Output schema: `sharegpt` or `alpaca`. |
| `--model` | `-m` | `gemini-2.5-flash` | Gemini model used for generation. |

---

## Output formats

**ShareGPT**
```json
{"conversations": [{"from": "human", "value": "..."}, {"from": "gpt", "value": "..."}]}
```

**Alpaca**
```json
{"instruction": "...", "input": "", "output": "..."}
```

Each line is validated by `QualityCritic` (structure, minimum length, no duplicates) before being written, and results are streamed to disk as they're generated — no data is lost if the run is interrupted partway through.

---

## Testing

```bash
uv run pytest tests/ -v
```

---

## Related projects

- [**Project Peony**](https://pypi.org/project/peony-core/) (`pip install peony-core`) — mechanistic interpretability and diagnostics framework that produces the blueprints Z-Temper consumes.
- [**DataFlux**](https://pypi.org/project/dataflux-core/) (`pip install dataflux-core`) — unified dataset loading library spanning multiple providers, used here for seed exemplars.

---

## License

MIT License. Built for the Project Peony / Z-Temper research pipeline.