Metadata-Version: 2.4
Name: syndatalab
Version: 0.1.0
Summary: Production-grade synthetic data generation using LLMs with structured outputs, caching, checkpointing, and monitoring.
Project-URL: Homepage, https://github.com/codyphi/syndatalab
Project-URL: Documentation, https://github.com/codyphi/syndatalab/blob/main/README.md
Project-URL: Repository, https://github.com/codyphi/syndatalab
Author: deepak yadav
License-Expression: MIT
Keywords: data-generation,llm,pydantic,structured-output,synthetic-data
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: click>=8.1
Requires-Dist: datasets>=2.18
Requires-Dist: json-repair>=0.30
Requires-Dist: pydantic>=2.0
Requires-Dist: rich>=13.0
Requires-Dist: structlog>=24.1
Requires-Dist: tenacity>=8.2
Provides-Extra: all
Requires-Dist: anthropic>=0.30; extra == 'all'
Requires-Dist: litellm>=1.40; extra == 'all'
Requires-Dist: ollama>=0.3; extra == 'all'
Requires-Dist: openai>=1.0; extra == 'all'
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.30; extra == 'anthropic'
Provides-Extra: dev
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: respx>=0.21; extra == 'dev'
Requires-Dist: ruff>=0.4; extra == 'dev'
Provides-Extra: litellm
Requires-Dist: litellm>=1.40; extra == 'litellm'
Provides-Extra: ollama
Requires-Dist: ollama>=0.3; extra == 'ollama'
Provides-Extra: openai
Requires-Dist: openai>=1.0; extra == 'openai'
Provides-Extra: quality
Requires-Dist: datasketch>=1.6; extra == 'quality'
Requires-Dist: sentence-transformers>=3.0; extra == 'quality'
Description-Content-Type: text/markdown

# syndatalab

Production-grade synthetic data generation using LLMs with structured outputs, caching, checkpointing, and monitoring.

## Installation

```bash
pip install syndatalab

# Install with specific provider support
pip install syndatalab[openai]
pip install syndatalab[anthropic]
pip install syndatalab[all]          # all providers
pip install syndatalab[all,quality]  # all providers + quality filters
```

## Quick Start

```python
import syndatalab as sd
from pydantic import BaseModel

class QAPair(BaseModel):
    question: str
    answer: str

class QAGenerator(sd.LLM):
    response_format = QAPair

    def prompt(self, input: dict) -> str:
        return f"Generate a Q&A pair about {input['topic']}"

    def parse(self, input: dict, response: QAPair) -> dict:
        return {"topic": input["topic"], "q": response.question, "a": response.answer}

gen = QAGenerator(model="gpt-4o-mini")
result = gen(dataset=[{"topic": "python"}, {"topic": "rust"}])

print(result.dataset)      # HuggingFace Dataset
print(result.stats)        # Token usage, cost, performance
print(result.run_id)       # Resume ID if interrupted
```

## Features

- **Multi-provider**: OpenAI, Anthropic, LiteLLM, Ollama with automatic detection
- **Provider failover**: Automatic fallback chain across providers
- **Structured outputs**: Pydantic models with layered validation and auto-repair
- **Checkpoint/resume**: Crash recovery with deterministic run IDs
- **Rate limiting**: Dual token-bucket (RPM + TPM) with adaptive backoff
- **Caching**: SQLite-based with WAL mode, TTL, and atomic writes
- **Quality filters**: Deduplication, diversity scoring, custom filters
- **TUI dashboard**: Real-time progress, stats, cost tracking
- **Audit trail**: Per-request JSONL logging for debugging
- **CLI tools**: Cache management, run history, output inspection

## Pipeline Chaining

```python
gen1 = TopicGenerator(model="gpt-4o-mini")
gen2 = QAGenerator(model="gpt-4o-mini")
pipeline = gen1 | gen2
result = pipeline(dataset=[{"seed": "machine learning"}])
```

## Quality Filtering

```python
result = gen(
    dataset=inputs,
    filters=[
        sd.filters.min_length("answer", 50),
        sd.filters.no_duplicates("answer", threshold=0.85),
        sd.filters.custom(lambda row: "I cannot" not in row["answer"]),
    ],
)
```

## CLI

```bash
syndatalab cache stats
syndatalab cache clear
syndatalab history
syndatalab inspect <run_id>
```

## Configuration

Create `~/.syndatalab/config.toml`:

```toml
[defaults]
model = "gpt-4o-mini"
max_retries = 3
max_concurrent_requests = 50
show_dashboard = true

[rate_limits]
max_requests_per_minute = 500
max_tokens_per_minute = 200000

[logging]
level = "INFO"
```

## License

MIT
