Metadata-Version: 2.4
Name: datasetforge-kit
Version: 0.1.0
Summary: Turn messy public data into clean, ML-ready datasets.
Author: GitHub Copilot
License-Expression: MIT
Keywords: data,etl,cli,json,csv,dataset
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Utilities
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: parquet
Requires-Dist: pyarrow>=16.0.0; extra == "parquet"
Provides-Extra: dev
Requires-Dist: pytest>=8.0.0; extra == "dev"
Requires-Dist: build>=1.2.0; extra == "dev"
Dynamic: license-file

# DatasetForge

DatasetForge turns messy JSON sources into clean, reproducible datasets for analysis and ML workflows.

## What it does

- Load local JSON or REST API responses
- Flatten nested JSON into tabular records
- Rename, select, and drop fields
- Fill missing values and deduplicate records
- Infer a simple schema summary
- Export CSV, JSON, and optional Parquet
- Write reproducible config, metadata, and dataset cards
- Run from a single CLI command

## Install

```bash
pip install datasetforge-kit
```

For local development:

```bash
pip install -e .
```

Optional Parquet support:

```bash
pip install datasetforge-kit[parquet]
```

## Quick start

Create a config file:

```bash
datasetforge init-config --path datasetforge.config.json
```

Run the pipeline:

```bash
datasetforge run --config datasetforge.config.json
```

Inspect the source:

```bash
datasetforge inspect --config datasetforge.config.json
```

Generate a schema summary:

```bash
datasetforge schema --config datasetforge.config.json
```

## Config format

```json
{
  "name": "sample-dataset",
  "source": {
    "kind": "json_file",
    "location": "data/input.json",
    "records_key": null,
    "line_delimited": false
  },
  "transforms": {
    "flatten_separator": ".",
    "rename": {"user.name": "name"},
    "select": ["id", "name", "score"],
    "drop": [],
    "missing_tokens": ["", "null", "N/A"],
    "fill_values": {"score": 0},
    "dedupe_keys": ["id"],
    "fuzzy_dedupe_keys": [],
    "fuzzy_threshold": 0.0
  },
  "validation": {
    "required_fields": ["id", "name"],
    "field_types": {"id": "integer"},
    "allowed_values": {}
  },
  "export": {
    "format": "csv",
    "path": "dist/sample-dataset.csv",
    "indent": 2,
    "include_metadata": false
  },
  "versioning": {
    "enabled": true,
    "directory": "dist/versions"
  }
}
```

## Python API

```python
from datasetforge import load_config, run_pipeline

config = load_config("datasetforge.config.json")
result = run_pipeline(config)
print(result.export_path)
```

## Notes

- Parquet export requires `pyarrow`.
- REST sources use the standard library HTTP client and expect JSON responses.
- Versioned runs write a deterministic content hash to the output path and manifest.

## Roadmap

The library is structured for scheduled jobs, deeper schema rules, version history, fuzzy dedupe, and richer notebook helpers without changing the CLI contract.
