Metadata-Version: 2.4
Name: silver-data
Version: 0.1.0
Summary: Inspectable, deterministic dataset contracts and loaders for Silver.
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/adfgdartec/silver-data
Project-URL: Repository, https://github.com/adfgdartec/silver-data
Project-URL: Issues, https://github.com/adfgdartec/silver-data/issues
Keywords: machine-learning,datasets,csv,data-validation,python
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: pandas
Requires-Dist: pandas>=1.0.0; extra == "pandas"
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pandas>=1.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
Requires-Dist: flake8>=6.0.0; extra == "dev"
Requires-Dist: mypy>=1.0.0; extra == "dev"
Requires-Dist: build>=0.10.0; extra == "dev"
Requires-Dist: twine>=4.0.0; extra == "dev"
Dynamic: license-file

# silver-data

[![Python Version](https://img.shields.io/badge/python-3.8%2B-blue.svg)](https://www.python.org/downloads/)
[![License](https://img.shields.io/badge/license-Apache%202.0-green.svg)](LICENSE)
[![Tests](https://img.shields.io/badge/tests-passing-brightgreen.svg)](tests/)
[![Code Style](https://img.shields.io/badge/code%20style-flake8-blue.svg)](https://flake8.pycqa.org/)

Inspectable, deterministic dataset contracts and loaders for Silver. A Python package designed for ML researchers who need reliable dataset handling with built-in validation and reproducibility features.

The base install uses only the Python standard library for records, JSON, JSONL,
and CSV. Add pandas only when you need DataFrame conversion:

```bash
pip install 'silver-data[pandas]'
```

## Installation

```bash
pip install silver-data
```

## Quick Start

```python
from silver_data import Dataset

# Load from CSV file
dataset = Dataset.from_csv("my_data", "path/to/data.csv")

# Optional: load from pandas
import pandas as pd
df = pd.read_csv("path/to/data.csv")
dataset = Dataset.from_pandas("my_data", df)

# Inspect dataset
report = dataset.inspect()
print(f"Rows: {report.rows}, Columns: {len(report.columns)}")
for col in report.columns:
    print(f"  {col.name}: {col.value_type} ({col.unique} unique, {col.missing} missing)")

# Validate dataset
validation = dataset.validate()
if not validation.valid:
    print("Errors:", validation.errors)
if validation.warnings:
    print("Warnings:", validation.warnings)

# Split dataset for ML workflows
train, val, test = dataset.split(train=0.8, validation=0.1, test=0.1)
print(f"Train: {len(train.records())}, Val: {len(val.records())}, Test: {len(test.records())}")
```

## Features

- **Multiple Data Sources**: Load from CSV, JSON, JSONL, and pandas DataFrames
- **Dataset Inspection**: Get detailed column statistics and metadata
- **Data Validation**: Automatic detection of missing values, inconsistent columns, and data quality issues
- **Deterministic Fingerprinting**: Generate unique identifiers for datasets to ensure reproducibility
- **Smart Splitting**: Train/validation/test splitting with customizable ratios
- **Immutable Design**: Safe data handling with copy-on-write semantics
- **Type Safety**: Full type hints for better IDE support and fewer bugs

## Use Cases

### ML Pipeline Integration

```python
from silver_data import Dataset
import pandas as pd

# Load and validate training data
df = pd.read_csv("train.csv")
dataset = Dataset.from_pandas("training", df)

# Ensure data quality before training
validation = dataset.validate()
if not validation.valid:
    raise ValueError(f"Dataset validation failed: {validation.errors}")

# Split for cross-validation
train_split, val_split, test_split = dataset.split(train=0.7, validation=0.15, test=0.15)

# Use fingerprints for caching
cache_key = dataset.fingerprint()
print(f"Dataset fingerprint: {cache_key}")
```

### Data Quality Monitoring

```python
from silver_data import Dataset

# Monitor data drift over time
dataset_v1 = Dataset.from_csv("data_v1", "data_2024_01.csv")
dataset_v2 = Dataset.from_csv("data_v2", "data_2024_02.csv")

if dataset_v1.fingerprint() != dataset_v2.fingerprint():
    print("Dataset has changed - retrain models")

# Check for new data quality issues
report_v2 = dataset_v2.inspect()
for col in report_v2.columns:
    if col.missing > len(dataset_v2.records()) * 0.1:  # More than 10% missing
        print(f"Warning: {col.name} has high missing rate: {col.missing}")
```

### Experiment Reproducibility

```python
from silver_data import Dataset

# Ensure exact same data across experiments
dataset = Dataset.from_csv("experiment", "data.csv")
experiment_id = f"exp_{dataset.fingerprint()}"

# Log for reproducibility
print(f"Running experiment {experiment_id} with dataset fingerprint {dataset.fingerprint()}")
```

## Advanced Usage

### Custom Data Loading

```python
from silver_data import Dataset
import json

# Load from custom JSON format
with open("custom_data.json") as f:
    data = json.load(f)
dataset = Dataset.from_json("custom", data)

# Load from streaming JSONL
with open("streaming_data.jsonl") as f:
    dataset = Dataset.from_jsonl("streaming", f.read())
```

### Data Type Analysis

```python
from silver_data import Dataset

dataset = Dataset.from_csv("analysis", "mixed_data.csv")
report = dataset.inspect()

# Analyze column types
string_cols = [c.name for c in report.columns if c.value_type == "string"]
numeric_cols = [c.name for c in report.columns if c.value_type == "number"]
mixed_cols = [c.name for c in report.columns if c.value_type == "mixed"]

print(f"String columns: {string_cols}")
print(f"Numeric columns: {numeric_cols}")
print(f"Mixed type columns: {mixed_cols}")
```

## Requirements

- Python 3.8+
- pandas 1.0+

## Development

```bash
# Install development dependencies
pip install -e ".[dev]"

# Run tests
pytest

# Run tests with coverage
pytest --cov=silver_data --cov-report=html

# Run linting
flake8 src/ tests/
mypy src/
```

## Contributing

Contributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.

## License

Apache-2.0 - see [LICENSE](LICENSE) file for details.

## Related Packages

- [silver-run](https://github.com/adfgdartec/silver-run) - Training lifecycle management
- [silver-diagnostics](https://github.com/adfgdartec/silver-diagnostics) - ML diagnostics
- [silver-adapters](https://github.com/adfgdartec/silver-adapters) - Framework adapters
