Metadata-Version: 2.4
Name: silver-data
Version: 0.3.0
Summary: Inspectable, deterministic dataset contracts and loaders for Silver.
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/adfgdartec/silver-data
Project-URL: Repository, https://github.com/adfgdartec/silver-data
Project-URL: Issues, https://github.com/adfgdartec/silver-data/issues
Keywords: machine-learning,datasets,csv,data-validation,python
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: pandas
Requires-Dist: pandas>=1.0.0; extra == "pandas"
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pandas>=1.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
Requires-Dist: flake8>=6.0.0; extra == "dev"
Requires-Dist: mypy>=1.0.0; extra == "dev"
Requires-Dist: build>=0.10.0; extra == "dev"
Requires-Dist: twine>=4.0.0; extra == "dev"
Dynamic: license-file

# silver-data

[![Python Version](https://img.shields.io/badge/python-3.8%2B-blue.svg)](https://www.python.org/downloads/)
[![License](https://img.shields.io/badge/license-Apache%202.0-green.svg)](LICENSE)
[![Tests](https://img.shields.io/badge/tests-passing-brightgreen.svg)](tests/)
[![Code Style](https://img.shields.io/badge/code%20style-flake8-blue.svg)](https://flake8.pycqa.org/)

Inspectable, deterministic dataset contracts and loaders for Silver. A Python package designed for ML researchers who need reliable dataset handling with built-in validation and reproducibility features.

The base install uses only the Python standard library for records, JSON, JSONL,
and CSV. Add pandas only when you need DataFrame conversion:

```bash
pip install 'silver-data[pandas]'
```

## Installation

```bash
pip install silver-data
```

## Quick Start

```python
from silver_data import Dataset

# Load from CSV file
dataset = Dataset.from_csv("my_data", "path/to/data.csv")

# Optional: load from pandas
import pandas as pd
df = pd.read_csv("path/to/data.csv")
dataset = Dataset.from_pandas("my_data", df)

# Inspect dataset
report = dataset.inspect()
print(f"Rows: {report.rows}, Columns: {len(report.columns)}")
for col in report.columns:
    print(f"  {col.name}: {col.value_type} ({col.unique} unique, {col.missing} missing)")

# Validate dataset
validation = dataset.validate()
if not validation.valid:
    print("Errors:", validation.errors)
if validation.warnings:
    print("Warnings:", validation.warnings)

# Split dataset for ML workflows
train, val, test = dataset.split(train=0.8, validation=0.1, test=0.1)
print(f"Train: {len(train.records())}, Val: {len(val.records())}, Test: {len(test.records())}")
```

### Architecture-aware visualizations

Silver can select only the visualizations justified by the packages and data
signals present in an architecture. Omitted visualizations explain what is
missing, so dashboards do not claim to show metrics that were never produced.

```python
from silver_data import design_visualization_plan

plan = design_visualization_plan({"architecture": {
    "uses": ["data", "diagnostics", "run", "torch"],
    "signals": ["dataset", "missingness", "labels", "predictions", "metrics", "history", "run", "events", "features", "pipeline"],
}})
for visualization in plan.selected:
    print(visualization.key, visualization.reason)
```

Call `plan.to_dict()` to pass the explainable plan to a UI or report builder,
and use `max_items` when a surface has limited space.

## Features

- **Multiple Data Sources**: Load from CSV, JSON, JSONL, and pandas DataFrames
- **Dataset Inspection**: Get detailed column statistics and metadata
- **Data Validation**: Automatic detection of missing values, inconsistent columns, and data quality issues
- **Deterministic Fingerprinting**: Generate unique identifiers for datasets to ensure reproducibility
- **Smart Splitting**: Train/validation/test splitting with customizable ratios
- **Immutable Design**: Safe data handling with copy-on-write semantics
- **Type Safety**: Full type hints for better IDE support and fewer bugs

## Use Cases

### ML Pipeline Integration

```python
from silver_data import Dataset
import pandas as pd

# Load and validate training data
df = pd.read_csv("train.csv")
dataset = Dataset.from_pandas("training", df)

# Ensure data quality before training
validation = dataset.validate()
if not validation.valid:
    raise ValueError(f"Dataset validation failed: {validation.errors}")

# Split for cross-validation
train_split, val_split, test_split = dataset.split(train=0.7, validation=0.15, test=0.15)

# Use fingerprints for caching
cache_key = dataset.fingerprint()
print(f"Dataset fingerprint: {cache_key}")
```

### Data Quality Monitoring

```python
from silver_data import Dataset

# Monitor data drift over time
dataset_v1 = Dataset.from_csv("data_v1", "data_2024_01.csv")
dataset_v2 = Dataset.from_csv("data_v2", "data_2024_02.csv")

if dataset_v1.fingerprint() != dataset_v2.fingerprint():
    print("Dataset has changed - retrain models")

# Check for new data quality issues
report_v2 = dataset_v2.inspect()
for col in report_v2.columns:
    if col.missing > len(dataset_v2.records()) * 0.1:  # More than 10% missing
        print(f"Warning: {col.name} has high missing rate: {col.missing}")
```

### Experiment Reproducibility

```python
from silver_data import Dataset

# Ensure exact same data across experiments
dataset = Dataset.from_csv("experiment", "data.csv")
experiment_id = f"exp_{dataset.fingerprint()}"

# Log for reproducibility
print(f"Running experiment {experiment_id} with dataset fingerprint {dataset.fingerprint()}")
```

## Advanced Usage

### Custom Data Loading

```python
from silver_data import Dataset
import json

# Load from custom JSON format
with open("custom_data.json") as f:
    data = json.load(f)
dataset = Dataset.from_json("custom", data)

# Load from streaming JSONL
with open("streaming_data.jsonl") as f:
    dataset = Dataset.from_jsonl("streaming", f.read())
```

### Data Type Analysis

```python
from silver_data import Dataset

dataset = Dataset.from_csv("analysis", "mixed_data.csv")
report = dataset.inspect()

# Analyze column types
string_cols = [c.name for c in report.columns if c.value_type == "string"]
numeric_cols = [c.name for c in report.columns if c.value_type == "number"]
mixed_cols = [c.name for c in report.columns if c.value_type == "mixed"]

print(f"String columns: {string_cols}")
print(f"Numeric columns: {numeric_cols}")
print(f"Mixed type columns: {mixed_cols}")
```

## Requirements

- Python 3.8+
- pandas 1.0+

## Development

```bash
# Install development dependencies
pip install -e ".[dev]"

# Run tests
pytest

# Run tests with coverage
pytest --cov=silver_data --cov-report=html

# Run linting
flake8 src/ tests/
mypy src/
```

## Contributing

Contributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.

## License

Apache-2.0 - see [LICENSE](LICENSE) file for details.

## Related Packages

- [silver-run](https://github.com/adfgdartec/silver-run) - Training lifecycle management
- [silver-diagnostics](https://github.com/adfgdartec/silver-diagnostics) - ML diagnostics
- [silver-adapters](https://github.com/adfgdartec/silver-adapters) - Framework adapters
