Metadata-Version: 2.4
Name: datalineage-thedebuggers
Version: 1.0.3
Summary: Git-like versioning system for ML datasets — immutable, reproducible, auditable.
Home-page: https://github.com/Rakshita2305/DataLineage
Author: The Debuggers
Author-email: 
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Version Control
Requires-Python: >=3.8
Description-Content-Type: text/markdown
Requires-Dist: pandas>=1.5.0
Requires-Dist: numpy>=1.23.0
Dynamic: author
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# DataLineage: Minimal Dataset Versioning System for Text Data

## Overview

**DataLineage** is a minimal, reproducible dataset versioning system designed for text data with strong emphasis on immutability, transparency, and auditability. It treats datasets like code: immutable, versioned, measurable, and explainable.

This system addresses critical failures in large-scale ML pipelines arising from poorly managed, ad-hoc data preprocessing without explicit tracking of transformations and their impact.

---

## Core Features

### 1. **Immutable Dataset Versions**
- Once created, dataset versions cannot be modified
- Any change in raw data or preprocessing creates a new version
- Ensures reproducibility, traceability, and auditability
- All versions stored in `.mydata/versions/{version_id}/`

### 2. **Cryptographic Version Identification**
- Each version has a unique SHA256 hash
- Hash formula: `version_hash = SHA256(SHA256(raw_input_bytes) + ':' + SHA256(config_json))`
- Hash encodes both: the raw data identity AND the preprocessing configuration
- Two versions with same data but different configs will have different version IDs
- Two versions with different data but same config will also have different version IDs
- Graph-based parent/child linkage maintains version history

### 3. **Configurable Preprocessing Pipeline**
- **Basic Cleaning:**
  - Smart null handling (median for numeric, mode for categorical)
  - Duplicate removal
  - Text normalization (unicode, lowercasing, punctuation removal)
  - Unwanted value filtering

- **Advanced Transformations:**
  - Column selection/dropping
  - Row filtering (numeric ranges, value inclusion/exclusion)
  - Value mapping (categorical encoding, replacements)
  - Column renaming

- **Deterministic & Reproducible:**
  - Same input + same config → identical output every time
  - No randomness, no ML models, CPU-only execution
  - Full preprocessing config stored with each version

### 4. **Version Comparison**
- Side-by-side analysis of two dataset versions
- Metrics compared: row count, column differences, data changes
- Detailed diff reports stored as JSON
- Clear explanations of differences

### 5. **Filesystem-Based Storage** (No Database)
- Lightweight, portable, Git-compatible
- `.mydata/` directory structure:
  ```
  .mydata/
  ├── HEAD                          # Current version pointer
  ├── logs.jsonl                    # All version commit history (JSON Lines format)
  ├── repo_meta.json               # Repository metadata
  ├── versions/
  │   ├── {version_hash}/
  │   │   ├── raw_snapshot.csv     # Original input data
  │   │   ├── processed.csv        # Preprocessed output
  │   │   └── metadata.json        # Version metadata & config
  │   └── ...
  └── raw_data/                    # Raw input archive
      └── {hash}__{filename}
  ```

---

## System Architecture

### Version Creation Modes

#### **Mode 1: Raw Dataset Input (Option 3)**
```
Raw Dataset → Default Preprocessing (smart_fill, text cleanup, dedup) → Version
```
- User chooses: apply preprocessing or store raw as-is
- Default preprocessing: deterministic, data-preserving (smart_fill strategy)
- Suitable for: initial data ingestion, baseline versioning

#### **Mode 2: HEAD + Configuration (Option 2)**
```
Current HEAD Version → User Provided Config (transformations) → New Version
```
- Takes already-processed data and applies custom preprocessing/transformations
- Supports: column selection, filtering, encoding, renaming
- Suitable for: feature engineering, comparative analysis, incremental pipelines

#### **Mode 3: Deduplication**
```
New Config Applied → Same Output as Existing Version → Mark as Duplicate
```
- Automatically detects when different configs produce identical output
- Reuses existing version (storage efficient)
- Logged as "dedupe_hit" event

---

## Preprocessing Configuration

### Configuration Structure
```json
{
  "drop_nulls": true,
  "null_strategy": "smart_fill",
  "null_fill_text": "",
  "null_fill_numeric": 0,
  
  "cleanup_text": true,
  "lowercase_text": true,
  "remove_punctuation": true,
  "collapse_spaces": true,
  "normalize_unicode": true,
  
  "drop_duplicates": true,
  "sort_rows": true,
  
  "select_columns": ["col1", "col2"],
  "drop_columns": ["col3"],
  
  "filter_numeric": {
    "age": {"min": 18, "max": 65}
  },
  "filter_include": {
    "pclass": [1, 2]
  },
  
  "value_mappings": {
    "sex": {"male": "m", "female": "f"}
  },
  
  "rename_columns": {
    "old_name": "new_name"
  }
}
```

### Null Handling Strategies
- **drop_any**: Drop rows with ANY null values (aggressive)
- **drop_all**: Drop only rows where ALL values are null
- **fill**: Fill nulls with specified defaults
- **smart_fill**: Intelligent filling (numeric → median, categorical → mode)
- **keep**: Keep nulls as-is

---

## Reproducibility Guarantees

### 1. **Deterministic Preprocessing**
- No randomness in any operation
- Median/mode used for null filling (stable statistics)
- Unicode normalization deterministic
- Sorting stable and reproducible

### 2. **Version Hash Encodes Input and Config**
- Same raw data + same config → same version hash
- Version hash is computed from the raw input bytes and the config JSON, not from processed output
- Bit-for-bit reproducible across machines

### 3. **Full Auditability**
- All preprocessing steps logged
- Config stored with each version
- Parent/child version tracking
- Commit messages document intent

### 4. **Immutability**
- Versions never modified (read-only after creation)
- Changes require creating new version
- Complete history preserved

### 5. **Reproduce Command (Option 13)**
- Re-runs preprocessing from stored raw_snapshot + config_snapshot
- Recomputation of version_hash, verifies it matches stored hash — live proof of reproducibility

---

## Usage Guide

### 6. Reproduce Version (Verify Reproducibility)
```bash
# Choose: 13) Reproduce Version (Verify Reproducibility)
# Enter version ID to reproduce and verify
```

### 1. Initialize Repository
```bash
python app.py
# Choose: 1) Initialize / Show Repository Status
```

### 2. Commit Raw Dataset
```bash
# Choose: 3) Input of Raw Dataset
# Dataset path: path/to/dataset.csv
# Apply preprocessing: yes (smart_fill) or no (raw as-is)
```

### 3. Apply Configuration & Create Version
```bash
# Choose: 2) Commit from Current HEAD + Config
# Config path: path/to/config.json
```

### 4. List All Versions
```bash
# Choose: 4) List Versions
```

### 5. Compare Two Versions
```bash
# Choose: 7) Compare Two Versions
```

---

## Data Formats Supported

- **CSV** - Comma-separated values
- **TSV** - Tab-separated values
- **JSON** - JSON arrays of objects
- **JSONL** - Line-delimited JSON
- **TXT** - One text sample per line (raw text)

---

## Constraints Met

✅ CPU-Only Execution  
✅ Python + Pandas + NumPy Only  
✅ Filesystem-Based Storage  
✅ Supports CSV, TSV, JSON, JSONL, and TXT input formats  
✅ Deterministic & Reproducible  

---

## Quick Start

```bash
cd DataLineage
python app.py
# Follow prompts to create versions and compare them
```

---

## CLI Menu Reference

When you run `python app.py`, you will see the following menu options:

```
  1) Init/Status
     - Initializes project structure or shows current status
  2) Commit from HEAD + Config
     - Takes config file, creates new version from HEAD
  3) Commit from Raw Dataset
     - Takes raw dataset path + message, optionally applies default preprocessing
  4) List Versions
     - Shows committed versions with parent linkage and summary
  5) Checkout Version
     - Shows versions, then lets you select directly or view one first
  6) View Specific Version Logs
     - Shows detailed metadata for only the version you request
  7) Compare Two Versions
     - Shows concise differences and stores detailed diff report
  8) Verify Version Integrity
     - Checks integrity of a specific version by its ID
  9) Train Bag-of-Words Classifier
     - Trains a NumPy-only Naive Bayes classifier on a text column for a selected version
 10) Evaluate Version
     - Run BoW evaluation and store metrics for a version
 11) Show Metric Trend
     - Show F1 progression across version history
 12) Recommend Best Version
     - Scan all evaluated versions and rank by F1 score
 13) Exit
```

---

## Example Workflow

1. **Initialize the repository:**
   ```bash
   python app.py
   # Choose: 1) Init/Status
   ```
2. **Commit a raw dataset:**
   ```bash
   # Choose: 3) Commit from Raw Dataset
   # Provide dataset path and commit message
   # Optionally apply default preprocessing
   ```
3. **Apply a config to HEAD:**
   ```bash
   # Choose: 2) Commit from HEAD + Config
   # Provide config file path
   ```
4. **List and compare versions:**
   ```bash
   # Choose: 4) List Versions
   # Choose: 7) Compare Two Versions
   ```
5. **Verify version integrity:**
   ```bash
   # Choose: 8) Verify Version Integrity
   # Enter version ID to check
   ```
6. **Train and evaluate classifier:**
   ```bash
   # Choose: 9) Train Bag-of-Words Classifier
   # Choose: 10) Evaluate Version
   ```
7. **Show metric trend and best version:**
   ```bash
   # Choose: 11) Show Metric Trend
   # Choose: 12) Recommend Best Version
   ```

---

## How to Verify Integrity

To check if a version is uncorrupted and exactly as originally committed:

- Use menu option **8) Verify Version Integrity**
- Enter the version ID you want to check
- The system will recompute the hash of the processed data and compare it to the stored hash
- If the hashes match, the version is valid and unmodified

---

## Reproducibility Note

- All versioning and preprocessing is deterministic and fully auditable.
- The integrity check guarantees that your data and config have not changed since commit.
- If you ever need to prove a version is bit-for-bit identical to its original state, use the integrity check.

---

## Support

For questions or issues, contact the hackathon team or open an issue in your project repository.
