Metadata-Version: 2.4
Name: milky
Version: 0.1.2
Summary: Minimal dataset builder for User/Assistant ML training data
Requires-Python: >=3.8
Description-Content-Type: text/markdown

# 🥛 milky

Minimal dataset toolkit for building and cleaning **User/Assistant `.mlk` training data**.

Designed for small language model pipelines like *Author*.

---

# 📌 Overview

milky is a lightweight utility for:

* appending training pairs
* reading `.mlk` datasets
* validating dataset format
* sanitizing corrupted samples
* repairing broken entries
* dataset statistics
* dataset quality scoring
* automatic formatting (multiline-safe)

It is intentionally simple and file-based.

---

# ⚙️ Initialization

When imported, milky prints:

```text id="init1"
[YYYY-MM-DD] Milky initialized.
```

---

# 📄 Dataset Format (.mlk)

Each line follows:

```text id="fmt1"
User: <prompt> Assistant: <response><eos>
```

---

## Example

```text id="fmt2"
User: What is AI? Assistant: AI is the study of intelligent systems.<eos>
User: Write a poem Assistant: The sea is calm tonight...<eos>
```

---

## 🧠 Multiline Handling

milky automatically converts real newlines into safe escaped format:

```text id="fmt3"
\n → \\n
```

So all samples remain **single-line safe**.

---

# 🚀 API Reference

---

## append()

```python id="api1"
append(prompt: str, response: str, target_file: str) -> None
```

Appends a User/Assistant pair to a `.mlk` dataset.

### Behavior:

* validates `.mlk` extension
* automatically formats input (cleaning + newline escaping)
* ensures single-line storage
* appends `<eos>` token

---

## read()

```python id="api2"
read(target_file: str) -> list
```

Reads dataset file.

### Returns:

* raw list of dataset lines

### Notes:

* validates `.mlk` extension
* no parsing or decoding applied

---

## check()

```python id="api3"
check(file_path: str) -> list
```

Validates dataset structure.

### Detects:

* empty lines
* missing `User:`
* missing `Assistant:`
* missing `<eos>`

### Output:

* colored CLI diagnostics
* returns `(line_number, error_type)` list

---

## sanitize()

```python id="api4"
sanitize(file_path: str, apply: bool = False)
```

Removes invalid dataset entries.

### Removes:

* empty lines
* malformed samples

### Behavior:

* `apply=False` → preview only
* `apply=True` → overwrites file

---

## repair()

```python id="api5"
repair(file_path: str, apply: bool = False)
```

Attempts to fix malformed entries instead of deleting them.

### Fixes:

* adds missing `<eos>`
* normalizes spacing
* drops unfixable lines

### Behavior:

* `apply=False` → preview mode
* `apply=True` → writes file

---

## stats()

```python id="api6"
stats(file_path: str) -> dict
```

Computes dataset metrics.

### Measures:

* total lines
* valid samples
* empty lines
* malformed lines
* missing `<eos>`
* average prompt length
* average response length

### Notes:

* automatically decodes escaped newlines (`\\n → \n`)
* used for dataset inspection before training

---

## score()

```python id="api7"
score(file_path: str) -> float
```

Returns a **0–100 dataset quality score**.

### What it evaluates:

* ratio of valid samples
* malformed lines penalty
* missing `<eos>` penalty
* empty line penalty

### Interpretation:

* 90–100 → clean dataset (train-ready)
* 70–89 → good but slightly noisy
* 40–69 → needs cleaning
* <40 → unsafe / unstable dataset

---

# 🧠 Design Philosophy

milky is:

* minimal
* dependency-free
* file-based
* strict format enforcement
* optimized for small LLM pipelines

It does NOT:

* support JSON / Parquet datasets
* perform tokenization
* handle streaming data
* manage multi-file corpora

---

# ⚠️ Important Notes

* `.mlk` format is strict
* each sample must be single-line
* `<eos>` is required for training stability
* sanitize is destructive (use carefully)
* repair is heuristic-based
* formatting is automatic in `append()`

---

# 🥛 Why “milky”?

Because it turns raw text into structured training pairs — cleaning noise into model-ready data.

---

# 📦 Typical Workflow

```text id="flow1"
append → check → stats → score → sanitize → repair → train
```

---

# 🚀 Example Workflow

```python id="ex1"
append("Hello", "Hi!", "data.mlk")

errors = check("data.mlk")

info = stats("data.mlk")

quality = score("data.mlk")

cleaned, removed = sanitize("data.mlk", apply=True)

fixed, changes = repair("data.mlk", apply=True)
```

