Metadata-Version: 2.4
Name: yobro
Version: 0.1.1
Summary: A small language and library for checked, honestly evaluated LLM fine-tuning
Author: omparekh54-lgtm
License: Apache-2.0
Project-URL: Homepage, https://github.com/omparekh54-lgtm/yobro
Project-URL: Issues, https://github.com/omparekh54-lgtm/yobro/issues
Keywords: llm,fine-tuning,lora,dpo,evaluation,dsl
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.24
Provides-Extra: train
Requires-Dist: torch; extra == "train"
Requires-Dist: transformers>=4.45; extra == "train"
Requires-Dist: trl==1.14.0; extra == "train"
Requires-Dist: peft>=0.13; extra == "train"
Requires-Dist: datasets>=2.20; extra == "train"
Requires-Dist: accelerate; extra == "train"
Provides-Extra: parquet
Requires-Dist: pyarrow; extra == "parquet"
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Dynamic: license-file

# Yobro

A small language and Python library for fine-tuning LLMs that **checks your project before it uses a GPU**, **reports honestly whether the model improved**, **refuses to ship a worse model**, and **turns production failures into reviewed training data**.

Yobro does not replace the trainers. It drives Hugging Face TRL and PEFT for the heavy work and adds the checking, evaluation, release and monitoring around them.

```text
yobro check project.yobro     # every check and estimate, no GPU
yobro run project.yobro       # train, compare with the base model, gate, ship
yobro monitor project.yobro   # watch live traffic, cluster failures, queue them for review
```

Version 0.1.1. Status is honest and specific; see [What is tested](#what-is-tested-and-what-is-not).

---

## Install

```bash
python -m venv .venv && source .venv/bin/activate     # Windows: .venv\Scripts\activate
pip install -e ".[train]"      # from this folder; installs torch, transformers, trl, peft, datasets
yobro doctor                   # shows versions and whether a GPU was found
```

Without `[train]` you can still run checks, splits, reports against a served model, gates and monitoring.

## Connect your 3B model

There are two ways, and you will usually use both.

**1. A local Hugging Face folder, needed for training.** It must hold the model in Hugging Face format: `config.json`, `*.safetensors`, and the tokenizer files. Training needs these weights.

```yobro
model base = "/path/to/your-3b-model" { params = "3B" }
```

**2. A running server, for evaluation, judging and monitoring.** Anything that speaks the OpenAI chat API works, including llama.cpp's `llama-server`, Ollama, vLLM and LM Studio.

```yobro
model local = "http://localhost:8080/v1" { served_model = "your-3b" }
```

Answers from servers are cached. If you swap the model behind the same address, add `version = "..."` to the model or run with `--fresh`.

Or keep the base model local for training but answer evaluation questions through the server:

```bash
yobro run project.yobro --serve base=http://localhost:8080/v1#your-3b
```

> If your 3B model only exists as a GGUF file (for example in Ollama or llama.cpp), you can evaluate and monitor it, but **you cannot fine-tune a GGUF file**. Download the same model's Hugging Face weights for training. After training, `export "release@1" -> "gguf"` converts the result back to GGUF so llama.cpp or Ollama can run it.

## Quick start

```bash
yobro new myproject --model /path/to/your-3b-model
cd myproject
yobro check project.yobro            # fix anything marked [error]
yobro preview project.yobro --rows 3 # see rows exactly as the model will see them
yobro run project.yobro
yobro report r1                      # print the saved report
```

The starter project has only 30 example rows, so the release gate will refuse to ship. That is deliberate. Replace `data/*.jsonl` with your own data.

## Data formats (JSONL, one object per line)

Chat, the main format:

```json
{"id": "t1", "messages": [{"role": "system", "content": "You are a support agent."},
                          {"role": "user", "content": "How do I reset my password?"},
                          {"role": "assistant", "content": "Open Settings, then Security."}],
 "meta": {"topic": "account"}}
```

Instruction: `{"id", "instruction", "input", "output", "meta"}`. Preference (for DPO): `{"id", "prompt", "chosen", "rejected", "meta"}`.

`meta` is free-form. `meta.topic` (or any field you name in `slices = by("...")`) gives per-slice results. `meta.source = "generated"` marks model-written rows, and `generated_share()` caps their share. Safety probe rows use `meta.expected = "refuse"` or `"comply"`.

JSON, CSV and Parquet are also read, as is `hf://owner/dataset:split` when the `datasets` package is installed.

Traces for monitoring:

```json
{"trace_id": "t-001", "time": "2026-10-08T10:00:00Z", "model": "support-bot@1",
 "messages": [{"role": "user", "content": "..."}], "output": "...",
 "feedback": {"rating": -1, "correction": "the right answer"}, "meta": {"topic": "billing"}}
```

OpenTelemetry JSON span exports with `gen_ai.*` attributes are also read, best effort.

## The language

Thirteen statements. Names are bound once; a new version gets a new name.

| Statement | Example |
|---|---|
| `model` | `model base = "/models/my-3b" { params = "3B"; max_length = 2048 }` |
| `data` | `data support: Chat = load("data/support.jsonl")` · `data all = a + b tagged "generated"` · `data live = traces("logs.jsonl")` |
| `split` | `split support -> train, val, test = 80%, 10%, 10% { seed = 1 }` · `... = 90%, 10%, reuse(test)` |
| `check` | `check train { no_duplicates(near = true)  no_overlap_with(val, test) }` |
| `tune` | `tune v1 = lora(base) { data = train; eval = val; rank = 64; stop = early(patience = 3) }` (also `qlora`, `full`, `dpo`) |
| `report` | `report r1 = compare(base, v1) on test { task = token_f1(); slices = by("topic") }` |
| `gate` | `gate ok = r1 { min_test_rows = 300; v1.task.diff_low >= 0.03 }` |
| `ship` | `ship v1 -> "support-bot@1" gated_by ok` |
| `export` | `export "support-bot@1" -> "gguf"` (also `"merged"`, `"adapter"`) |
| `monitor` | `monitor m1 on live { sample = 5%; score = user_feedback(); alert = score_drop(over = "7d", by = 0.1) }` |
| `review` | `review fixes = groups { assign = "support-team" }` |
| `use` | `use python "my_company.metrics" as mc` then `task = mc.policy_score` |
| `eval` | Written inside `tune` as `eval = val`. Every training run needs one. |

Rules the compiler enforces:

- A `tune` without `eval` does not compile.
- A test set cannot be used as training or evaluation data.
- A `report` on training data does not compile.
- `ship` without `gated_by` does not compile.
- A name cannot be bound twice, and a setting cannot be written twice.
- Unknown settings and misspelled arguments (`smaple`, `early(patiance = 3)`) are errors, not silently ignored.
- Training settings are checked before anything runs: a rank of -4, a learning rate of 0 or `epochs = "two"` do not compile.
- Every gate condition is a comparison that mentions a model; `v1.task >= 0.5` (no field) or `1 >= 0` do not compile.
- Release names may only use letters, digits, `.`, `_`, `-` and one `@version`, because they become folder names on export.

What `ship` refuses at run time:

- a gate that failed;
- a model whose scores none of the gate's conditions tested (a gate on `v2` cannot ship `v1`, and a row count such as `v1.task.n` does not count). In Python, built-in rules such as `improves("v1")` record their model; mark a custom rule with `yobro.gate.tests("v1", rule)`;
- a model that was evaluated through a replacement such as `--serve v1=...`, since the gate did not see the weights that would ship;
- a dry run (`--backend dry`): the gate is rehearsed, nothing is recorded;
- a release name that already points at different weights. Shipping the same weights again does nothing; new weights need a new version (`support-bot@2`).

Every error says where the problem is, what is wrong, and how to fix it.

**Checks:** `schema`, `min_rows(n)`, `no_duplicates(near)`, `no_overlap_with(...)`, `no_empty_answers`, `label_conflicts`, `encoding`, `personal_data`, `generated_share(max_share)`, `template_matches(model)`, `truncation(model, max_length)`. `schema`, `encoding`, `no_empty_answers`, `label_conflicts` and `personal_data` always run.

**Metrics:** `exact_match()`, `contains()`, `token_f1()`, `json_valid(keys = [...])`, `choice()` (multiple choice), `behaviour()` (refuse or comply), `judge(model = ..., rubric = "file.md")`, `user_feedback()`, or any Python function `f(output, reference[, row]) -> float`.

**Report settings:** `general = forgetting_check(data = "general.jsonl", sample = 200)` (`sample = 200` keeps 200 rows; `sample = 10%` keeps a tenth), `general_metric = choice()`, `safety = safety_probe(data = "safety.jsonl")`, `slices = by("topic")`, `labels = "human_labels.jsonl"` (checks judge accuracy; one `{"id": ..., "label": 1}` per line, where a label is 1/0, true/false or "pass"/"fail"), `min_rows`, `min_slice_rows`. Below 5 rows nothing is judged, whatever `min_rows` says.

A judge rubric can be text or a file. If it looks like a file name (`rubrics/policy.md`) and the file is missing, that is an error rather than a rubric made of the file name.

`monitor ... { sample = 5% }` scores a share of traces; `sample = 200` scores up to 200. Trace times can be ISO 8601 (offsets are converted to UTC) or Unix timestamps in seconds, milliseconds or nanoseconds.

**Gate fields:** `<model>.<metric>.score | low | high | diff | diff_low`, `<model>.general.drop`, `<model>.safety.drop`, `<model>[slice].score`, and `every s in r1.slices: ...`. Intervals are 95% paired bootstrap intervals over test rows. Slices smaller than `min_slice_rows` are not judged.

See [`examples/full_lifecycle.yobro`](examples/full_lifecycle.yobro) for the whole loop, and [`examples/python_api.py`](examples/python_api.py) for the same project in Python.

## Commands

| Command | What it does |
|---|---|
| `yobro new DIR [--model PATH]` | Starter project |
| `yobro check FILE` | Checks, estimates and training plans; no GPU |
| `yobro preview FILE --data train --rows 3` | Rows rendered with the model's chat template |
| `yobro run FILE [--backend dry] [--all] [--force] [--retrain] [--fresh]` | Run the project. `dry` trains nothing; `--retrain` trains a new attempt without overwriting earlier weights; `--fresh` ignores cached model answers |
| `yobro monitor FILE` | Monitoring, failure clustering and review queue |
| `yobro review .yobro/review/fixes.jsonl` | Approve or correct queued failures |
| `yobro report NAME [--project P]` · `yobro releases` | Saved reports and shipped releases |
| `yobro estimate --params 3B --gpu-memory 12` | Rough memory estimate |
| `yobro export-config FILE --tune v1` | The run as an Axolotl config |
| `yobro bench answer_key.json [--model PATH]` | Score the checks on a bundle of planted bugs (see `DATASET_PROMPT.md`) |
| `yobro calibrate-judge FILE --url URL --model NAME --rubric FILE` | How often a served judge agrees with your labels |
| `yobro doctor` | Installation and hardware |

Everything is stored in `.yobro/` next to your file: datasets by hash, runs with `manifest.json` (data hashes, settings, seed, library versions, hardware), reports (under `reports/<project file name>/`, so two projects in one folder never overwrite each other), releases (each records the hash of the report that allowed it), caches and review queues. Nothing leaves your machine unless your program sends it to a server.

`yobro check` and `yobro run` exit with code 1 when a blocking check fails, and 2 on any other error, so they can gate a CI job.

Answers are cached only for Yobro's own local and served generators, whose identity covers the weights, template and server. Answers from a Python function or your own `Generator` class are produced fresh each run (set `cacheable = True` on your class if its `ident()` changes whenever its answers could).

## What is tested and what is not

157 automated tests pass (`pytest`). 19 of them guard against 15 bugs found by an independent review, and 81 against more than 50 problems found in a stress test and its follow-up verification (all on 8 October 2026); all are fixed. The stress test covered 18,000 randomly broken `.yobro` files, about 70 hand-written bad programs, 25 malformed data files, data sets of up to 50,000 rows, malformed server replies, odd trace files and every CLI command with bad input. The training tests use a tiny randomly initialised model built locally, on CPU.

| Area | Status |
|---|---|
| Data loading, splits, all checks, the planted-bug suite, dedupe | Tested. Files with a byte-order mark load; non-UTF-8 files give a clear error. TRL's conversational preference rows and OpenAI-style text parts are accepted |
| Near-duplicate speed | 10,000 rows in about 11 s and 50,000 in about 2 minutes on one CPU core (realistic word distribution). Data with a tiny vocabulary is slower: 30,000 rows took about 100 s |
| Language parser, compiler errors, gates, ship, releases | Tested |
| Evaluation, confidence intervals, slices, forgetting, safety, judge agreement | Tested with stand-in models |
| LoRA and DPO training through TRL and PEFT, adapter loading, merging | Tested on a tiny model, CPU only |
| OpenAI-compatible server connection | Tested against fake local servers, including broken replies (not JSON, no choices, error objects, null or part-list content) |
| Monitoring, alerts, clustering, nearest training rows, review queue, dashboard | Tested on synthetic traces |
| **A real 3B run on a GPU** | **Not run.** This is your first job. |
| QLoRA (4-bit) | Not tested; needs an NVIDIA GPU and bitsandbytes |
| GGUF export | Not tested; needs llama.cpp (`LLAMA_CPP_DIR`) |
| Memory and time estimates | Formula-based, not checked against real runs |
| OpenTelemetry trace reading | Best effort; one export per file or one per line, with `content` or `parts`. Not tested against real exporters |
| Axolotl config export | Field names not verified against your Axolotl version |
| Heavily paraphrased leaks | Not caught; near-duplicate checks catch copies, light rewording and added greetings or filler only |
| The planted-bug bench | 18 of 18 bug types caught, 0 false alarms, on a synthetic mini bundle; real-language accuracy is unknown until you run the bundle from `DATASET_PROMPT.md` |
| Alert thresholds | Examples, not validated defaults |
| Real Llama 3, Qwen, Gemma and Phi tokenizers | Not available offline here; the template checks were tested with stand-in tokenizers. Run `yobro check` on your model first |

Strings in `.yobro` files treat only `\"`, `\\`, `\n` and `\t` as escapes. On Windows, use forward slashes in paths (`C:/models/my-3b`) to avoid `\n` or `\t` surprises.

## Test datasets

`DATASET_PROMPT.md` is a prompt to give an AI assistant. It produces a full evaluation bundle covering:
- planted data bugs with an answer key
- clean train, val and test sets with a deliberately small slice
- forgetting and safety sets
- judge calibration answers
- structured-output rows
- preference pairs
- six weeks of drifting production traces

`yobro bench` scores the checks on the bundle; the rest feeds `yobro run` and `yobro monitor`.

## Development

```bash
pip install -e ".[train,dev]"
pytest -q
```

Licence: Apache-2.0.
