Metadata-Version: 2.5
Name: sysone
Version: 0.3.0
Summary: Fine-tune and calibrate system-one decision models with a scikit-learn style fit / predict interface.
Project-URL: Homepage, https://github.com/ForwardMetric/sysone
Project-URL: Repository, https://github.com/ForwardMetric/sysone
Project-URL: Issues, https://github.com/ForwardMetric/sysone/issues
Project-URL: Design, https://github.com/ForwardMetric/sysone/blob/main/docs/DESIGN.md
Author: Goutham Balaraman
License-Expression: MIT
License-File: LICENSE
Keywords: calibration,classification,decision models,fine-tuning,lora,modernbert,qlora,qwen,transformers
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: bitsandbytes>=0.45; sys_platform == 'linux' or sys_platform == 'win32'
Requires-Dist: huggingface-hub>=1.3
Requires-Dist: numpy>=1.26
Requires-Dist: peft>=0.13
Requires-Dist: safetensors>=0.4.3
Requires-Dist: torch>=2.8
Requires-Dist: transformers>=5.1
Provides-Extra: pandas
Requires-Dist: pandas>=2.0; extra == 'pandas'
Description-Content-Type: text/markdown

# sysone

Fine-tune and calibrate "system one" decision models: small transformers that answer typed
questions (choice, score, yes/no) about a piece of text in a single forward pass. You pick a base
model, call `fit` on labelled rows and `predict` on new ones. sysone handles tokenization,
position ids, attention masks, layer freezing, the decision head and the training loop.

Supported base models: **ModernBERT** (`EncoderModel`) and **Qwen3** (`DecoderModel`). Others can
be added by writing one architecture class.

## Install

```bash
pip install sysone
pip install "sysone[pandas]"   # also installs pandas, for the comparison tables of `compare`
```

On Linux and Windows this also installs bitsandbytes, for 4-bit training (`tuning="qlora"`).

## Quickstart

```python
from sysone import DecoderModel

questions = {
    "team":     {"type": "choice", "instructions": "Which team should handle it?",
                 "criteria": ["sales", "operations", "servicing"]},
    "severity": {"type": "score", "instructions": "How severe is it?",
                 "criteria": ["none", "minor", "major", "critical"]},
    "refund":   {"type": "noul", "instructions": "Is the customer asking for a refund?"},
}

rows = [
    {"state": "Hi, my order is stuck in the warehouse and our line is down.",
     "category": "operations",
     "questions": questions,
     "expected": {"team": "operations", "severity": 3, "refund": False}},
    # ... more rows; rows may mix different question sets
]

model = DecoderModel(tuning="head").fit(rows)

model.predict("I'd like my money back for the broken unit.", questions)
# {"answers": {"team": {"type": "choice", "choice": "servicing", "probabilities": {...}, "confidence": ...},
#              "severity": {"type": "score", "score": 1.2, "legend": {...}, ...},
#              "refund": {"type": "noul", "noul": 0.97, ...}},
#  "usage": {"input_tokens": 118, "output_tokens": 0}}

model.save("my-model")
model = DecoderModel.load("my-model")
```

To measure the model on held-out labelled rows, and compare it with simple baselines:

```python
from sysone import baselines, compare

results = baselines(test_rows, train=rows)      # random guess, average answer
results["trained"] = model.evaluate(test_rows)
compare(results)                                # accuracy per question type, as a table
```

Answers follow the Jev `/v1/systemone` response shape. Training data can be records like the
above, records plus a shared question catalog, a long DataFrame, or a `.jsonl` / `.csv` /
`.parquet` file. Labels go in `expected`, in a `label` inside each question, or as soft `gold`
probabilities.

Example notebooks, each comparing accuracy before and after training:

- [quickstart.ipynb](https://github.com/ForwardMetric/sysone/blob/main/examples/quickstart.ipynb): synthetic support tickets, a few minutes;
- bev-decision-150K, 18 domains, hard labels:
  [decoder_bev_decision.ipynb](https://github.com/ForwardMetric/sysone/blob/main/examples/decoder_bev_decision.ipynb) (Qwen3) and
  [encoder_bev_decision.ipynb](https://github.com/ForwardMetric/sysone/blob/main/examples/encoder_bev_decision.ipynb) (ModernBERT);
- typed-decisions, soft labels from several raters:
  [decoder_typed_decisions.ipynb](https://github.com/ForwardMetric/sysone/blob/main/examples/decoder_typed_decisions.ipynb) (Qwen3) and
  [encoder_typed_decisions.ipynb](https://github.com/ForwardMetric/sysone/blob/main/examples/encoder_typed_decisions.ipynb) (ModernBERT);
- measuring `ordinal_weight` for score questions on typed-decisions:
  [encoder_typed_decisions_ordinal_weight.ipynb](https://github.com/ForwardMetric/sysone/blob/main/examples/encoder_typed_decisions_ordinal_weight.ipynb).

Notebooks other than the quickstart are named `<model type>_<dataset>.ipynb`.

## Status

This is an early version (0.3). Training covers every `tuning` mode:

- `"head"`: the backbone is frozen, its outputs are computed once and cached (in memory, and on
  disk with `cache_dir`), and only the decision head trains;
- `"lora"`, `"qlora"` and `"full"`: adapters or whole layers of the backbone train along with the
  head. These run on a laptop GPU (Apple MPS) for small datasets; a CUDA GPU is recommended for
  full-size runs. `"qlora"` needs bitsandbytes and an NVIDIA GPU and falls back to `"lora"`
  elsewhere, with a warning. It has trained, saved and reloaded in real 4-bit on a Colab GPU.

`DecoderModel` (Qwen3) defaults to `"qlora"`. `EncoderModel` (ModernBERT) defaults to `"full"`:
adapters barely moved ModernBERT, while full tuning reached 0.79 test accuracy on typed-decisions.

Training stops at `epochs`, by early stopping, or at `target_loss`. A run that is interrupted
(Ctrl-C), or saved and loaded, resumes exactly where it stopped. With `checkpoint_dir`, `fit` also
saves the model when it ends or is interrupted, and every `checkpoint_every` steps if set, so a
disconnected notebook can pick the run up again; nothing is written to disk by default.

Score questions add an ordinal term to the loss (`ordinal_weight`), so a near miss costs less than
a far one. After training, probabilities are calibrated with temperatures fitted on the validation
slice, so `answer_confidence` matches how often answers are right; reports include Brier score and
expected calibration error, before and after calibration. `head="full"` is designed but not yet
implemented.

See [docs/DESIGN.md](https://github.com/ForwardMetric/sysone/blob/main/docs/DESIGN.md) for the full design: every parameter, the layouts,
architectures, heads, training and checkpoint format.

## Development

```bash
uv sync                  # or: pip install -e . with the dev dependencies
uv run pytest            # fast tests on tiny random-weight models, no downloads
uv run pytest -m slow    # end-to-end tests on the real base models (~1.8 GB download)
uv run ruff check src tests
```
