Metadata-Version: 2.4
Name: hulder
Version: 0.1.0
Summary: Release gate for thresholded scores, such as activation probes, across LLM serving configs
Keywords: activation probes,llm,vllm,quantization,release gate,mlops,monitoring
Author: Utku Özbek
License-Expression: Apache-2.0
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Dist: numpy>=2.3
Requires-Dist: scipy>=1.16
Requires-Dist: scikit-learn>=1.7 ; extra == 'train'
Requires-Dist: vllm==0.30.0 ; extra == 'vllm'
Requires-Dist: pyyaml>=6.0.3 ; extra == 'vllm'
Requires-Dist: huggingface-hub>=1.33.0 ; extra == 'vllm'
Requires-Python: >=3.12
Project-URL: Source, https://github.com/revtheundead/hulder
Project-URL: Issues, https://github.com/revtheundead/hulder/issues
Provides-Extra: train
Provides-Extra: vllm
Description-Content-Type: text/markdown

# Hulder

[![CI](https://github.com/revtheundead/hulder/actions/workflows/ci.yml/badge.svg)](https://github.com/revtheundead/hulder/actions/workflows/ci.yml)

A release gate for activation probes, and any other thresholded scorer, across LLM serving configs.

Teams tune a probe's threshold on one serving config, then change it: FP8 or AWQ weights, an FP8 KV cache, a vLLM upgrade. The hidden states move, and the threshold silently stops giving its target false-positive rate. `hulder check` compares a reference and a candidate on the same labelled examples and returns PASS, THRESHOLD STALE, SCORER DEGRADED or INCONCLUSIVE, with confidence intervals and a CI exit code.

## Install

```bash
pip install hulder
```

The base install is numpy and scipy only and runs on CPU. Capturing activations from vLLM needs the `vllm` extra (`pip install "hulder[vllm]"`), which adds `hulder.capture`, experimental in v0.1.

## Quickstart

```bash
hulder quickstart
```

It runs two checks on bundled data with a demo probe, under a minute on CPU. An fp16 rerun as candidate gives PASS. The official AWQ int4 checkpoint of Qwen2.5-1.5B-Instruct gives INCONCLUSIVE, with the shift flagged against the rerun's noise floor. `hulder check --help` lists the options.

v0.1 is an early release. It has run only on a Turing GPU in fp16 and AWQ, and its tolerance bands are fixed.

## More

Design, statistical method, results and the cases where it loses: https://github.com/revtheundead/hulder

Apache-2.0. The bundled demo data is derived from a CC-BY-4.0 dataset, attributed in `hulder/data/NOTICE.md`.
