Metadata-Version: 2.4
Name: traintrack-ai
Version: 0.1.6
Summary: TrainTrack Client: PyTorch training-time evaluation
Author: TrainTrack Team
License: MIT
Project-URL: Homepage, https://github.com/traintrack/traintrack
Keywords: pytorch,llm,evaluation,training,ml
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: torch>=2.0.0
Requires-Dist: requests>=2.28.0
Requires-Dist: pydantic>=2.0.0
Provides-Extra: tinker
Requires-Dist: tinker; extra == "tinker"
Requires-Dist: tinker-cookbook; extra == "tinker"

# TrainTrack

**Training-time evaluation and win-rate tracking for LLMs.**

TrainTrack helps you monitor model behavior during training by running automated LLM-as-a-Judge evaluations on every checkpoint.

## Features

- 🚀 **Real-time Metrics**: Get immediate feedback on conciseness, helpfulness, and reasoning quality.
- 📊 **Win-rate Tracking**: Automatically track win-rates against an anchor checkpoint (baseline) or the previous step.
- 📚 **Built-in Benchmarks**: Integrated support for GPQA, MMLU-Pro, IFEval, and TruthfulQA.
- 🛠️ **Seamless Integration**: Works with standard PyTorch loops and HuggingFace Trainer.

## Quick Installation

```bash
pip install traintrack-ai
```

## Minimal Example

```python
from traintrack import TrainTrackHook

# 1. Initialize the hook
hook = TrainTrackHook(
    model=model,
    tokenizer=tokenizer,
    run_name="my-first-run",
    datasets=["reasoning", "helpfulness"]
)

# 2. Capture a baseline (optional)
hook.capture_anchor()

# 3. Add to your training loop
for step, batch in enumerate(train_dataloader):
    # ... training logic ...

    hook.step(step)
```

## Documentation

For full documentation and advanced configuration (custom metrics, rubrics, and category-based evaluation), visit:
[github.com/traintrack/traintrack](https://github.com/traintrack/traintrack)

## Tinker Integration

TrainTrack includes a Tinker inline evaluator integration that can be used with
`evaluator_builders`:

```python
from traintrack import BuildTrainTrackTinkerEvaluator

traintrack_builder = BuildTrainTrackTinkerEvaluator(
    run_name="my-tinker-run",
    categories=["reasoning", "helpfulness"],
    model_name="Qwen/Qwen2.5-1.5B-Instruct",
    eval_every_steps=10,
)

# Add `traintrack_builder` to your Tinker `evaluator_builders` list.
```

If you're using a low-level custom training loop, call:
`evaluator.evaluate_training_step(training_client=..., step=...)`
to let TrainTrack handle scheduling + snapshot + sampling + ingest.

For a runnable single-eval smoke test, see:
`Canary/examples/tinker_sampling_evaluator_example.py`

For low blocking overhead during training, wire a small frequent evaluator into
`evaluator_builders` and larger suites into `infrequent_evaluator_builders`.

For a full training run example (real training + eval every N steps), see:
`Canary/examples/tinker_supervised_traintrack_training.py`

For a low-level Tinker quickstart-style example (Pig Latin with manual
forward/backward updates + TrainTrack eval every N steps), see:
`Canary/examples/tinker_pig_latin_with_traintrack.py`

For a real NoRobots finetune with TrainTrack evaluation on the NoRobots test set
(helpfulness metric, criteria + pairwise_anchor), see:
`Canary/examples/tinker_norobots_traintrack_finetune.py`
