Metadata-Version: 2.3
Name: syvain-training-utils
Version: 0.0.331
Summary: Shared runtime, diagnostics, and checkpoint utilities for Syvain training runs
Requires-Dist: numpy>=2.5.1
Requires-Dist: obstore>=0.11.0,<0.12.0
Requires-Dist: pydantic>=2.13.4
Requires-Dist: torch>=2.13.0
Requires-Dist: pytest>=8.0.0 ; extra == 'dev'
Requires-Dist: ruff>=0.15.12 ; extra == 'dev'
Requires-Dist: ty>=0.0.34 ; extra == 'dev'
Requires-Python: >=3.14, <3.15
Provides-Extra: dev
Description-Content-Type: text/markdown

# syvain-training-utils

Internal [Syvain](https://syvain.com/) helpers for small, explicit ML training
runs. No secret sauce here, just shared runtime, device-diagnostic, and
checkpoint patterns.

## Install

```bash
uv add syvain-training-utils
```

## Runtime setup

```python
from syvain_training_utils import (
    generate_run_id,
    require_torch_compile_toolchain,
    select_device,
)

run_id = generate_run_id()
device = select_device()
require_torch_compile_toolchain()
```

## Device smoke test

```python
import json

from syvain_training_utils import run_device_smoke_test

report = run_device_smoke_test(require_cuda=True)
print(json.dumps({"smoke": report}, indent=2, sort_keys=True))
```

## Checkpoint a training run

```python
from syvain_training_utils import (
    StorageConfig,
    TrainingLoopState,
    load_training_checkpoint_if_available,
    save_model_checkpoint,
)

storage_config = StorageConfig(
    bucket="my-training-bucket",
    s3_base_url="https://t3.storage.dev",
    region="auto",
    access_key_id="...",
    secret_access_key="...",
)
checkpoint_base_path = "models/my-model"

loop_state = TrainingLoopState(
    global_step=global_step,
    curriculum_stage=curriculum_stage,
    curriculum_step=curriculum_step,
)

save_model_checkpoint(
    storage_config=storage_config,
    base_path=checkpoint_base_path,
    experiment_slug=experiment_slug,
    run_id=run_id,
    model=model,
    optimizer=optimizer,
    scheduler=scheduler,
    loop_state=loop_state,
    checkpoint_label=f"step-{global_step:012d}",
)

resume = load_training_checkpoint_if_available(
    storage_config=storage_config,
    base_path=checkpoint_base_path,
    model=model,
    optimizer=optimizer,
    scheduler=scheduler,
    device=device,
)
```

The library owns the object-store clients. Every transient retry opens a fresh
client, and expired Tigris multipart sessions restart the complete upload at the
same checkpoint key. Checkpoint bodies are written through a temporary local
file, uploaded with an adaptive multipart size, and downloaded with resumable
range reads. Ensure the machine has temporary disk capacity for one checkpoint.

The manifest remains a pointer to the current checkpoint key within the
configured bucket and also records its byte size, ETag, and SHA-256 digest. The
manifest is published only after the checkpoint upload succeeds. Loading
verifies the complete digest before deserializing the model, optional optimizer,
scheduler, and PyTorch RNG state.

Observation-only runs can pass `optimizer=None` and `scheduler=None` to both
checkpoint functions. Model parameters and buffers, PyTorch RNG state, run
identity, and loop progress are still saved and restored. A buffer-only module
needs no dummy parameter or optimizer. Save and resume must agree on whether
the optimizer and scheduler exist; incompatible configurations fail before
restoring model state. A scheduler always requires an optimizer. To load only
model state across configurations, use `load_model_checkpoint_from_key`.

The storage transport uses a 10-second connect timeout, a 60-second
read-inactivity timeout, and a 10-minute overall request timeout. A bounded
15-minute outer no-progress retry window owns recovery and client replacement.
