Metadata-Version: 2.4
Name: dataspires
Version: 0.1.0
Summary: DataSpires SDK — Train, finetune, and generate on GPUs from your notebook
Home-page: https://github.com/DataSpires/backend-server
Author: DataSpires
Author-email: DataSpires <info@dataspires.com>
License: MIT
Project-URL: Homepage, https://dataspires.com
Project-URL: Documentation, https://www.dataspires.com/docs
Project-URL: Repository, https://github.com/DataSpires/afrilink-sdk
Project-URL: Bug Tracker, https://github.com/DataSpires/afrilink-sdk/issues
Keywords: hpc,high-performance-computing,finetuning,pretrain,generate,vlm,llm,lora,notebook,gpu,dataspires,afrilink
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: System :: Distributed Computing
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.28.0
Provides-Extra: full
Requires-Dist: psutil>=5.9.0; extra == "full"
Provides-Extra: build
Requires-Dist: cryptography>=41.0.0; extra == "build"
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: black>=23.0.0; extra == "dev"
Requires-Dist: mypy>=1.0.0; extra == "dev"
Dynamic: author
Dynamic: home-page
Dynamic: license-file
Dynamic: requires-python

# DataSpires SDK

**Version:** 0.1.0

Train, finetune, and run inference on DataSpires GPUs from any notebook.

```bash
pip install dataspires
```

**Works in:** Google Colab · Kaggle · Jupyter · VS Code · any Python 3.8+ environment

---

## Contents

1. [60-second quickstart](#60-second-quickstart)
2. [Authentication](#authentication)
3. [Which method should I use?](#which-method-should-i-use)
4. [Backend](#backend)
5. [Guides by task](#guides-by-task)
   — [Pretrain](#pretrain) · [Finetune](#finetune) · [Generate](#generate)
   — [Checkpointing](#checkpointing-and-resuming)
6. [Working with your output model](#working-with-your-model)
7. [API reference](#api-reference)
8. [Hardware & billing](#hardware--billing)
9. [Model & dataset registry](#model--dataset-registry)
10. [Troubleshooting](#troubleshooting)
11. [Built-in help](#built-in-help)

---

## 60-second quickstart

```python
from dataspires import DataSpiresClient

client = DataSpiresClient()
client.authenticate(api_key="afk_live_…")  # or set API_KEY

job = client.pretrain(
    architecture="cnn",
    kind="resnet18",
    init="pretrained",
    data="./images/",
    config={"n_epoch": 5, "num_classes": 2, "seed": 42},
    gpus=1,
)
result = job.run(wait=True)
print(result["status"])  # "completed" | "failed" | "timeout" | …
if result["status"] != "completed":
    print(result.get("error") or result.get("reason"))
```

`finetune()` and `generate()` use the same `client → job → job.run()` pattern.

---

## Authentication

### Get a key

1. Sign up at [dataspires.com](https://dataspires.com).
2. **Profile → DataSpires SDK keys → Create new key.** Copy the `afk_live_…` value — it's shown once.
3. Store it as `API_KEY`.

### Set the key

| Environment | How |
| --- | --- |
| Google Colab | Secrets → `API_KEY` |
| Kaggle | Secrets → `API_KEY` |
| Local | `os.environ["API_KEY"] = "afk_live_…"` |
| Anywhere | `client.authenticate(api_key="afk_live_…")` |

```python
client = DataSpiresClient()
client.authenticate()
```

`authenticate()` raises on a bad or revoked key. Rotate a key by revoking it on the dashboard and creating a new one.

---

## Which method should I use?

| Your goal | Call | You provide |
| --- | --- | --- |
| Detector, CNN, or scratch transformer | `client.pretrain()` | `architecture`, `kind`, `data`, optional `init` / `config` |
| Fine-tune an LLM / VLM foundation model | `client.finetune()` | `model`, `training_mode`, `data`, optional `task=` |
| Run a model without changing weights | `client.generate()` | `model`, prompts / image+question manifest |

> Train from zero → `pretrain()`. Adapt existing weights → `finetune()`. Frozen answers → `generate()`.

---

## Backend

Jobs run on the DataSpires GPU cluster by default.

| | |
| --- | --- |
| **Job model** | Async — submit, then poll or `list_jobs()` |
| **Outputs** | `job.download()` and `client.download_model()` |
| **Multi-GPU** | Supported when the cluster has capacity |
| **Wall-clock max** | `time_limit` capped at **10 hours** |

### `status` vs `state`

| API | Field | Example values |
| --- | --- | --- |
| `job.run()` return dict | **`status`** | `submitted`, `completed`, `failed`, `timeout`, `cancelled` |
| `client.list_jobs()` / `client.get_job_status()` | **`state`** | same lifecycle strings; may also include `queued`, `running` |

```python
result = job.run(wait=True)
print(result["status"])

for entry in client.list_jobs():
    print(entry["job_id"], entry["state"], entry.get("reason"))
```

---

## Guides by task

### Pretrain

```python
job = client.pretrain(
    architecture="cnn",
    kind="resnet18",
    init="pretrained",
    data="./images/",
    config={"n_epoch": 5, "num_classes": 2, "batch_size": 32, "seed": 42},
    gpus=1,
    time_limit="02:00:00",
)
result = job.run(wait=True)
rows = job.get_metrics()
job.export_metrics("./run.csv")
```

**Detector.** Metrics are loss only in v1.

```python
job = client.pretrain(
    architecture="detector",
    kind="detect",          # or segment, pose
    init="pretrained",
    weights="yolo11n.pt",
    data="./dataset/",
    data_config="data.yaml",
    config={"n_epoch": 100, "imgsz": 640, "batch_size": 16, "seed": 42},
)
```

**Decoder from scratch.** `tokenizer` is required for decoder kinds only.

```python
job = client.pretrain(
    architecture="transformer",
    kind="gpt-decoder-tiny",
    init="scratch",
    tokenizer="gpt2",
    data="./corpus/",
    config={"max_steps": 1000, "n_epoch": 1, "seed": 42},
)
```

| `architecture` | `kind` |
| --- | --- |
| `detector` | `detect`, `segment`, `pose` |
| `cnn` | `resnet18`, `resnet50` |
| `transformer` | `gpt-decoder-tiny`, `qwen-decoder-tiny`, `llama-3-decoder-tiny`, `vit-tiny` |

`time_limit` is an `HH:MM:SS` string.

```python
job.download("./my-outputs")
job.download("./ckpts", include="checkpoints")
client.download_model(job.job_id, "./my-outputs")
```

### Finetune

```python
import pandas as pd

data = pd.DataFrame({"text": ["Below is an instruction...\n\n### Response:\n..."]})

job = client.finetune(
    model="qwen2.5-0.5b",
    training_mode="low",     # low | medium | high | custom
    data=data,
    gpus=1,
    time_limit="01:00:00",
)
result = job.run(wait=True)
if result["status"] == "completed":
    client.download_model(result["job_id"], "./my-model")
```

| Mode | Strategy | Precision |
| --- | --- | --- |
| `low` | QLoRA (rank 8) | 4-bit |
| `medium` | LoRA (rank 16) | 8-bit on 1 GPU, bf16 otherwise |
| `high` | High-rank LoRA (rank 64) | bf16 |
| `custom` | User-defined (see `config=`) | `bf16`, `fp16`, `4bit`, or `8bit` |

`training_mode="custom"` requires `config=`. Presets reject `config`. `gpus` and `time_limit` stay outside `config`.

```python
job = client.finetune(
    model="qwen2.5-0.5b",
    training_mode="custom",
    data=data,
    config={
        "batch_size": 2,
        "learning_rate": 2e-4,
        "scheduler": "cosine",
        "n_epoch": 3,
        "precision": "bf16",
        "lora": {"r": 16, "alpha": 32, "dropout": 0.05},
        "max_seq_length": 2048,
        "optimizer": "adamw_torch",
        "n_checkpoint": 100,
        "n_evals": 0,
        "seed": 42,
    },
)
rows = job.get_metrics()
job.export_metrics("./run_metrics.csv")
```

Causal-LM accuracy is token accuracy, not a task score. If `n_evals` is above 0 and there is no eval split, the SDK fails before submit.

For VLMs, pass `task=` (captioning / VQA) with image+text data. See `dataspires.docs("finetune")`.

Optional: `checkpoint={"every": 200, "keep": 2}` and `resume_from=` (prior job, job id, or `{job_id, step}`).

### Checkpointing and resuming

Jobs honour `time_limit` (capped at 10 hours). A stopped run can be continued with `resume_from=`.

```python
job = client.finetune(
    model="qwen2.5-0.5b",
    training_mode="low",
    data=data,
    time_limit="08:00:00",
    checkpoint={"every": 200, "keep": 2},
)
result = job.run(wait=True)

if result["status"] != "completed":
    job2 = client.finetune(
        model="qwen2.5-0.5b",
        training_mode="low",
        data=data,
        resume_from=job,
    )
    job2.run(wait=True)
```

| Path | Meaning |
| --- | --- |
| `output/checkpoints/checkpoint-N/` | Mid-run snapshots |
| `output/final/` | Weights after a clean exit |

`client.list_checkpoints(job_id)` lists prefixes. See `dataspires.docs("checkpoints")`.

### Generate

*(Frozen inference — no weight updates)*

```python
job = client.generate(
    model="smolvlm-256m",
    data="./manifest_dir/",
    outputs=["text", "confidence"],
    max_new_tokens=64,
    temperature=0.0,
)
result = job.run(wait=True)
```

### Job failures

`job.run(wait=True)` returns a dict on failure instead of raising. Check `result["status"] == "completed"` before downloading. Auth and submit-time validation errors do raise.

---

## Working with your model

### Convert to GGUF for Ollama / llama.cpp

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B")
model = PeftModel.from_pretrained(base, "./my-model")
merged = model.merge_and_unload()
merged.save_pretrained("./my-model-merged")
AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B").save_pretrained("./my-model-merged")
```

Then convert with [llama.cpp](https://github.com/ggerganov/llama.cpp) `convert_hf_to_gguf.py`.

### Publish to Hugging Face Hub

```python
from huggingface_hub import HfApi
api = HfApi()
api.upload_folder(folder_path="./my-model", repo_id="you/my-model", repo_type="model")
```

---

## API reference

### `DataSpiresClient`

| Method | Description |
| --- | --- |
| `authenticate(api_key=None)` | Resolve key; raises on failure |
| `pretrain(architecture, kind, init, config, …)` | Returns a job — call `.run()` |
| `finetune(model, training_mode, data, config, …)` | Returns `FinetuneJob` — call `.run()` |
| `generate(model, data, outputs, …)` | Returns `GenerateJob` — call `.run()` |
| `download_model(job_id, local_dir, include=("final",))` | Download outputs |
| `list_checkpoints(job_id)` | List checkpoint prefixes |
| `upload_dataset(local_path, dataset_name=None)` | Optional separate staging |
| `list_available_models()` / `list_available_datasets()` | Registry lookups |
| `get_model_requirements(model, training_mode)` | GPU/memory recommendation |
| `cancel_job(job_id)` | Stop a job |
| `list_jobs(include_completed=False)` | List jobs → dicts with **`state`** |
| `get_job_status(job_id)` | Poll by ID → dict with **`state`** |

### Job objects

| Member | Description |
| --- | --- |
| `run(wait=True, poll_interval=30)` | Submit; with `wait=True`, return a result **dict** with **`status`** |
| `cancel()` | Stop the job |
| `get_logs(tail=100)` | Recent log lines |
| `get_metrics()` | Loss/acc rows |
| `export_metrics(path)` | Write metrics to `.csv` or `.json` |
| `download(local_dir, include=("final",))` | Download outputs |
| `checkpoints()` | List checkpoint prefixes |
| `estimated_cost_usd()` | Planning ceiling from `gpus × time_limit` |
| `status`, `job_id` | Current tracked state / job ID |

---

## Hardware & billing

Prefer small models and `training_mode="low"` for lighter runs.

| Model size | Mode | Fits on 1 GPU? |
| --- | --- | --- |
| 0.5B – 3B | low / medium | yes |
| 3B – 7B | low (QLoRA 4-bit) | yes |
| 7B | medium | yes |
| 13B | low | tight |
| 30B+ | low | unlikely |

**Limits:** **10h** `time_limit`; hard max **32 GPUs** (SDK warns above 8).

**Billing:** invoices at [dataspires.com/dashboard/billing](https://dataspires.com/dashboard/billing). GPU time is **$0.60/GPU-hr for an L4**. `job.estimated_cost_usd()` is a planning ceiling from `time_limit`, not the invoice.

---

## Model & dataset registry

```python
client.list_available_models()
client.list_available_models(size="tiny")
client.list_available_datasets()
client.get_model_requirements("qwen2.5-0.5b", "low")
```

---

## Troubleshooting

| Symptom | Likely cause | Fix |
| --- | --- | --- |
| `API key auth failed` | Bad / revoked key | Create a new key; set `API_KEY` |
| Job stuck in `queued` | Cluster at capacity | `client.list_jobs()`; wait or try later |
| `KeyError: 'status'` on `list_jobs()` | Wrong field | Use `entry["state"]` for list/status APIs |
| `KeyError: 'state'` on `job.run()` result | Wrong field | Use `result["status"]` for `job.run()` |
| Gated HF model fails | Missing token | Add `HUGGINGFACE_TOKEN` |
| `download_model()` returns nothing | No finals written | Confirm the run completed; try `include="checkpoints"` |
| `reason` is `DeadlineExceeded` | Hit `time_limit` | `resume_from=job` |

---

## Built-in help

```python
import dataspires
dataspires.docs("help")
dataspires.docs("finetune")
dataspires.docs("pretrain")
dataspires.docs("checkpoints")
```

`AfriLinkClient` and `import afrilink` remain available as aliases for existing notebooks.
