Metadata-Version: 2.4
Name: dataspires
Version: 0.1.5
Summary: DataSpires SDK — Train, finetune, and generate on GPUs from your notebook
Home-page: https://github.com/DataSpires/backend-server
Author: DataSpires
Author-email: DataSpires <info@dataspires.com>
License: MIT
Project-URL: Homepage, https://dataspires.com
Project-URL: Documentation, https://www.dataspires.com/docs
Project-URL: Repository, https://github.com/DataSpires/afrilink-sdk
Project-URL: Bug Tracker, https://github.com/DataSpires/afrilink-sdk/issues
Keywords: hpc,high-performance-computing,finetuning,pretrain,generate,vlm,llm,lora,notebook,gpu,dataspires,afrilink
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: System :: Distributed Computing
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.28.0
Provides-Extra: full
Requires-Dist: psutil>=5.9.0; extra == "full"
Provides-Extra: build
Requires-Dist: cryptography>=41.0.0; extra == "build"
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: black>=23.0.0; extra == "dev"
Requires-Dist: mypy>=1.0.0; extra == "dev"
Dynamic: author
Dynamic: home-page
Dynamic: license-file
Dynamic: requires-python

# DataSpires SDK

**Version:** 0.1.5

Train, fine-tune, and run models on DataSpires GPUs from any notebook.

```bash
pip install dataspires
```

**Works in:** Google Colab · Kaggle · Jupyter · VS Code · any Python 3.9+ environment

---

## Contents

1. [Quickstart](#quickstart)
2. [Authentication](#authentication)
3. [Which method should I use?](#which-method-should-i-use)
4. [Models & data sources](#models--data-sources)
5. [Resource guide](#resource-guide)
6. [Jobs](#jobs)
7. [Guides by task](#guides-by-task)
   — [Pretrain](#pretrain) · [Finetune](#finetune) · [Generate](#generate)
   — [Checkpointing](#checkpointing-and-resuming) · [Long-running jobs](#long-running-jobs)
8. [Working with your model](#working-with-your-model)
9. [API reference](#api-reference)
10. [Hardware & billing](#hardware--billing)
11. [Model & dataset registry](#model--dataset-registry)
12. [Troubleshooting](#troubleshooting)
13. [Built-in help](#built-in-help)

---

## Quickstart

Fine-tune a model on a Hugging Face dataset in a few lines:

```python
from dataspires import DataSpiresClient

client = DataSpiresClient()
client.authenticate()

job = client.finetune(
    model="qwen2.5-0.5b",
    training_mode="low",
    data="tatsu-lab/alpaca",
    gpus=1,
)
result = job.run(wait=True)

if result["status"] == "completed":
    job.download("./my-model")
else:
    print(result.get("error") or result.get("reason"))
```

`pretrain()` and `generate()` follow the same pattern: create a job, then call `job.run()`.

---

## Authentication

### Get an API key

1. Sign in at [dataspires.com](https://dataspires.com).
2. Go to **Profile → SDK keys → Create new key**, then copy the `dsp_live_…` value. It is shown only once.
3. Save it as `DATASPIRES_API_KEY`.

### Set the key

| Environment | How |
| --- | --- |
| Google Colab | Secrets → `DATASPIRES_API_KEY` |
| Kaggle | Add-ons → Secrets → `DATASPIRES_API_KEY` |
| Terminal | `export DATASPIRES_API_KEY=dsp_live_…` |
| In code | `client.authenticate(api_key="dsp_live_…")` |

```python
client = DataSpiresClient()
client.authenticate()
```

`authenticate()` raises an error if the key is invalid or revoked. To rotate a key, revoke it in the dashboard and create a new one.

### Gated Hugging Face models

Some registry models (for example Llama or Gemma) are gated on Hugging Face: you must accept the model's license on its Hugging Face page, then provide a token. `finetune()` and `generate()` check this before the job is submitted and raise a clear error if the model needs a token and none is found.

1. Accept the license at `https://huggingface.co/<model-id>` (see the model's `huggingface_id` in the registry, e.g. `client.list_models()`).
2. Create a token at [huggingface.co/settings/tokens](https://huggingface.co/settings/tokens).
3. Save it as `HUGGINGFACE_TOKEN`:

| Environment | How |
| --- | --- |
| Google Colab | Secrets → `HUGGINGFACE_TOKEN` |
| Kaggle | Add-ons → Secrets → `HUGGINGFACE_TOKEN` |
| Terminal | `export HUGGINGFACE_TOKEN=hf_…` |

---

## Which method should I use?

| Your goal | Method | You provide |
| --- | --- | --- |
| Train a detector, CNN, or transformer from scratch | `client.pretrain()` | `architecture`, `kind`, `data`, optional `init`, `config`, `tokenizer` |
| Adapt an existing LLM or vision-language model | `client.finetune()` | `model`, `training_mode`, `data`, optional `task` |
| Run a model without changing its weights | `client.generate()` | `model`, prompts or an image manifest |

---

## Models & data sources

Pass a registry name, a Hugging Face ID, a URL, a Google Drive file, or local data. Remote sources are downloaded directly by the job, so large datasets never pass through your notebook. Local files and DataFrames are uploaded for you.

### `model=`

| Input | Example |
| --- | --- |
| Registry name | `qwen2.5-0.5b`, `llama-3.2-1b` |
| Hugging Face ID (optional revision) | `Qwen/Qwen2.5-0.5B`, `Qwen/Qwen2.5-0.5B@main` |
| Hugging Face URL | `https://huggingface.co/Qwen/Qwen2.5-0.5B` |
| Google Drive file | `gdrive:FILE_ID` or a Drive share link |

### `data=`

| Input | Example |
| --- | --- |
| In-memory | Python list, pandas DataFrame, Hugging Face `Dataset` |
| Local file or folder | `./train.jsonl`, `./corpus/` |
| Hugging Face dataset | `tatsu-lab/alpaca` |
| Dataset with config | `Salesforce/wikitext#wikitext-2-raw-v1` |
| HTTPS file | `https://example.com/train.jsonl` (jsonl, csv, parquet, txt, zip) |
| Google Drive file | `gdrive:FILE_ID` or a Drive share link |

### `tokenizer=` (transformer pretraining)

Accepts `gpt2`, any Hugging Face tokenizer ID (optionally with `@revision`), a Hugging Face URL, or a Google Drive file containing tokenizer files.

### Preview how a source resolves

```python
from dataspires import resolve_model, resolve_data, resolve_tokenizer

resolve_model(model="Qwen/Qwen2.5-0.5B")
resolve_data(data="Salesforce/wikitext#wikitext-2-raw-v1")
resolve_tokenizer("gpt2")
```

### Dataset format (finetune)

A single `text` column works best. These common layouts are converted automatically:

| Columns | Converted to |
| --- | --- |
| `messages` (chat format) | `text` |
| `user` + `final` (optional `analysis`) | Instruction-style `text` |
| `instruction` + `output`, or `prompt` + `completion` | Instruction-style `text` |

### Google Drive

Pass a Drive file the same way as any other source:

```python
job = client.finetune(
    model="qwen2.5-0.5b",
    training_mode="low",
    data="gdrive:FILE_ID",
    gpus=1,
)
```

- **Shared files** (“Anyone with the link”) work immediately.
- **Private files** open a Google sign-in window the first time. After you approve read-only access, the job continues and future jobs reuse your sign-in.

For headless environments, set `GOOGLE_DRIVE_TOKEN` instead. To turn off the sign-in prompt, set `DATASPIRES_GDRIVE_AUTO_CONNECT=0`. To sign out, call `client.disconnect_google_drive()`.

To check a file before submitting a job:

```python
client.inspect_source("gdrive:FILE_ID")
```

### Current limitations

| Feature | Status |
| --- | --- |
| Google Drive folders | Share a `.zip` of the folder |
| Resume from a Hub, HTTPS, or Drive checkpoint | Coming soon — resume from a previous job today |
| Custom YOLO/CNN `weights=` from Hub or Drive | Coming soon — built-in weight names are supported |

---

## Resource guide

Check whether a model fits before you start a run:

```python
est = client.estimate_model("qwen2.5-0.5b", training_mode="low", gpus=1)
print(est["fits"], est["min_vram_gb"], est["parameters_b"])
```

| `fits` | Meaning |
| --- | --- |
| `True` | Expected to fit with this mode and GPU count |
| `False` | Likely too large — try `training_mode="low"` or more GPUs |
| `"unknown"` | Model size not known — start with `low` on 1 GPU |

If a configuration is unlikely to fit, the SDK shows a warning before submitting.

---

## Jobs

Jobs run asynchronously on the DataSpires GPU cluster.

| | |
| --- | --- |
| **Lifecycle** | Submit, then wait or poll with `list_jobs()` |
| **Status** | `status` and `state` are the same field. Values: `queued`, `running`, `submitted`, `completed`, `failed`, `timeout`, `cancelled`, `missing` |
| **Outputs** | `job.download()` or `client.download_model()` |
| **Multi-GPU** | Available on request with `gpus=` |
| **GPU type** | Optional. Pass `gpu_type=` as `"l4"`, `"v100"`, `"4090"`, or `"5090"`. Omit it to run on any eligible GPU. |
| **Maximum duration** | Set per workspace or cohort. Pass `time_limit="HH:MM:SS"` within that limit. |

---

## Guides by task

### Pretrain

**Image classification (CNN):**

```python
job = client.pretrain(
    architecture="cnn",
    kind="resnet18",
    init="pretrained",
    data="./images/",
    config={"n_epoch": 5, "num_classes": 2, "batch_size": 32, "seed": 42},
    gpus=1,
    time_limit="02:00:00",
)
result = job.run(wait=True)
job.export_metrics("./run.csv")
```

**Object detection:**

```python
job = client.pretrain(
    architecture="detector",
    kind="detect",          # or "segment", "pose"
    init="pretrained",
    weights="yolo11n.pt",
    data="./dataset/",
    data_config="data.yaml",
    config={"n_epoch": 100, "imgsz": 640, "batch_size": 16, "seed": 42},
)
```

Detector runs report training loss.

**Language model from scratch:**

```python
job = client.pretrain(
    architecture="transformer",
    kind="gpt-decoder-tiny",
    init="scratch",
    tokenizer="gpt2",
    data="roneneldan/TinyStories",   # or a local folder such as "./corpus/"
    config={"max_steps": 200, "n_epoch": 1, "batch_size": 2, "seed": 42},
)
print(job.estimate())
```

| `architecture` | `kind` |
| --- | --- |
| `detector` | `detect`, `segment`, `pose` |
| `cnn` | `resnet18`, `resnet50` |
| `transformer` | `gpt-decoder-tiny`, `qwen-decoder-tiny`, `llama-3-decoder-tiny`, `vit-tiny` |

Download results:

```python
job.download("./my-outputs")
job.download("./checkpoints", include="checkpoints")
```

### Finetune

```python
import pandas as pd

data = pd.DataFrame({"text": ["Below is an instruction...\n\n### Response:\n..."]})

job = client.finetune(
    model="qwen2.5-0.5b",
    training_mode="low",        # low | medium | high | custom
    data=data,
    gpus=1,
    time_limit="01:00:00",
)
result = job.run(wait=True)
if result["status"] == "completed":
    job.download("./my-model")
```

| Mode | Strategy | Precision |
| --- | --- | --- |
| `low` | QLoRA (rank 8) | 4-bit |
| `medium` | LoRA (rank 16) | 8-bit on 1 GPU, bf16 on more |
| `high` | LoRA (rank 64) | bf16 |
| `custom` | Your settings via `config=` | `bf16`, `fp16`, `4bit`, or `8bit` |

For full control, use `training_mode="custom"` with a `config`:

```python
job = client.finetune(
    model="qwen2.5-0.5b",
    training_mode="custom",
    data=data,
    config={
        "batch_size": 2,
        "learning_rate": 2e-4,
        "scheduler": "cosine",
        "n_epoch": 3,
        "precision": "bf16",
        "lora": {"r": 16, "alpha": 32, "dropout": 0.05},
        "max_seq_length": 2048,
        "optimizer": "adamw_torch",
        "n_checkpoint": 100,
        "n_evals": 0,
        "seed": 42,
    },
)
```

`gpus`, `gpu_type`, and `time_limit` are passed as arguments, not inside `config`. Reported accuracy is token-level accuracy.

When `n_evals` is greater than 0, choose how evaluation data is supplied:

**Hub dataset with a built-in validation split** — used automatically:

```python
job = client.finetune(
    model="qwen2.5-0.5b",
    training_mode="custom",
    data="glue",
    config={
        "batch_size": 2,
        "learning_rate": 2e-4,
        "scheduler": "cosine",
        "n_epoch": 1,
        "precision": "bf16",
        "n_evals": 50,
    },
)
```

**Local eval file or in-memory eval data:**

```python
job = client.finetune(
    model="qwen2.5-0.5b",
    training_mode="custom",
    data="./train.jsonl",
    eval_data="./eval.jsonl",  # or a DataFrame / list of rows
    config={
        "batch_size": 2,
        "learning_rate": 2e-4,
        "scheduler": "cosine",
        "n_epoch": 1,
        "precision": "bf16",
        "n_evals": 50,
    },
)
```

**Hold out part of the training data** (when there is no separate eval split):

```python
job = client.finetune(
    model="qwen2.5-0.5b",
    training_mode="low",
    data="tatsu-lab/alpaca",
    validation_split=0.1,
)
```

For vision-language models, pass `task="captioning"` or `task="vqa"` with image-text data. See `dataspires.docs("finetune")`.

### Checkpointing and resuming

Save checkpoints during a run, then continue from a previous job:

```python
job = client.finetune(
    model="qwen2.5-0.5b",
    training_mode="low",
    data=data,
    time_limit="08:00:00",
    checkpoint={"every": 200, "keep": 2},
)
result = job.run(wait=True)

if result["status"] != "completed":
    resumed = client.finetune(
        model="qwen2.5-0.5b",
        training_mode="low",
        data=data,
        resume_from=job,          # or a job ID
    )
    resumed.run(wait=True)
```

| Path | Contents |
| --- | --- |
| `output/checkpoints/checkpoint-N/` | Snapshots saved during training |
| `output/final/` | Final weights |

List checkpoints with `client.list_checkpoints(job_id)`. See `dataspires.docs("checkpoints")`.

### Generate

Run a model without changing its weights:

```python
job = client.generate(
    model="Qwen/Qwen2.5-0.5B",
    data=[{"text": "Write one short sentence about GPUs."}],
    max_new_tokens=48,
    gpus=1,
)
result = job.run(wait=True)

if result["status"] == "completed":
    job.download("./generate-output")
```

Generated text is saved to the output folder. Use `job.get_logs()` to view progress while the job runs.

With a vision-language model and a local image manifest:

```python
job = client.generate(
    model="smolvlm-256m",
    data="./manifest_dir/",
    outputs=["text", "confidence"],
    max_new_tokens=64,
    temperature=0.0,
)
result = job.run(wait=True)
if result["status"] == "completed":
    job.download("./vlm-output")
```

### Long-running jobs

Notebook sessions can end before a long job finishes. Submit without waiting, save the job ID, and check back later:

```python
job.run(wait=False)
job_id = job.job_id
```

```python
# Later, in the same or a new session
status = client.get_job_status(job_id)
print(status["state"])

if status["state"] == "completed":
    client.download_model(job_id, "./my-model")
```

Jobs keep running on the cluster after your notebook disconnects.

### Handling failures

`job.run(wait=True)` returns a result instead of raising when a job fails. Check `result["status"]` before downloading. Authentication and validation errors raise immediately.

---

## Working with your model

The steps below apply to `finetune()` outputs. Fine-tuning produces a **LoRA adapter**, which is loaded on top of the base model.

### Convert to GGUF for Ollama or llama.cpp

Merge the adapter into the base model first:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B")
model = PeftModel.from_pretrained(base, "./my-model")
merged = model.merge_and_unload()
merged.save_pretrained("./my-model-merged")
AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B").save_pretrained("./my-model-merged")
```

Then convert `./my-model-merged` with [llama.cpp](https://github.com/ggerganov/llama.cpp)'s `convert_hf_to_gguf.py`.

### Publish to Hugging Face Hub

```python
from huggingface_hub import HfApi

api = HfApi()

# Publish the adapter
api.upload_folder(folder_path="./my-model", repo_id="you/my-adapter", repo_type="model")

# Or publish full merged weights
api.upload_folder(folder_path="./my-model-merged", repo_id="you/my-model", repo_type="model")
```

---

## API reference

### `DataSpiresClient`

| Method | Description |
| --- | --- |
| `authenticate(api_key=None)` | Authenticate with your API key |
| `pretrain(architecture, kind, init, config, gpu_type, …)` | Create a pretraining job |
| `finetune(model, training_mode, data, config, gpu_type, …)` | Create a fine-tuning job |
| `generate(model, data, outputs, gpu_type, …)` | Create an inference job |
| `estimate_model(model, training_mode, gpus)` | Estimate memory and fit |
| `inspect_source(ref)` | Check how a source resolves and whether it is accessible |
| `connect_google_drive()` / `disconnect_google_drive()` | Sign in to or out of Google Drive |
| `download_model(job_id, local_dir, include=("final",))` | Download job outputs |
| `list_checkpoints(job_id)` | List saved checkpoints |
| `upload_dataset(local_path, dataset_name=None)` | Upload a dataset ahead of time |
| `list_available_models()` / `list_available_datasets()` | Browse the registry |
| `get_model_requirements(model, training_mode)` | Recommended GPUs and memory |
| `cancel_job(job_id)` | Cancel a job |
| `list_jobs(include_completed=False)` | List your jobs |
| `get_job_status(job_id)` | Get the status of a job |

### Job objects

| Member | Description |
| --- | --- |
| `run(wait=True, poll_interval=30)` | Submit the job; with `wait=True`, block until it finishes |
| `cancel()` | Cancel the job |
| `get_logs(tail=100)` | Recent log lines |
| `get_metrics()` | Training metrics |
| `export_metrics(path)` | Save metrics as `.csv` or `.json` |
| `download(local_dir, include=("final",))` | Download outputs |
| `checkpoints()` | List saved checkpoints |
| `estimated_cost_usd()` | Maximum cost based on `gpus` and `time_limit` |
| `job_id`, `status` | Job ID and current status |

Fine-tuning jobs also provide `validate()`. Pretraining jobs provide `estimate()`.

### Source helpers

| Helper | Description |
| --- | --- |
| `resolve_model` | Resolve a model reference |
| `resolve_data` | Resolve a data reference |
| `resolve_tokenizer` | Resolve a tokenizer reference |
| `estimate_model` | Same as `client.estimate_model` |

---

## Hardware & billing

Start with a small model and `training_mode="low"`, and use [`estimate_model`](#resource-guide) before trying larger models.

| Model size | Mode | Single GPU |
| --- | --- | --- |
| 0.5B – 3B | low, medium | Yes |
| 3B – 7B | low | Yes |
| 7B | medium | Yes |
| 13B | low | Tight |
| 30B+ | low | Unlikely |

Jobs can use up to 32 GPUs. Maximum duration is set per workspace or cohort.

Pass `gpu_type` on `pretrain()`, `finetune()`, or `generate()` to pin a job to one hardware type. Omit it to run on any eligible GPU. Any other value is rejected.

| `gpu_type` | Hardware |
| --- | --- |
| `"l4"` | NVIDIA L4 |
| `"v100"` | NVIDIA V100 |
| `"4090"` | NVIDIA RTX 4090 |
| `"5090"` | NVIDIA RTX 5090 |

```python
job = client.finetune(
    model="qwen2.5-0.5b",
    training_mode="low",
    data=data,
    gpus=1,
    gpu_type="l4",
)
```

GPU time is billed at **$0.60 per GPU-hour (NVIDIA L4)**. View invoices at [dataspires.com/dashboard/billing](https://dataspires.com/dashboard/billing). `job.estimated_cost_usd()` shows the maximum cost for a job's `time_limit`; you are billed for actual usage.

---

## Model & dataset registry

Browse curated models and datasets:

```python
client.list_available_models()
client.list_available_models(size="tiny")
client.list_available_datasets()
client.get_model_requirements("qwen2.5-0.5b", "low")
```

Any public Hugging Face model or dataset also works. See [Models & data sources](#models--data-sources).

---

## Troubleshooting

| Issue | Cause | Solution |
| --- | --- | --- |
| `API key auth failed` | Invalid or revoked key | Create a new key and update `DATASPIRES_API_KEY` |
| Job stays `queued` | Cluster at capacity | The job starts automatically when capacity is available |
| `Dataset has no 'text' column` | Unsupported format | Add a `text` column or use a [supported layout](#dataset-format-finetune) |
| Drive file not accessible | File is private or not shared with you | Sign in when prompted, or share the file as “Anyone with the link” |
| Drive folder link rejected | Folders are not supported | Share a `.zip` of the folder |
| Google sign-in unavailable | Drive sign-in is not configured in this environment | Set `GOOGLE_DRIVE_TOKEN`, or see [Google Drive setup](../docs/google-drive-oauth-setup.md) |
| HTTPS download fails | URL is not publicly reachable | Use a public `https://` URL |
| Gated Hugging Face model fails | Missing token | Set `HUGGINGFACE_TOKEN` |
| `download_model()` returns nothing | Job did not finish | Check the status, or download with `include="checkpoints"` |
| Job ends with `DeadlineExceeded` | Reached `time_limit` | Resume with `resume_from=job` |
| Notebook disconnected during `wait=True` | Session ended | See [Long-running jobs](#long-running-jobs) |

---

## Built-in help

```python
import dataspires

dataspires.docs("help")
dataspires.docs("finetune")
dataspires.docs("pretrain")
dataspires.docs("checkpoints")
```
