Metadata-Version: 2.4
Name: separateai
Version: 0.1.0
Summary: A Together-shaped client that trains and serves fine-tuned models on your own Vast.ai GPUs
Author: Shahzoda
License-Expression: MIT
Project-URL: Homepage, https://github.com/Shahzzoda/separateai
Project-URL: Repository, https://github.com/Shahzzoda/separateai
Project-URL: Issues, https://github.com/Shahzzoda/separateai/issues
Keywords: fine-tuning,lora,vast.ai,vllm,unsloth,together
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: vastai<2.0,>=0.2
Dynamic: license-file

# separateai

A **Together-shaped** Python client that trains and serves fine-tuned models on
your own **Vast.ai** GPUs. Swap `from together import Together` →
`from separateai import Separate` and your existing fine-tuning + hosting code
runs on rented hardware instead of a managed service.

Scope is deliberately narrow: **train custom fine-tunes (QLoRA) and host them**.
It is not a general Together replacement — no embeddings, no image gen, no
serverless inference over stock models.

```python
from separateai import Separate

client = Separate(api_key=VAST_API_KEY, hf_token=HF_TOKEN)

# ── train ───────────────────────────────────────────────────────────────
ref = client.files.upload(open("data.jsonl"))          # str / bytes / path / file
job = client.fine_tuning.create(
    training_file=ref,
    hyperparameters={"n_epochs": 20, "learning_rate": 1e-4, "lora_r": 64},
)
status = client.fine_tuning.retrieve(job.id)           # .status, .steps_completed…
# job.output_name is the private HF repo the trained adapter is pushed to

# ── host + chat ─────────────────────────────────────────────────────────
ep = client.endpoints.ensure(model="unsloth/gemma-4-E4B-it",
                             adapter=job.output_name)  # blocks until STARTED
out = client.chat.completions.create(
    messages=[{"role": "user", "content": "hi"}],
    model="adapter", endpoint_id=ep.id)
print(out.choices[0].message.content)
client.endpoints.stop(ep.id)   # endpoints do NOT idle-stop — always stop when done
```

## Install

```bash
pip install separateai
```

The `vastai` CLI arrives as a pip dependency — no separate install. Credentials
come from constructor args or env vars, never hardcoded:

| var | what for |
| --- | --- |
| `VAST_API_KEY` | renting/destroying GPU boxes (the `vastai` CLI is authed automatically — note: the key is written once to `~/.config/vastai/vast_api_key`, the CLI's own key file) |
| `HF_TOKEN` | pulling base models on the box + persisting/serving trained adapters |

## How it works

- `fine_tuning.create()` rents the cheapest matching Vast box, launches a pinned
  Unsloth QLoRA trainer image (`shahzodad/trainer` — built from
  [`trainer/`](https://github.com/Shahzzoda/separateai/tree/main/trainer)
  in this repo, so you can audit exactly what your `HF_TOKEN` is handed to),
  and appends an upload step so the trained adapter lands in a **private HF
  repo** before the box dies. The Vast instance id is the job id.
- `fine_tuning.retrieve()` polls instance state + scrapes container logs into a
  Together-shaped status (`pending → running → completed/failed`). A job only
  reads `completed` after the adapter upload finished — and on the first
  terminal status the box is **destroyed automatically** so it stops billing.
- `endpoints.create()/ensure()` rents a box running `vllm/vllm-openai`
  (OpenAI-compatible) serving a base model + your LoRA adapter. The endpoint is
  protected with a bearer token derived from your Vast key (vLLM `--api-key`),
  so a public-IP box doesn't hand out free inference; `chat.completions.create()`
  sends it automatically.

## Cost model (read this)

You are renting real GPUs by the hour:
- training boxes are destroyed automatically the first time `retrieve()` sees a
  terminal status (or explicitly via `fine_tuning.cancel()`) — but a job you
  never poll again is a box you keep paying for;
- serving boxes have **no idle auto-stop** and bill until you call
  `endpoints.stop()`.

`endpoints.ensure()` destroys the box itself if startup fails or times out, so
a failed launch never bills idle.

## Repo layout

```
separateai/   the Python SDK (what `pip install separateai` gives you)
tests/        offline unit tests — every Vast call mocked, nothing rented
trainer/      Dockerfile + entrypoint for shahzodad/trainer, the custom
              image every TRAINING box runs (audit/rebuild it here)
```

Note the asymmetry: only training needs a custom image. HOSTING boxes run the
official upstream `vllm/vllm-openai` image as-is — all serving behaviour
(bearer-token auth, LoRA rank, adapter wiring) is set via the launch command,
so there is no hosting image to maintain or audit beyond upstream vLLM.

## Module map

| file | role |
| --- | --- |
| `client.py` | `Separate` facade — wires `.files`, `.fine_tuning`, `.endpoints`, `.chat` |
| `files.py` | `upload()` — stage the JSONL dataset (inline, base64 env) |
| `fine_tuning.py` | `create()` / `retrieve()` / `cancel()` |
| `serving.py` | `endpoints.*` + `chat.completions.create()` (vLLM on Vast) |
| `vast.py` | `vastai` CLI wrappers — the only file that talks to Vast |
| `hf.py` | minimal HF Hub HTTP helpers (private repos, small-file commits) |
| `logs.py` | container logs → `{state, steps, total}` |
| `config.py` | pinned images + GPU/CUDA/disk requirements (the hidden IaC) |
| `models.py` | `DatasetRef`, `FineTuneJob`, `Endpoint` (Together-shaped returns) |

## Security & trust model

Read this before running untrusted models or handling other people's data.

- **The rented GPU host can see your `HF_TOKEN`.** Training/serving boxes get the
  token in their container env (Vast even stores the launch command), and Vast
  hosts are third parties. Use a **fine-grained, repo-scoped, short-lived** HF
  token, not an account-wide one.
- **Inference traffic is plaintext HTTP.** Chat requests, the bearer token, and
  completions travel unencrypted to `http://<box-ip>:8000`. Don't send secrets
  in prompts over untrusted networks; front the endpoint with TLS (a tunnel /
  reverse proxy) if you need confidentiality.
- **Endpoints are protected by a bearer token derived from your Vast API key.**
  It keeps random internet scanners off your public-IP box; it is *not* secret
  from Vast or the host operator. A Vast API key is therefore **required** to
  serve — `endpoints.*` raises without one.
- **`model` / `adapter` must be trusted repo ids.** They are validated against
  the HF id charset before use, but a malicious *model repo* can still run code
  at load time — only point `fine_tuning.create(model=...)` / `endpoints` at
  repos you trust.
- **Images are digest-pinned** (`config.py`) so a repushed tag can't swap in a
  token-stealing image.

## Limits

- Small datasets ride inline in a container env var (Vast caps total env at
  32 KB); bigger ones are automatically committed to a private HF dataset repo
  and fetched on-box with `HF_TOKEN` — no caller-visible difference.
- One pinned base model family is tested end-to-end today (Gemma-4-E4B, 4-bit
  QLoRA on an RTX 4090). Other Unsloth-supported bases should be drop-in via
  `fine_tuning.create(model=...)`, but are not yet verified.
- Hyperparameters supported: `n_epochs`, `batch_size`, `learning_rate`,
  `lr_scheduler_type`, `warmup_ratio`, `weight_decay`, `max_grad_norm`,
  `lora_r`, `lora_alpha`, `lora_dropout`. Others are ignored.
