Metadata-Version: 2.5
Name: inferweave
Version: 0.4.0
Summary: Unified AI inference deployment SDK across cloud GPU providers
Project-URL: Homepage, https://github.com/AmaLS367/InferWeave
Project-URL: Repository, https://github.com/AmaLS367/InferWeave
Project-URL: Issues, https://github.com/AmaLS367/InferWeave/issues
Project-URL: Changelog, https://github.com/AmaLS367/InferWeave/blob/master/CHANGELOG.md
Author: AmaLS367
License-Expression: Apache-2.0
License-File: LICENSE
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: <3.13,>=3.11
Requires-Dist: credweave<0.2,>=0.1.0
Requires-Dist: httpx<1.0,>=0.28.1
Requires-Dist: msgpack<2.0,>=1.1.0
Requires-Dist: pydantic>=2.10.0
Requires-Dist: pyyaml>=6.0.2
Requires-Dist: rich>=13.0.0
Requires-Dist: typer>=0.12.0
Provides-Extra: all
Requires-Dist: fastapi>=0.110.0; extra == 'all'
Requires-Dist: modal<1.7,>=1.6; extra == 'all'
Requires-Dist: skypilot[aws,azure,cudo,fluidstack,gcp,ibm,kubernetes,lambda,nebius,oci,paperspace,runpod,scp,seeweb,shadeform,vast,verda]<0.14,>=0.13.0; extra == 'all'
Requires-Dist: uvicorn>=0.28.0; extra == 'all'
Provides-Extra: aws
Requires-Dist: skypilot[aws]<0.14,>=0.13.0; extra == 'aws'
Provides-Extra: azure
Requires-Dist: skypilot[azure]<0.14,>=0.13.0; extra == 'azure'
Provides-Extra: clouds
Requires-Dist: skypilot[aws,azure,cudo,fluidstack,gcp,ibm,kubernetes,lambda,nebius,oci,paperspace,runpod,scp,seeweb,shadeform,vast,verda]<0.14,>=0.13.0; extra == 'clouds'
Provides-Extra: gcp
Requires-Dist: skypilot[gcp]<0.14,>=0.13.0; extra == 'gcp'
Provides-Extra: kubernetes
Requires-Dist: skypilot[kubernetes]<0.14,>=0.13.0; extra == 'kubernetes'
Provides-Extra: lambda
Requires-Dist: skypilot[lambda]<0.14,>=0.13.0; extra == 'lambda'
Provides-Extra: lightning
Requires-Dist: lightning-sdk<2026.10.2,>=2026.10.1; extra == 'lightning'
Provides-Extra: modal
Requires-Dist: modal<1.7,>=1.6; extra == 'modal'
Provides-Extra: nebius
Requires-Dist: skypilot[nebius]<0.14,>=0.13.0; extra == 'nebius'
Provides-Extra: runpod
Requires-Dist: skypilot[runpod]<0.14,>=0.13.0; extra == 'runpod'
Provides-Extra: vast
Requires-Dist: skypilot[vast]<0.14,>=0.13.0; extra == 'vast'
Provides-Extra: workers
Requires-Dist: fastapi>=0.110.0; extra == 'workers'
Requires-Dist: uvicorn>=0.28.0; extra == 'workers'
Description-Content-Type: text/markdown

<p align="center">
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="assets/inferweave_banner.png">
    <source media="(prefers-color-scheme: light)" srcset="assets/inferweave_banner_light.png">
    <img alt="InferWeave Logo" src="assets/inferweave_banner.png" width="580">
  </picture>
</p>

<p align="center">
  <em>Unified AI inference deployment SDK across cloud GPU providers</em>
</p>

<p align="center">
  <a href="https://www.python.org/"><img src="https://img.shields.io/badge/python-3.11%20%7C%203.12-blue" alt="Python Version"></a>
  <a href="LICENSE"><img src="https://img.shields.io/badge/License-Apache%202.0-green.svg" alt="License"></a>
</p>

> **Unified AI inference deployment SDK across cloud GPU providers.**  
> Choose your model or workload (Audio, LLM, Image, Video) — InferWeave provisions the GPU infrastructure, sets up the container environment, manages CUDA/VRAM requirements, and exposes a ready-to-use endpoint.

---

## 🌟 The Vision

Deploying open-weight AI models into production is notoriously complex:
- Different hardware specs (VRAM sizing, tensor parallelism, compute capability).
- Fragmented container runtimes (vLLM, SGLang, ComfyUI, custom FastAPI/PyTorch wrappers).
- Provider lock-in and disparate provisioning APIs (RunPod, Lambda, AWS, GCP, Modal, etc.).

**InferWeave hides all infrastructural complexity behind a unified Python interface:**

Configure account and Proxy Token credentials using the [TTS tutorial](docs/tutorials/first-tts-on-modal.md), then:

<!-- example: tts -->
```python
"""Run with python examples/tts.py after configuring Modal and proxy credentials."""

import asyncio
from pathlib import Path

from inferweave import InferWeave


async def main() -> None:
    weave = InferWeave()
    deployment = None
    try:
        deployment = await weave.deploy(
            model="fish-s2-pro",
            provider="modal",
            custom_args={"cleanup_on_failure": True},
        )
        audio = await deployment.synthesize("Hello from InferWeave", format="wav")
        Path("speech.wav").write_bytes(audio)
    finally:
        try:
            if deployment is not None:
                await deployment.stop()
        finally:
            await weave.close()


if __name__ == "__main__":
    asyncio.run(main())
```


InferWeave automatically handles model profile detection, VRAM budgeting, container runtime templates, healthcheck polling, and endpoint proxying.

---

## Inference, recovery and lifecycle

`await weave.attach(id)` rebuilds a handle after restart; `await weave.find(model=...,
provider=...)` reuses a unique active deployment from SQLite. Ambiguity raises
`AmbiguousDeploymentError`. Mount the database directory in persistent services.

Modal endpoints are protected by default. SDK credentials provision resources; separate
`MODAL_PROXY_TOKEN_ID`/`MODAL_PROXY_TOKEN_SECRET` authenticate probes/inference.
Scale-to-zero releases idle GPU containers while keeping the app reusable;
`destroy_after_idle_mins` stops the entire app. `close()` releases local resources.

## Multiple accounts per provider

Register several Modal, Lightning AI, RunPod or Vast.ai accounts and InferWeave rotates new
deployments across them with [CredWeave](https://github.com/AmaLS367/CredWeave): round-robin,
weighted or priority failover, cooldown on rate limits and quota exhaustion, and bounded failover
to the next account. Every deployment stays bound to the account that created it, across
restarts, for status, stop, autostop and inference.

```python
weave = InferWeave(accounts="accounts.yaml")  # or export INFERWEAVE_ACCOUNTS_FILE
deployment = await weave.deploy("fish-s2-pro", provider="modal")
print(deployment.account)  # e.g. "modal-team-b"
```

The accounts file only names environment variables or a protected credentials file; secrets are
kept out of deployment records, logs and model containers. Uncertain creation blocks
retries until the original operation has settled and owner-bound absence/cleanup is confirmed.
Changed control keys cannot silently take over existing pooled deployments. Read [use multiple accounts](docs/how-to/use-multiple-accounts.md).

## Documentation and examples

Lightning AI uses the same deployment, inference and recovery API:

```python
weave = InferWeave()
deployment = await weave.deploy("fish-s2-pro", provider="lightning")
audio = await deployment.synthesize("Hello from Lightning AI")
await deployment.stop()
await weave.close()
```

Install `pip install "inferweave[lightning]"` and configure `LIGHTNING_USER_ID`,
`LIGHTNING_API_KEY` and `LIGHTNING_TEAMSPACE=owner/teamspace` locally.
The user API key also authenticates inference; scoped keys are unsupported.
Lightning creates one container Deployment with no Studio; stop deletes it.
Read the [Lightning guide](docs/how-to/use-lightning.md) for first-pull readiness,
autoscaling, restart recovery and opt-in paid tests.
The verified SDK is `lightning-sdk==2026.10.1` (Python 3.11/3.12).
Its Click requirement conflicts with SkyPilot; InferWeave runs Lightning and SkyPilot
operations in worker processes, so each can live in its own virtual environment
(`INFERWEAVE_LIGHTNING_PYTHON`, `INFERWEAVE_SKYPILOT_PYTHON`) while one InferWeave process
drives all providers. `inferweave[all]` retains SkyPilot, Modal and workers; it excludes Lightning.

- [Use multiple provider accounts](docs/how-to/use-multiple-accounts.md)
- [Use Lightning AI](docs/how-to/use-lightning.md)
- [First TTS on Modal](docs/tutorials/first-tts-on-modal.md)
- [Generate images](docs/how-to/generate-an-image.md)
- [Recover deployments](docs/how-to/reuse-deployments-across-restarts.md)
- [Secure endpoints](docs/how-to/secure-your-endpoint.md)
- [Handle cold starts](docs/how-to/handle-cold-starts.md)
- [Long-lived services](docs/how-to/run-in-a-long-lived-service.md)
- [API](docs/reference/client-api.md), [configuration](docs/reference/configuration.md), [models](docs/reference/supported-models.md)
- [Architecture](docs/explanation/architecture.md), [lifecycle and cost](docs/explanation/lifecycle-and-cost.md), [usage metering and estimates](docs/reference/client-api.md#usage-and-estimates)
- [Runnable examples](examples/README.md), [roadmap](docs/ROADMAP.md), [changelog](CHANGELOG.md)

## 🏗️ Architecture

```text
               ┌────────────────────────┐
               │    Your Application    │
               └───────────┬────────────┘
                           │
               ┌───────────▼────────────┐
               │     InferWeave SDK     │
               │ (Registry, Router, API)│
               └───────────┬────────────┘
                           │
       ┌───────────────────┴───────────────────┐
       ▼                                       ▼
┌───────────────┐                       ┌───────────────┐
│ Runtime Layer │                       │ Compute Layer │
├───────────────┤                       ├───────────────┤
│ • Audio (TTS) │                       │ • SkyPilot:   │
│ • LLM (vLLM)  │                       │   RunPod/Vast │
│ • Image (FLUX)│                       │   AWS/GCP/etc.│
│ • Video (WAN) │                       │ • Modal       │
│ • Custom      │                       │ • Cloud GPUs  │
└───────────────┘                       └───────────────┘
```

---

## 📦 Installation

InferWeave provides a modular installation model so you only install dependencies for the clouds you actually use.

### 1. Base Installation (Core SDK & CLI)
Installs the unified client, schema definitions, model registry, and CLI tools:
```bash
pip install inferweave
```

### 2. Provider-Specific Extras
Install drivers for the clouds you plan to target:
```bash
# Individual cloud providers (via SkyPilot engine)
pip install "inferweave[runpod]"
pip install "inferweave[aws]"
pip install "inferweave[gcp]"
pip install "inferweave[azure]"
pip install "inferweave[lambda]"
pip install "inferweave[nebius]"
pip install "inferweave[kubernetes]"
pip install "inferweave[vast]"

# Serverless compute engine (verified with modal>=1.6,<1.7)
pip install "inferweave[modal]"

# Inference worker runtime (FastAPI / Uvicorn stack)
pip install "inferweave[workers]"
```

### 3. Multi-Cloud Bundles
```bash
# All cloud providers supported by SkyPilot
pip install "inferweave[clouds]"

# SkyPilot cloud engines + Modal + workers runtime (Lightning uses a separate environment)
pip install "inferweave[all]"
```

---

## ⚡ CLI Quickstart

InferWeave includes a high-performance command-line interface for deploying and inspecting models across GPU providers:

```bash
# 1. Deploy models directly to cloud GPUs
inferweave deploy fish-s2-pro --provider modal
inferweave deploy meta-llama/Meta-Llama-3-8B-Instruct --provider runpod --gpu A100

# 2. Dry-run deployment to preview configuration and hardware budgets
inferweave deploy fish-s2-pro --provider modal --dry-run

# 3. Explore supported models and cloud providers
inferweave models
inferweave models --workload audio
inferweave providers

# 4. Inspect, monitor, and stop deployments across processes
inferweave list
inferweave status <deployment-id>
inferweave stop <deployment-id>
```

---

## 💻 Platform & OS Support Guidelines

| Operating System | InferWeave Core / Client | Modal Engine | SkyPilot Engine (Multi-Cloud) |
| :--- | :---: | :---: | :---: |
| **Linux (Ubuntu/Debian/RHEL)** | ✅ Native | ✅ Native | ✅ Native |
| **macOS (Apple Silicon / Intel)** | ✅ Native | ✅ Native | ✅ Native |
| **Windows (WSL2)** | ✅ Native (WSL) | ✅ Native (WSL) | ✅ Native (WSL) |
| **Windows (Native PowerShell/CMD)** | ✅ Native | ✅ Native | ⚠️ Requires WSL2 / Remote Runner |

### ⚠️ Important Note for Windows Users
SkyPilot’s underlying execution engine relies on POSIX system primitives (`termios`, `resource`) that are not present in native Windows. 

- **If you are on Windows:** We recommend running InferWeave inside **WSL2** (Windows Subsystem for Linux), where all multi-cloud provisioning features work seamlessly.
- **InferWeave Graceful Fallback:** InferWeave utilizes lazy loading for SkyPilot modules. If invoked directly on native Windows, InferWeave will safely guide you to WSL2 or allow targeting native-supported backends (such as Modal or remote runners) without crashing during import.

---

---

## 🛠️ Contributing & Development Setup

InferWeave uses [`uv`](https://docs.astral.sh/uv/) for lightning-fast dependency management.

```bash
# Clone the repository
git clone https://github.com/AmaLS367/InferWeave.git
cd InferWeave

# Install extras needed for mocked tests and dev tools
uv sync --extra modal --extra workers --group dev

# Run linting and type checking
uv run ruff check .
uv run mypy src
uv run pytest -m "not integration" --strict-markers -ra
uv run python examples/simulate.py
```

#### Live integration tests (opt-in, spend real money)

Tests marked `integration` talk to real clouds. They are excluded from CI and skip
cleanly unless explicitly enabled and configured:

| Test | Required environment |
| --- | --- |
| `tests/test_modal_integration.py` | `MODAL_TOKEN_ID`, `MODAL_TOKEN_SECRET`, `MODAL_PROXY_TOKEN_ID`, `MODAL_PROXY_TOKEN_SECRET` |
| `tests/test_runpod_integration.py` | `INFERWEAVE_RUNPOD_INTEGRATION=1`, `RUNPOD_API_KEY` (or `~/.runpod/config.toml`), `inferweave[runpod]` on Linux/macOS/WSL2 |

For local testing, copy [`.env.example`](.env.example) to `.env` and fill in the
four Modal credentials. `.env` is ignored by Git. The library does not load it
automatically; `uv` can load it for the test process:

```bash
uv run --env-file .env --extra modal pytest -m integration tests/test_modal_integration.py --strict-markers -ra
```

This test checks protected access, TTS output, recovery with a fresh SDK and
deployment cleanup. It provisions a real GPU and incurs charges.

The RunPod test also reads optional `INFERWEAVE_RUNPOD_GPU` (default `L4`),
`INFERWEAVE_RUNPOD_MODEL` (default `facebook/opt-125m`), `INFERWEAVE_RUNPOD_READY_TIMEOUT_SECS`,
`INFERWEAVE_RUNPOD_AUTOSTOP_MINS` and `INFERWEAVE_RUNPOD_ALLOW_SPOT`; see the module docstring.

```bash
INFERWEAVE_RUNPOD_INTEGRATION=1 RUNPOD_API_KEY=... \
  uv run --extra runpod pytest -m integration tests/test_runpod_integration.py -s
```

### 🚀 Release Process

Publishing to PyPI is automated via GitHub Actions using PyPI Trusted Publishing (OIDC):

1. Ensure the CI suite is green on `master`.
2. Set the package version in `src/inferweave/__init__.py`.
3. Create and push a Git tag: `vX.Y.Z`.
4. Publish a GitHub Release from that tag.
5. GitHub Actions (`publish.yml`) verifies artifact metadata, validates that the release tag matches `inferweave.__version__`, and publishes the artifacts to PyPI via Trusted Publishing.

---

## 📄 License

InferWeave's source code is licensed under the Apache License 2.0. See [LICENSE](LICENSE) for details.

### Third-party licenses

The Apache-2.0 license covers InferWeave's own source code only. It does **not** cover anything InferWeave deploys or references:

- **Model weights** (including those named by the built-in model profiles) keep their upstream licenses.
- **Container images and third-party runtimes** (vLLM, Fish Speech, PyTorch, and the Python packages installed into them) keep their upstream licenses.
- You are responsible for reviewing and complying with the upstream terms of every model, image and runtime you deploy. This is not legal advice.

**Fish Audio S2 Pro (`fish-s2-pro`):** the `fishaudio/s2-pro` weights are published under Fish Audio's own license (`fish-audio-research-license` on [Hugging Face](https://huggingface.co/fishaudio/s2-pro)), not Apache-2.0. The built-in S2 runtime is **beta**: it uses the upstream `v2.0.0-beta` pre-release of [fishaudio/fish-speech](https://github.com/fishaudio/fish-speech). Review the upstream model and runtime licenses before any production or commercial use.
