Metadata-Version: 2.5
Name: inferweave
Version: 0.2.0
Summary: Unified AI inference deployment SDK across cloud GPU providers
Project-URL: Homepage, https://github.com/AmaLS367/InferWeave
Project-URL: Repository, https://github.com/AmaLS367/InferWeave
Project-URL: Issues, https://github.com/AmaLS367/InferWeave/issues
Project-URL: Changelog, https://github.com/AmaLS367/InferWeave/blob/master/CHANGELOG.md
Author: AmaLS367
License-Expression: Apache-2.0
License-File: LICENSE
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: <3.13,>=3.11
Requires-Dist: httpx<1.0,>=0.28.1
Requires-Dist: msgpack<2.0,>=1.1.0
Requires-Dist: pydantic>=2.10.0
Requires-Dist: pyyaml>=6.0.2
Requires-Dist: rich>=13.0.0
Requires-Dist: typer>=0.12.0
Provides-Extra: all
Requires-Dist: fastapi>=0.110.0; extra == 'all'
Requires-Dist: modal<1.7,>=1.6; extra == 'all'
Requires-Dist: skypilot[aws,azure,cudo,fluidstack,gcp,ibm,kubernetes,lambda,nebius,oci,paperspace,runpod,scp,seeweb,shadeform,vast,verda]>=0.13.0; extra == 'all'
Requires-Dist: uvicorn>=0.28.0; extra == 'all'
Provides-Extra: aws
Requires-Dist: skypilot[aws]>=0.13.0; extra == 'aws'
Provides-Extra: azure
Requires-Dist: skypilot[azure]>=0.13.0; extra == 'azure'
Provides-Extra: clouds
Requires-Dist: skypilot[aws,azure,cudo,fluidstack,gcp,ibm,kubernetes,lambda,nebius,oci,paperspace,runpod,scp,seeweb,shadeform,vast,verda]>=0.13.0; extra == 'clouds'
Provides-Extra: gcp
Requires-Dist: skypilot[gcp]>=0.13.0; extra == 'gcp'
Provides-Extra: kubernetes
Requires-Dist: skypilot[kubernetes]>=0.13.0; extra == 'kubernetes'
Provides-Extra: lambda
Requires-Dist: skypilot[lambda]>=0.13.0; extra == 'lambda'
Provides-Extra: lightning
Requires-Dist: lightning-sdk<2026.10.2,>=2026.10.1; extra == 'lightning'
Provides-Extra: modal
Requires-Dist: modal<1.7,>=1.6; extra == 'modal'
Provides-Extra: nebius
Requires-Dist: skypilot[nebius]>=0.13.0; extra == 'nebius'
Provides-Extra: runpod
Requires-Dist: skypilot[runpod]>=0.13.0; extra == 'runpod'
Provides-Extra: vast
Requires-Dist: skypilot[vast]>=0.13.0; extra == 'vast'
Provides-Extra: workers
Requires-Dist: fastapi>=0.110.0; extra == 'workers'
Requires-Dist: uvicorn>=0.28.0; extra == 'workers'
Description-Content-Type: text/markdown

<p align="center">
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="assets/inferweave_banner.png">
    <source media="(prefers-color-scheme: light)" srcset="assets/inferweave_banner_light.png">
    <img alt="InferWeave Logo" src="assets/inferweave_banner.png" width="580">
  </picture>
</p>

<p align="center">
  <em>Unified AI inference deployment SDK across cloud GPU providers</em>
</p>

<p align="center">
  <a href="https://www.python.org/"><img src="https://img.shields.io/badge/python-3.11%20%7C%203.12-blue" alt="Python Version"></a>
  <a href="LICENSE"><img src="https://img.shields.io/badge/License-Apache%202.0-green.svg" alt="License"></a>
</p>

> **Unified AI inference deployment SDK across cloud GPU providers.**  
> Choose your model or workload (Audio, LLM, Image, Video) — InferWeave provisions the GPU infrastructure, sets up the container environment, manages CUDA/VRAM requirements, and exposes a ready-to-use endpoint.

---

## 🌟 The Vision

Deploying open-weight AI models into production is notoriously complex:
- Different hardware specs (VRAM sizing, tensor parallelism, compute capability).
- Fragmented container runtimes (vLLM, SGLang, ComfyUI, custom FastAPI/PyTorch wrappers).
- Provider lock-in and disparate provisioning APIs (RunPod, Lambda, AWS, GCP, Modal, etc.).

**InferWeave hides all infrastructural complexity behind a unified Python interface:**

Configure account and Proxy Token credentials using the [TTS tutorial](docs/tutorials/first-tts-on-modal.md), then:

<!-- example: tts -->
```python
"""Run with python examples/tts.py after configuring Modal and proxy credentials."""

import asyncio
from pathlib import Path

from inferweave import InferWeave


async def main() -> None:
    weave = InferWeave()
    deployment = None
    try:
        deployment = await weave.deploy(
            model="fish-s2-pro",
            provider="modal",
            custom_args={"cleanup_on_failure": True},
        )
        audio = await deployment.synthesize("Hello from InferWeave", format="wav")
        Path("speech.wav").write_bytes(audio)
    finally:
        try:
            if deployment is not None:
                await deployment.stop()
        finally:
            await weave.close()


if __name__ == "__main__":
    asyncio.run(main())
```


InferWeave automatically handles model profile detection, VRAM budgeting, container runtime templates, healthcheck polling, and endpoint proxying.

---

## Inference, recovery and lifecycle

`await weave.attach(id)` rebuilds a handle after restart; `await weave.find(model=...,
provider=...)` reuses a unique active deployment from SQLite. Ambiguity raises
`AmbiguousDeploymentError`. Mount the database directory in persistent services.

Modal endpoints are protected by default. SDK credentials provision resources; separate
`MODAL_PROXY_TOKEN_ID`/`MODAL_PROXY_TOKEN_SECRET` authenticate probes/inference.
Scale-to-zero releases idle GPU containers while keeping the app reusable;
`destroy_after_idle_mins` stops the entire app. `close()` releases local resources.

## Documentation and examples

Lightning AI uses the same deployment, inference and recovery API:

```python
weave = InferWeave()
deployment = await weave.deploy("fish-s2-pro", provider="lightning")
audio = await deployment.synthesize("Hello from Lightning AI")
await deployment.stop()
await weave.close()
```

Install `pip install "inferweave[lightning]"` and configure `LIGHTNING_USER_ID`,
`LIGHTNING_API_KEY` and `LIGHTNING_TEAMSPACE=owner/teamspace` locally.
The user API key also authenticates inference; scoped keys are unsupported.
Lightning creates one container Deployment with no Studio; stop deletes it.
Read the [Lightning guide](docs/how-to/use-lightning.md) for first-pull readiness,
autoscaling, restart recovery and opt-in paid tests.
The verified SDK is `lightning-sdk==2026.10.1` (Python 3.11/3.12).
Its Click requirement conflicts with SkyPilot, so use a separate environment.
`inferweave[all]` retains SkyPilot, Modal and workers; it excludes Lightning.

- [Use Lightning AI](docs/how-to/use-lightning.md)
- [First TTS on Modal](docs/tutorials/first-tts-on-modal.md)
- [Generate images](docs/how-to/generate-an-image.md)
- [Recover deployments](docs/how-to/reuse-deployments-across-restarts.md)
- [Secure endpoints](docs/how-to/secure-your-endpoint.md)
- [Handle cold starts](docs/how-to/handle-cold-starts.md)
- [Long-lived services](docs/how-to/run-in-a-long-lived-service.md)
- [API](docs/reference/client-api.md), [configuration](docs/reference/configuration.md), [models](docs/reference/supported-models.md)
- [Architecture](docs/explanation/architecture.md), [lifecycle and cost](docs/explanation/lifecycle-and-cost.md)
- [Runnable examples](examples/README.md), [roadmap](docs/ROADMAP.md), [changelog](CHANGELOG.md)

## 🏗️ Architecture

```text
               ┌────────────────────────┐
               │    Your Application    │
               └───────────┬────────────┘
                           │
               ┌───────────▼────────────┐
               │     InferWeave SDK     │
               │ (Registry, Router, API)│
               └───────────┬────────────┘
                           │
       ┌───────────────────┴───────────────────┐
       ▼                                       ▼
┌───────────────┐                       ┌───────────────┐
│ Runtime Layer │                       │ Compute Layer │
├───────────────┤                       ├───────────────┤
│ • Audio (TTS) │                       │ • SkyPilot:   │
│ • LLM (vLLM)  │                       │   RunPod/Vast │
│ • Image (FLUX)│                       │   AWS/GCP/etc.│
│ • Video (WAN) │                       │ • Modal       │
│ • Custom      │                       │ • Cloud GPUs  │
└───────────────┘                       └───────────────┘
```

---

## 📦 Installation

InferWeave provides a modular installation model so you only install dependencies for the clouds you actually use.

### 1. Base Installation (Core SDK & CLI)
Installs the unified client, schema definitions, model registry, and CLI tools:
```bash
pip install inferweave
```

### 2. Provider-Specific Extras
Install drivers for the clouds you plan to target:
```bash
# Individual cloud providers (via SkyPilot engine)
pip install "inferweave[runpod]"
pip install "inferweave[aws]"
pip install "inferweave[gcp]"
pip install "inferweave[azure]"
pip install "inferweave[lambda]"
pip install "inferweave[nebius]"
pip install "inferweave[kubernetes]"
pip install "inferweave[vast]"

# Serverless compute engine (verified with modal>=1.6,<1.7)
pip install "inferweave[modal]"

# Inference worker runtime (FastAPI / Uvicorn stack)
pip install "inferweave[workers]"
```

### 3. Multi-Cloud Bundles
```bash
# All cloud providers supported by SkyPilot
pip install "inferweave[clouds]"

# SkyPilot cloud engines + Modal + workers runtime (Lightning uses a separate environment)
pip install "inferweave[all]"
```

---

## ⚡ CLI Quickstart

InferWeave includes a high-performance command-line interface for deploying and inspecting models across GPU providers:

```bash
# 1. Deploy models directly to cloud GPUs
inferweave deploy fish-s2-pro --provider modal
inferweave deploy meta-llama/Meta-Llama-3-8B-Instruct --provider runpod --gpu A100

# 2. Dry-run deployment to preview configuration and hardware budgets
inferweave deploy fish-s2-pro --provider modal --dry-run

# 3. Explore supported models and cloud providers
inferweave models
inferweave models --workload audio
inferweave providers

# 4. Inspect, monitor, and stop deployments across processes
inferweave list
inferweave status <deployment-id>
inferweave stop <deployment-id>
```

---

## 💻 Platform & OS Support Guidelines

| Operating System | InferWeave Core / Client | Modal Engine | SkyPilot Engine (Multi-Cloud) |
| :--- | :---: | :---: | :---: |
| **Linux (Ubuntu/Debian/RHEL)** | ✅ Native | ✅ Native | ✅ Native |
| **macOS (Apple Silicon / Intel)** | ✅ Native | ✅ Native | ✅ Native |
| **Windows (WSL2)** | ✅ Native (WSL) | ✅ Native (WSL) | ✅ Native (WSL) |
| **Windows (Native PowerShell/CMD)** | ✅ Native | ✅ Native | ⚠️ Requires WSL2 / Remote Runner |

### ⚠️ Important Note for Windows Users
SkyPilot’s underlying execution engine relies on POSIX system primitives (`termios`, `resource`) that are not present in native Windows. 

- **If you are on Windows:** We recommend running InferWeave inside **WSL2** (Windows Subsystem for Linux), where all multi-cloud provisioning features work seamlessly.
- **InferWeave Graceful Fallback:** InferWeave utilizes lazy loading for SkyPilot modules. If invoked directly on native Windows, InferWeave will safely guide you to WSL2 or allow targeting native-supported backends (such as Modal or remote runners) without crashing during import.

---

---

## 🛠️ Contributing & Development Setup

InferWeave uses [`uv`](https://docs.astral.sh/uv/) for lightning-fast dependency management.

```bash
# Clone the repository
git clone https://github.com/AmaLS367/InferWeave.git
cd InferWeave

# Install extras needed for mocked tests and dev tools
uv sync --extra modal --extra workers --group dev

# Run linting and type checking
uv run ruff check .
uv run mypy src
uv run pytest -m "not integration" --strict-markers -ra
uv run python examples/simulate.py
```

#### Live integration tests (opt-in, spend real money)

Tests marked `integration` talk to real clouds. They are excluded from CI and skip
cleanly unless explicitly enabled and configured:

| Test | Required environment |
| --- | --- |
| `tests/test_modal_integration.py` | `MODAL_TOKEN_ID`, `MODAL_TOKEN_SECRET`, `MODAL_PROXY_TOKEN_ID`, `MODAL_PROXY_TOKEN_SECRET` |
| `tests/test_runpod_integration.py` | `INFERWEAVE_RUNPOD_INTEGRATION=1`, `RUNPOD_API_KEY` (or `~/.runpod/config.toml`), `inferweave[runpod]` on Linux/macOS/WSL2 |

For local testing, copy [`.env.example`](.env.example) to `.env` and fill in the
four Modal credentials. `.env` is ignored by Git. The library does not load it
automatically; `uv` can load it for the test process:

```bash
uv run --env-file .env --extra modal pytest -m integration tests/test_modal_integration.py --strict-markers -ra
```

This test checks protected access, TTS output, recovery with a fresh SDK and
deployment cleanup. It provisions a real GPU and incurs charges.

The RunPod test also reads optional `INFERWEAVE_RUNPOD_GPU` (default `L4`),
`INFERWEAVE_RUNPOD_MODEL` (default `facebook/opt-125m`), `INFERWEAVE_RUNPOD_READY_TIMEOUT_SECS`,
`INFERWEAVE_RUNPOD_AUTOSTOP_MINS` and `INFERWEAVE_RUNPOD_ALLOW_SPOT`; see the module docstring.

```bash
INFERWEAVE_RUNPOD_INTEGRATION=1 RUNPOD_API_KEY=... \
  uv run --extra runpod pytest -m integration tests/test_runpod_integration.py -s
```

### 🚀 Release Process

Publishing to PyPI is automated via GitHub Actions using PyPI Trusted Publishing (OIDC):

1. Ensure the CI suite is green on `master`.
2. Set the package version in `src/inferweave/__init__.py`.
3. Create and push a Git tag: `vX.Y.Z`.
4. Publish a GitHub Release from that tag.
5. GitHub Actions (`publish.yml`) verifies artifact metadata, validates that the release tag matches `inferweave.__version__`, and publishes the artifacts to PyPI via Trusted Publishing.

---

## 📄 License

InferWeave's source code is licensed under the Apache License 2.0. See [LICENSE](LICENSE) for details.

### Third-party licenses

The Apache-2.0 license covers InferWeave's own source code only. It does **not** cover anything InferWeave deploys or references:

- **Model weights** (including those named by the built-in model profiles) keep their upstream licenses.
- **Container images and third-party runtimes** (vLLM, Fish Speech, PyTorch, and the Python packages installed into them) keep their upstream licenses.
- You are responsible for reviewing and complying with the upstream terms of every model, image and runtime you deploy. This is not legal advice.

**Fish Audio S2 Pro (`fish-s2-pro`):** the `fishaudio/s2-pro` weights are published under Fish Audio's own license (`fish-audio-research-license` on [Hugging Face](https://huggingface.co/fishaudio/s2-pro)), not Apache-2.0. The built-in S2 runtime is **beta**: it uses the upstream `v2.0.0-beta` pre-release of [fishaudio/fish-speech](https://github.com/fishaudio/fish-speech). Review the upstream model and runtime licenses before any production or commercial use.
