Metadata-Version: 2.4
Name: foundry-local-sdk
Version: 2.1.0
Summary: Foundry Local Python SDK (v2): in-process Python bindings for the Foundry Local native runtime.
Author-email: Microsoft Corporation <foundrylocaldevs@microsoft.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/microsoft/Foundry-Local
Project-URL: Issues, https://github.com/microsoft/Foundry-Local/issues
Project-URL: Source, https://github.com/microsoft/Foundry-Local
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX :: Linux
Classifier: Operating System :: MacOS
Requires-Python: <3.15,>=3.11
Description-Content-Type: text/markdown
License-File: LICENSE.txt
Requires-Dist: cffi>=1.16
Requires-Dist: typing_extensions>=4.5
Requires-Dist: pydantic>=2.0.0
Requires-Dist: requests>=2.32.4
Requires-Dist: openai>=2.24.0
Requires-Dist: onnxruntime==1.30.0
Requires-Dist: onnxruntime-genai-core==0.17.1
Provides-Extra: numpy
Requires-Dist: numpy>=1.23; extra == "numpy"
Provides-Extra: openai
Requires-Dist: openai>=1.0; extra == "openai"
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Requires-Dist: mypy; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Dynamic: license-file

# Foundry Local Python SDK (v2)

The Foundry Local Python SDK is a native Python binding for the Foundry Local C++ SDK. It lets you discover, download, load, and run inference against local AI models — chat completions (streaming and non-streaming), tool calling, embeddings, and audio transcription — directly in-process via a [cffi](https://cffi.readthedocs.io/) binding to the Foundry Local native library. No separate service, no HTTP hop.

## Features

- **Model Catalog and BYOM** – browse the public catalog or register existing local model assets without copying them
- **Model Management** – download, cache, load, and unload models
- **Chat Completions** – streaming and non-streaming, with first-class tool calling
- **Embeddings** – text embeddings via a typed tensor API
- **Audio Transcription** – speech-to-text (offline and live streaming)
- **Explicit EP Management** – discover, download, and register execution providers on demand
- **Built-in Web Service** – optional HTTP endpoint for multi-process scenarios
- **Native Performance** – cffi (API mode) binding to the Foundry Local native library

## Installation

```bash
pip install foundry-local-sdk
```

The wheel includes the Foundry Local native library and depends on `onnxruntime` and `onnxruntime-genai-core` on every platform. Windows wheels also include the reg-free WinML 2.x runtime.

### Building from source

The Python SDK is a cffi binding compiled against `sdk_v2/cpp/include/foundry_local/foundry_local_c.h`. The compiled extension calls into `foundry_local.{dll,so,dylib}` at runtime via `dlopen` — **the native library must already exist** before the wheel is usable. Build it first:

```bash
# Windows
sdk_v2\cpp\build.bat --skip_tests

# Linux / macOS
sdk_v2/cpp/build.sh --skip_tests
```

The output lands where `lib_loader.py` will discover it automatically:

- Windows (multi-config MSBuild): `sdk_v2/cpp/build/Windows/<Config>/bin/<Config>/foundry_local.dll`
- Linux: `sdk_v2/cpp/build/Linux/<Config>/bin/libfoundry_local.so`
- macOS: `sdk_v2/cpp/build/macOS/<Config>/bin/libfoundry_local.dylib`

To override the lookup, set `FOUNDRY_LOCAL_LIB_DIR` to a directory that contains `foundry_local.{dll,so,dylib}`.

Then build the wheel:

```bash
cd sdk_v2/python
python -m build --wheel
```

For editable installs during development:

```bash
pip install -e .
```

### Installing native runtime dependencies for development / CI

`foundry-local-install` installs the published wheel and verifies that `onnxruntime` and `onnxruntime_genai_core` are available.

```bash
foundry-local-install

# Add --verbose to print resolved binary paths after installation.
```

> **Do not run this against a source-build / editable install.** It will overwrite your `pip install -e .` (or any locally-built wheel install) with the published PyPI version. The source-build flow above (`pip install -e .` or `pip install <local.whl>`) already pulls the matching ORT and GenAI runtime packages via pyproject dependencies — no extra step needed.

## Requirements

- Python 3.11 or newer (single `cp311-abi3` wheel works on every CPython ≥ 3.11)
- Windows (x64), Linux (x64), or macOS (arm64)

## Quick start

```python
from foundry_local_sdk import (
    ChatSession,
    Configuration,
    FoundryLocalManager,
    MessageItem,
    Request,
    RequestOptions,
    SearchOptions,
    TextItem,
)

# 1. Initialize
config = Configuration(app_name="MyApp")
FoundryLocalManager.initialize(config)
manager = FoundryLocalManager.instance

# 2. Pick and load a model
model = manager.catalog.get_model("qwen2.5-0.5b")
model.download(lambda pct: print(f"\rDownloading: {pct:.1f}%", end="", flush=True))
print()
model.load()

# 3. Run a chat request through a typed session
with ChatSession(model) as session:
    session.set_options(RequestOptions(search=SearchOptions(temperature=0.0, max_output_tokens=128)))

    with Request().add_item(MessageItem.user("Why is the sky blue?")) as req:
        with session.process_request(req) as response:
            for item in response:
                if isinstance(item, TextItem):
                    print(item.text)

# 4. Cleanup
model.unload()
```

Pass a `threading.Event` as `cancel_event` to cancel an active download at the next native progress checkpoint:

```python
from threading import Event

cancel_event = Event()

def on_progress(percent: float) -> None:
    print(f"\rDownloading: {percent:.1f}%", end="", flush=True)
    cancel_event.set()

model.download(progress_callback=on_progress, cancel_event=cancel_event)
```

Runnable end-to-end examples live under [`samples/python/`](https://github.com/microsoft/Foundry-Local/tree/main/samples/python).

## Usage

### Initialization

Create a `Configuration` and initialize the singleton `FoundryLocalManager`.

```python
from foundry_local_sdk import Configuration, FoundryLocalManager, LogLevel

config = Configuration(
    app_name="MyApp",
    model_cache_dir="/path/to/cache",                # optional
    log_level=LogLevel.INFORMATION,                   # optional (default: Warning)
)
FoundryLocalManager.initialize(config)
manager = FoundryLocalManager.instance
```

### Discovering models

```python
catalog = manager.catalog

# List all models in the catalog
models = catalog.list_models()

# Get a specific model by alias
model = catalog.get_model("qwen2.5-0.5b")

# Get a specific variant by ID
variant = catalog.get_model_variant("qwen2.5-0.5b-instruct-generic-cpu:4")

# Locally cached / currently loaded
cached = catalog.get_cached_models()
loaded = catalog.get_loaded_models()
```

### Bring your own model (BYOM)

Use the local catalog to register an existing ONNX Runtime GenAI model directory. Registration does not copy,
download, or take ownership of the model assets. The directory must contain `genai_config.json`, and the model ID
must use the canonical `<name>:<version>` format. The `task` metadata property is required and must be one of
`chat-completion`, `vision-language-chat`, `embeddings`, or `automatic-speech-recognition`.

```python
from foundry_local_sdk import CatalogType, ModelInfoBuilder

local_catalog = manager.get_catalog(CatalogType.LOCAL)

with ModelInfoBuilder() as metadata:
    metadata.set_string_property("task", "chat-completion")
    metadata.set_string_property("display_name", "My Local Model")
    metadata.set_int_property("context_length", 4096)
    model = local_catalog.register_model(
        model_path="/path/to/model-directory",
        model_id="my-model-generic-cpu:1",
        metadata=metadata,
    )

print(model.id)
print(model.info.get_string_property("display_name"))

# Existing model handles remain safe for metadata and cleanup queries after
# unregistration, while future catalog lookups no longer return the model.
local_catalog.unregister_model(model.id)
```

`manager.catalog` and `manager.get_catalog()` both return the public catalog for backward compatibility.
`ModelInfoBuilder()` owns a mutable native metadata handle; call `close()` or use a `with` block. In contrast,
`model.info` is a frozen point-in-time `ModelInfo` value that supports normal dataclass operations such as
`dataclasses.asdict()`. Its declared fields remain safe after the native model or manager is released. Arbitrary
forward-compatible keys use guarded native lookup and therefore require the owning manager to remain open.

### Inspecting model metadata

`IModel` exposes metadata properties from the catalog:

```python
model = catalog.get_model("phi-3.5-mini")

# Identity
print(model.id)             # e.g. "phi-3.5-mini-instruct-generic-gpu:3"
print(model.alias)          # e.g. "phi-3.5-mini"

# Context and token limits
print(model.context_length)  # e.g. 131072 (tokens), or None if unknown

# Modalities and capabilities
print(model.input_modalities)        # e.g. "text" or "text,image"
print(model.output_modalities)       # e.g. "text"
print(model.capabilities)            # e.g. "chat,completion"
print(model.supports_tool_calling)   # True, False, or None

# Cache / load state
print(model.is_cached)
print(model.is_loaded)
```

### Explicit EP management

```python
# Discover available EPs and registration status
eps = manager.discover_eps()
for ep in eps:
    print(f"{ep.name} - registered: {ep.is_registered}")

# Download and register all available EPs
result = manager.download_and_register_eps()
print(f"Success: {result.success}, Status: {result.status}")

# Download only specific EPs
result2 = manager.download_and_register_eps([eps[0].name])
```

#### Per-EP download progress

Pass a `progress_callback` to receive `(ep_name, percent)` updates as each EP downloads (`percent` is 0–100):

```python
current_ep = ""

def on_progress(ep_name: str, percent: float) -> None:
    global current_ep
    if ep_name != current_ep:
        if current_ep:
            print()
        current_ep = ep_name
    print(f"\r  {ep_name}  {percent:5.1f}%", end="", flush=True)

manager.download_and_register_eps(progress_callback=on_progress)
print()
```

Catalog access does not block on EP downloads. Call `download_and_register_eps()` when you need hardware-accelerated execution providers.

### Chat completions with `ChatSession`

```python
from foundry_local_sdk import (
    ChatSession, MessageItem, Request, RequestOptions, SearchOptions, TextItem,
)

model = manager.catalog.get_model("qwen2.5-0.5b")
model.load()

with ChatSession(model) as session:
    session.set_options(RequestOptions(search=SearchOptions(temperature=0.0, max_output_tokens=256)))

    # Non-streaming
    with Request().add_item(MessageItem.user("What is 7 multiplied by 6?")) as req:
        with session.process_request(req) as response:
            for item in response:
                if isinstance(item, TextItem):
                    print(item.text)

    # Streaming — yields Item instances as the model produces them
    session.set_streaming(True)
    with Request().add_item(MessageItem.user("Tell me a joke")) as req:
        for item in session.process_streaming_request(req):
            if isinstance(item, TextItem):
                print(item.text, end="", flush=True)
    print()

model.unload()
```

`ChatSession` is stateful across turns. `session.turn_count` reports how many requests have been processed; `session.undo_turns(n)` rewinds history.

Before processing a request, use `preflight_request()` to synchronously ask the native runtime for
the request's token budget:

```python
model = manager.catalog.get_model("qwen2.5-0.5b")
model.load()
with ChatSession(model) as session:
    with Request().add_item(MessageItem.user("Summarize this conversation.")) as req:
        budget = session.preflight_request(req)
        print(f"Required tokens: {budget.required_tokens}; fits: {budget.fits}")
model.unload()
```

The SDK reports the native result without estimating tokens, changing options, truncating context,
or rewriting the request.
`context_limit_tokens` is the Engine's structural request capacity or the Generator's model context.
`fits` does not reserve cache or guarantee immediate admission while other requests are active.

### Multi-turn conversations

Each call to `process_request` extends the session's turn history. Build a new `Request` per turn:

```python
with ChatSession(model) as session:
    for prompt in ["Hi!", "What's your favorite color?", "Why?"]:
        with Request().add_item(MessageItem.user(prompt)) as req:
            with session.process_request(req) as resp:
                for item in resp:
                    if isinstance(item, TextItem):
                        print(f"> {prompt}\n{item.text}\n")
```

### Tool calling

Register tool definitions on the session, then watch for `ToolCallItem` in the response and reply with `ToolResultItem`:

```python
import json
from foundry_local_sdk import ToolCallItem, ToolResultItem

with ChatSession(model) as session:
    session.add_tool_definition(
        name="get_weather",
        description="Get the current weather for a city.",
        json_schema=json.dumps({
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
        }),
    )

    with Request().add_item(MessageItem.user("What's the weather in Seattle?")) as req:
        with session.process_request(req) as resp:
            for item in resp:
                if isinstance(item, ToolCallItem):
                    args = json.loads(item.arguments)
                    result = {"temperature_c": 12, "conditions": "rain"}
                    # Send the result back on the next turn
                    with Request().add_item(
                        ToolResultItem(call_id=item.call_id, content=json.dumps(result))
                    ) as follow_up:
                        with session.process_request(follow_up) as final:
                            for it in final:
                                if isinstance(it, TextItem):
                                    print(it.text)
```

For a custom tool, register only its name and description. Its `ToolCallItem.arguments` value is the
raw text generated by the model, not JSON:

```python
with ChatSession(model) as session:
    session.add_custom_tool_definition(
        name="apply_patch",
        description="Apply a patch supplied as raw unified-diff text.",
    )

    with Request().add_item(MessageItem.user("Change the greeting to hello.")) as req:
        with session.process_request(req) as resp:
            for item in resp:
                if isinstance(item, ToolCallItem) and item.name == "apply_patch":
                    patch_text = item.arguments
                    print(patch_text)
```

Tool names, descriptions, and function-tool JSON schemas must be valid UTF-8 strings without an
embedded NUL (`"\x00"`), because the native C ABI represents them as NUL-terminated strings.
Removal names have the same restriction. Passing an embedded NUL raises `ValueError` before native
code is called.

### Embeddings with `EmbeddingsSession`

`EmbeddingsSession` accepts `TextItem` inputs and returns one `TensorItem` per input containing the embedding vector. Sessions are stateless — reuse one session for many requests.

```python
from foundry_local_sdk import EmbeddingsSession, Request, TensorItem, TextItem

model = manager.catalog.get_model("qwen3-embedding-0.6b")
model.load()

with EmbeddingsSession(model) as session:
    # Single input
    with Request().add_item(TextItem("The quick brown fox")) as req:
        with session.process_request(req) as resp:
            tensor = next(it for it in resp if isinstance(it, TensorItem))
            print("Dimensions:", tensor.dimensions)
            print("First 5:", tensor.data[:5])

    # Batch input — one TextItem per string, one TensorItem out per input
    with Request() as req:
        for text in ["Machine learning", "Capital of France", "Rust language"]:
            req.add_item(TextItem(text))
        with session.process_request(req) as resp:
            for item in resp:
                if isinstance(item, TensorItem):
                    print(f"  dims={item.dimensions}")

model.unload()
```

### Audio transcription

`AudioSession` accepts `AudioItem` input (PCM bytes + sample rate / channels) and produces `TextItem` output. See [`samples/python/audio-transcription/`](https://github.com/microsoft/Foundry-Local/tree/main/samples/python/audio-transcription) and [`live-audio-transcription/`](https://github.com/microsoft/Foundry-Local/tree/main/samples/python/live-audio-transcription) for runnable end-to-end examples covering offline files and live PCM streaming through an `ItemQueue`.

### Web service (optional)

Start a built-in HTTP server for multi-process access:

```python
manager.start_web_service()
print(f"Listening on: {manager.urls}")

# ... use the service ...

manager.stop_web_service()
```

## API Reference

### Manager and configuration

| Class | Description |
|---|---|
| `Configuration` | SDK configuration (app name, cache dir, log level, web service settings) |
| `FoundryLocalManager` | Singleton entry point — initialization, typed catalog access, EP management, web service |
| `CatalogType` | Catalog selection enum: `PUBLIC` or `LOCAL` |
| `Catalog` | Model discovery plus local `register_model` / `unregister_model` operations |
| `ModelInfo` | Frozen point-in-time model metadata snapshot |
| `ModelInfoBuilder` | Caller-owned mutable BYOM registration metadata |
| `IModel` | Model interface — identity, metadata, lifecycle (`download`, `load`, `unload`), variant selection |
| `EpInfo` | Discoverable execution provider info (`name`, `is_registered`) |
| `EpDownloadResult` | Result of EP download / registration (`success`, `status`, `registered_eps`, `failed_eps`) |
| `LogLevel` | Logging verbosity enum |

### Sessions

All sessions wrap a native `flSession*` and are context managers. Closing a session releases the native handle and aborts any in-flight streaming request.

| Class | Description |
|---|---|
| `Session` | Abstract base class. Provides `process_request`, `process_streaming_request`, `set_options`, `set_streaming`, and the context-manager / `_close` lifecycle. |
| `ChatSession` | For `chat-completion` and `vision-language-chat` models. Adds function/custom tool registration, removal, `turn_count`, and `undo_turns`. |
| `EmbeddingsSession` | For `embeddings` models. Stateless — accepts `TextItem` inputs, returns one `TensorItem` per input. |
| `AudioSession` | For `automatic-speech-recognition` models. Accepts `AudioItem` input (and `ItemQueue` for live streaming), returns `TextItem`. |

Common session methods:

- `process_request(request) -> Response` — run synchronously, return the full response.
- `process_streaming_request(request) -> Iterator[Item]` — yield items as the model produces them. Requires `set_streaming(True)` first. Abandoning the iterator (`break`, exception, `gen.close()`) automatically cancels the request and joins the worker thread.
- `set_options(RequestOptions)` — apply session-level inference parameters (typed `SearchOptions` for sampling, optional `tool_choice`, and `additional_options` for passthrough).
- `set_streaming(enabled)` — install or remove the native streaming callback.

### Requests and responses

| Class | Description |
|---|---|
| `Request` | Owns an `flRequest*`. Build with `add_item(item)` (fluent — returns self). Use as a context manager so the native handle is released. `set_options(RequestOptions)` applies per-request overrides. |
| `Response` | Owns an `flResponse*`. Iterable over output items. Exposes `item_count`, `get_item(i)`, `finish_reason` (`FinishReason` enum), and `get_usage()` (`TokenUsage`). Read item data **inside** the response's `with` block — items returned by `get_item` borrow the response's handle. |
| `FinishReason` | `NONE`, `ERROR`, `STOP`, `LENGTH`, `TOOL_CALLS`. |
| `TokenUsage` | `prompt_tokens`, `completion_tokens`, `total_tokens`. |
| `RequestOptions` | Typed inference options passed to `set_options`. Wraps `search: SearchOptions` (sampling params: `temperature`, `top_p`, `top_k`, `max_output_tokens`, `frequency_penalty` and `presence_penalty`—currently zero only—`seed`, legacy Generator-only beam-search `early_stopping`, and `do_sample`), `tool_choice: ToolChoice | None` (`AUTO`/`NONE`/`REQUIRED`), and `additional_options: dict[str, str]` as the passthrough escape hatch. |

### Items

`Item` is the wire-format type for everything that flows in or out of a session — message turns, individual modality parts, tool calls, embeddings.

| Class | Description |
|---|---|
| `Item` | Abstract base. `item_type` returns the `ItemType`. `Item.from_native(ptr, owns)` dispatches to the right subclass. |
| `TextItem` | UTF-8 text. `type` is a `TextItemType` (`DEFAULT`, `REASONING`, `OPENAI_JSON`). |
| `MessageItem` | A chat turn with a `MessageRole` (`SYSTEM` / `USER` / `ASSISTANT` / `TOOL` / `DEVELOPER`) and either a string or a list of part items (text, image, audio). Convenience factories: `MessageItem.system(content)`, `MessageItem.user(content)`, `MessageItem.assistant(content)`. Borrows native pointers from supplied parts — keep the parts alive for the message's lifetime. |
| `BytesItem` | Raw binary blob (e.g. a base64-decoded payload). |
| `ImageItem` | Image input for vision models. |
| `AudioItem` | Audio input (uri or bytes + `format` +  `sample_rate` + `channels`). |
| `ToolCallItem` | Emitted by the model when invoking a tool. Carries NUL-free `call_id`, `name`, and `arguments` (JSON object text for function tools, raw UTF-8 text for custom tools). |
| `ToolResultItem` | Caller-supplied tool result. Carries NUL-free `call_id` and `result`; an empty result is valid. |
| `TensorItem` | Multi-dimensional tensor. Used for embedding output. Exposes `dimensions`, `data_type` (`TensorDataType`), and `data`. |
| `ItemQueue` | Streaming-input queue used with `AudioSession` for live PCM. The queue is itself an `Item` and is added to a `Request` with `transfer_ownership=False`. |

Enums: `ItemType`, `TextItemType`, `MessageRole`, `TensorDataType`.

### CLI entry point

| Function | CLI name | Description |
|---|---|---|
| `foundry_local_sdk._native.installer.main` | `foundry-local-install` | Install and verify native binaries (`--verbose` to print resolved paths) |

## Running tests

```bash
pip install -r requirements-dev.txt
python -m pytest test/ -v
```

See [test/README.md](test/README.md) for detailed test setup and structure.

## License

MIT — see `LICENSE.txt`.

## Links

- [Foundry Local on GitHub](https://github.com/microsoft/Foundry-Local)
- [Issues](https://github.com/microsoft/Foundry-Local/issues)
