Metadata-Version: 2.4
Name: llm-lmstudio
Version: 0.3.0
Summary: LLM plugin for models served by LMStudio (local LLM studio HTTP API)
Author: Agusti
Project-URL: Homepage, https://github.com/agustif/llm-lmstudio
Project-URL: Issues, https://github.com/agustif/llm-lmstudio/issues
Project-URL: CI, https://github.com/agustif/llm-lmstudio/actions
Project-URL: Changelog, https://github.com/agustif/llm-lmstudio/releases
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: llm>=0.32
Requires-Dist: requests>=2.0
Requires-Dist: httpx>=0.20
Dynamic: license-file

# llm-lmstudio

[![PyPI](https://img.shields.io/pypi/v/llm-lmstudio.svg)](https://pypi.org/project/llm-lmstudio/)
[![Changelog](https://img.shields.io/github/v/release/agustif/llm-lmstudio?include_prereleases&label=changelog)](https://github.com/agustif/llm-lmstudio/releases)
[![Tests](https://github.com/agustif/llm-lmstudio/actions/workflows/test.yml/badge.svg)](https://github.com/agustif/llm-lmstudio/actions/workflows/test.yml)
[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/agustif/llm-lmstudio/blob/main/LICENSE)

This is a plugin for [Simon Willison's LLM command-line utility](https://llm.datasette.io/) that lets you talk to models running on a local [LMStudio](https://lmstudio.ai/) server.

Version 0.3.0 and later require the `llm` Python package 0.32 or newer. This dependency provides structured messages, response parts, and resumable tool chains.

The plugin also requires LM Studio 0.4.0 or newer. This version introduced the native v1 REST API.

## Installation

Make sure you have `llm` installed, then install this plugin from PyPI:

```bash
uv pip install llm-lmstudio
```

Or, to install the latest development version directly from GitHub:
```bash
llm install -U git+https://github.com/agustif/llm-lmstudio.git
```
Alternatively, `llm install llm-lmstudio` will also find and install the plugin.

## Usage

First, start the LM Studio server. The API usually runs at `http://localhost:1234`.

The plugin finds downloaded models through the LM Studio API. A model does not need to be loaded first. You can check this using the `llm models | grep lmstudio` command:

```bash
llm models list
```
You should see your LMStudio models listed, prefixed with `lmstudio/` (e.g., `lmstudio/llava-v1.5-7b`).

To run a prompt against a model:

```bash
# Replace 'your-model-id' with the actual ID shown in 'llm models list'
# e.g., llm -m lmstudio/llava-v1.5-7b "Describe this image"
llm -m lmstudio/your-model-id -o temperature 0.7 -o max_tokens 100 "Tell me a joke"
```

To start an interactive chat session:

```bash
llm chat -m lmstudio/your-model-id
```

You can exit the chat by typing `exit` or `quit`.

### Vision Model Support

The plugin supports vision-language models (VLMs).
- When using a VLM, you can attach images using the standard `llm` attachment syntax:
  ```bash
  llm chat -m lmstudio/your-vlm-id -a path/to/image.png "Describe this image"
  ```
  Or for a single prompt:
  ```bash
  llm -m lmstudio/your-vlm-id -a path/to/image.png "What is in this picture?"
  ```
- The plugin will encode the image and send it to the model.
- This feature's success depends on the specific VLM, its configuration in LM Studio, and LM Studio's API correctly handling image data.
- Models that support vision may have a `👁️` (eye icon) in their `display_suffix` when inspected via `llm inspect -m lmstudio/your-vlm-id`, though this may not always render in `llm models list`.

### Tool Support

Some models loaded in LMStudio can call tools. The plugin will surface those tool invocations and their results when interacting with such models.

Example:

```bash
$ llm --tool llm_version "What version of LLM is this?" --td
```

For more information about tool calling support consult [the llm documentation on tools](https://llm.datasette.io/en/stable/tools.html).

### Embedding Models

If you have embedding models loaded in LMStudio (their names usually contain "embed"), the plugin will register them too. You can list them with:

```bash
llm embed-models
```

You should see models like `lmstudio/text-embedding-nomic-embed-text-v1.5@f16`.

To generate embeddings for text using one of these models:

```bash
llm embed -m lmstudio/your-embedding-model-id -c "This is the text to embed"
```

## Configuration

The plugin connects to the LMStudio server API. By default, it tries `http://localhost:1234`.

You can configure the server URL(s) using the `LMSTUDIO_API_BASE` environment variable.
- For a single server:
  ```bash
  export LMSTUDIO_API_BASE="http://your-server-address:port"
  ```
- For multiple servers (the plugin will try them in order):
  ```bash
  export LMSTUDIO_API_BASE="http://server1:1234,http://server2:5678,https://server3:custom_port"
  ```
The variable accepts one or more `http[s]://host:port` values, separated by commas (spaces around commas are optional). The plugin automatically attempts to append `/v1` or `/api/v0` to the determined base URL(s) as needed when probing the server.

## Model Options

You can pass generation options supported by the LMStudio API (like `temperature`, `max_tokens`, `top_p`, `stop`) using the `-o` flag:

```bash
llm -m lmstudio/your-model-id -o temperature 0.7 -o max_tokens 100 "Tell me a joke"
```

## Automatic Model Loading

If a selected model is not loaded, the plugin loads it through `POST /api/v1/models/load`. The plugin does not require the `lms` CLI.

The endpoint returns after the load operation finishes. Set `LLM_LMSTUDIO_DEBUG=1` to show load timing and instance details.

The plugin requires LM Studio 0.4.0 or newer. LM Studio 0.4.0 introduced the native v1 REST API at `/api/v1/*`.

## Development

To set up this plugin for development:
1. Clone the repository.
2. Run tests `uv run pytest`

Or do the classic complicated dance:
1. Create a virtual environment: `python -m venv .venv`
2. Activate it: `source .venv/bin/activate`
3. Install dependencies, including dev dependencies: `pip install -e . --group dev
4. Run tests: `pytest`

The asynchronous tests in `tests/test_llm_lmstudio_async.py` (among others) use `pytest-vcr` to record and replay HTTP interactions with the LM Studio server. To record new cassettes:
1. Ensure LM Studio is running with the target model(s) loaded (e.g., `llava-v1.5-7b`).
2. Temporarily set `record_mode='all'` in the `@pytest.mark.vcr` decorator for the relevant tests in `tests/test_llm_lmstudio_async.py`.
3. Run `pytest -v -s tests/test_llm_lmstudio_async.py`.
4. Cassettes will be saved in `tests/cassettes/`.
5. **Important:** Change `record_mode` back to `'once'` after recording.
6. Commit the new/updated cassettes.

## Missing features / Known Issues:

- The reliability and capabilities of image support can vary significantly based on the specific VLM and its implementation within LM Studio.
- The plugin does not support authentication for remote LM Studio servers. Community contributions for this feature are welcome.
- Automatic loading does not support `LLM_LMSTUDIO_TTL`. The `/api/v1/models/load` endpoint does not accept a TTL parameter.

## Future work

- Migrate `manual-testing.md` to an opt-in Pytest acceptance suite that runs against a live LM Studio server. Keep these slower tests out of normal `pytest` runs.

