Metadata-Version: 2.4
Name: llm-omlx
Version: 0.1.0
Summary: LLM plugin for models served by oMLX
License-Expression: Apache-2.0
Keywords: llm,mlx,omlx
Classifier: Development Status :: 3 - Alpha
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: llm>=0.34
Requires-Dist: openai<4,>=3
Dynamic: license-file

# llm-omlx

`llm-omlx` is a plugin for [LLM](https://llm.datasette.io/) that discovers and uses text,
vision, and embedding models served by [oMLX](https://github.com/jundot/omlx). It uses oMLX's
OpenAI-compatible API while presenting every model through LLM's public interfaces.

## Installation

Install the plugin into the same environment as LLM:

```sh
llm install llm-omlx
```

Point it at one oMLX server. When `OMLX_API_BASE` is unset, the plugin reads the host and
port from the local oMLX `settings.json` (under `OMLX_BASE_PATH`, default `~/.omlx`), so a
server managed by the oMLX app on the same machine works with no configuration. The API key
is optional when the server allows unauthenticated requests:

```sh
export OMLX_API_BASE=http://127.0.0.1:8000/v1  # optional when oMLX runs locally
export OMLX_API_KEY=your-key  # optional
```

You can also store a key under LLM's `omlx` alias or provide `--key` for an individual
command. Explicit keys take precedence over the stored alias and environment variable.

## Usage

Refresh discovery and list the canonical `omlx/` model IDs:

```sh
llm omlx models --refresh
llm models | grep '^omlx/'
```

Run text and vision prompts, or create an embedding:

```sh
llm -m omlx/your-model "Hello"
llm -m omlx/your-vlm -a image.png "Describe this image"
llm embed -m omlx/your-embedding-model -c "Embed this"
```

Model IDs come directly from the configured server and are always exposed as
`omlx/<provider-id>`.

## Discovery and offline behavior

The plugin first reads `GET /v1/models/status`. Models classified by oMLX as `llm`, `vlm`,
or `embedding` are registered as text, vision, or embedding models respectively. Helper,
hidden, reranker, audio, MarkItDown, and unknown special models are excluded.

If the status endpoint returns 404 or 405, discovery falls back to `GET /v1/models`. In that
fallback catalogue, IDs containing `embed` (case-insensitive) are classified as embeddings;
all other IDs become text-only chat models. Vision capability is never guessed from a model
name.

A sanitized catalogue is cached in LLM's user directory for the configured API base. Fresh
cache entries avoid repeated discovery. A matching stale cache can keep model listing usable
when the server is offline; `llm omlx models --refresh` attempts the server first and uses the
matching stale cache only if refresh fails. No model filesystem paths or raw status documents
are persisted.

## Model capabilities

Generation models support streaming, reasoning parts, conversations, tools, and structured
JSON schemas. Actual tool calling and schema enforcement still depend on the selected model,
its chat template, and the oMLX grammar configuration.

Only models classified as `vlm` accept attachments. Supported image MIME types are:

- `image/png`
- `image/jpeg`
- `image/gif`
- `image/webp`

Images may come from LLM-supported files, URLs, or bytes. Audio and video are not supported.
Embedding models accept text only; binary and oMLX multimodal embedding items are not
supported.

## Configuration

| Variable | Default | Purpose |
| --- | --- | --- |
| `OMLX_API_BASE` | local oMLX `settings.json`, else `http://127.0.0.1:8000/v1` | The single oMLX endpoint. A bare origin is normalized to end in `/v1`. |
| `OMLX_BASE_PATH` | `~/.omlx` | Where the local oMLX `settings.json` is read from when `OMLX_API_BASE` is unset. |
| `OMLX_API_KEY` | unset | Optional oMLX API key. |
| `OMLX_TIMEOUT` | `90` | Positive request timeout in seconds. |
| `OMLX_EMBED_BATCH_SIZE` | `32` | Positive default batch size for embedding requests. |

## Development

The project requires Python 3.10 or later and uses [uv](https://docs.astral.sh/uv/):

```sh
uv sync --locked --all-groups
uv lock --check
uv run ruff check .
uv run ruff format --check .
uv run pyright
uv run pytest
uv build
uv run twine check dist/*
```

Live acceptance tests are skipped by default. With an oMLX server running:

```sh
OMLX_API_BASE=http://127.0.0.1:8000/v1 \
OMLX_API_KEY=your-key \
uv run pytest --run-live -m live -q
```

See [CONTRIBUTING.md](CONTRIBUTING.md) for the contribution workflow.

## License

Apache-2.0. See [LICENSE](LICENSE).
