Metadata-Version: 2.4
Name: mcp-ollama-vllm
Version: 1.0.1
Summary: MCP server that calls a local Ollama or vLLM model for language work, with schema-validated JSON output checked by a dependency-free validator.
Author: Siegfried Emil Timothy Heerwagen
License-Expression: MIT
Project-URL: Homepage, https://github.com/setheerwagen/local-ai-mcp-servers
Project-URL: Repository, https://github.com/setheerwagen/local-ai-mcp-servers
Project-URL: Issues, https://github.com/setheerwagen/local-ai-mcp-servers/issues
Keywords: mcp,ollama,vllm,llm,self-hosted,local-ai,json-schema
Classifier: Development Status :: 5 - Production/Stable
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Environment :: Console
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: mcp==1.29.0
Requires-Dist: httpx==0.28.1
Dynamic: license-file

<!-- mcp-name: io.github.setheerwagen/mcp-ollama-vllm -->

# mcp-ollama-vllm

An MCP server that lets an MCP client such as Claude Code talk to local language models
directly. The backend is **wired up at startup: Ollama, vLLM, or a generic
OpenAI-compatible endpoint** (set once via `LOCAL_BACKEND`, see below). The
third, `openai`, is not a plugin loader for arbitrary tools; it is one more
fixed backend implementation, covering servers such as LM Studio, the
llama.cpp server, LocalAI, TGI, koboldcpp, SGLang or Jan, which all expose the
same OpenAI-compatible `/v1` endpoints as vLLM but are not vLLM itself.

## Why this one

**The one thing this server does that a plain Ollama/vLLM call does not:**
schema-validated JSON output, checked with its own dependency-free validator
and automatically retried on a violation (up to 3 attempts total, feeding the
concrete violations back to the model), the **same way, with the same tool
call, regardless of which of the two backends is configured.** Both Ollama's
`format` and vLLM's `response_format` steer the model toward JSON, but neither
guarantees that the result actually matches your schema; this server checks
that independently instead of trusting the backend's word for it.

**Before / after (illustrative, not a real run):**

```jsonc
// Backend's own schema enforcement alone (Ollama "format" / vLLM "response_format"):
// looks like valid JSON, but a required field went missing under load and nobody
// checked it against the schema before handing it onward.
{"amount": 42.10, "currency": "EUR"}

// local_structured: the same case, but validated against the schema and
// automatically retried with the concrete violation fed back to the model.
{"amount": 42.10, "currency": "EUR", "total": 42.10}
// -> attempts: 2, valid_on_first_try: false, outcome already re-validated
```

**Known limits, stated on purpose:**

- With **vLLM** the model listing reports less metadata than Ollama: no size
  or quantization, those fields read `unknown`/`0` instead of a made-up value.
- With the generic **`openai`** backend the model listing usually reports even
  less: most such servers answer `/v1/models` with only `id` (and
  `owned_by`), so `parameter_size`/`quantization` read `unknown`, sizes read
  `0`, there is no adapter signal, and `max_context` is only included when the
  endpoint genuinely reports it. The top-level `metadata_reported` field (see
  `list_models` below) makes this "not reported" honestly visible.
- The `think` parameter (reasoning trace) has **no effect on vLLM or
  `openai`**; it is only wired up for Ollama. See "The `think` parameter"
  below.

**Operating systems:** CI runs the self-tests on Linux, and the mock-backed
setup self-test selftest_vllm.py on Windows too; the server is pure Python with
no OS-specific calls, and its live backend path was additionally exercised once
on Windows on 2026-08-07. Needs a running backend service (Ollama or vLLM); the
setup self-test selftest_vllm.py uses a mock and needs no real service. macOS is
not separately tested.

## Purpose

A large part of an AI agent's work is pure language processing: writing,
summarizing, classifying, extracting structured data from text. These parts can
be offloaded to a local model, while searching, reading, writing and the
orchestration stay with the calling client.

This is explicitly about a **model call, not an agent.** The local model
receives text and returns text. It has no tools, no file access, and it does not
work in multiple steps.

## Security properties

These limits are part of the design, not just the docs:

- **No file access.** The server exposes no tool that reads or writes files. It
  reads only its own configuration from environment variables.
- **No shell execution.** There is no path to `subprocess`, `os.system` or
  anything comparable.
- **Local machine or local network only.** The server process itself makes no
  outbound calls other than to its configured backend endpoint, and that
  endpoint's address is checked both at startup and again on every call; if it
  points to the internet, the server refuses to start (or the call is
  rejected). Names are resolved for this, and **all** addresses behind them
  must be local, otherwise the guard could be bypassed via DNS; the resolved
  address is then pinned for the actual connection so a later re-resolution to
  a public address cannot slip past the check. Loopback and private networks
  (RFC 1918) are allowed, so a vLLM box on the local network is reachable;
  public addresses are rejected.
- **Two direct dependencies:** `mcp` and `httpx`. `mcp` in turn pulls in
  several further packages transitively (among others `anyio`, `pydantic`,
  `starlette`, `uvicorn`, `jsonschema`, `pyjwt`). The full resolved tree is
  pinned in `uv.lock`; CI checks with `uv lock --check` that the lock still
  matches `pyproject.toml`, then installs the exact locked tree for the tests,
  so a drifted lock fails CI. A separate, advisory `pip-audit` job checks the
  locked tree for known vulnerabilities on every push to main and pull
  request and again weekly; it does not block a pull request.

These properties bound what **this server** does; they are not a claim about the
AI tool that drives it. The model you call runs locally, but the Claude/ChatGPT
session that decides to call it is a separate hosted service and sees the calls
it makes, like any MCP call. Drive the server with a local model if that matters.

See [SECURITY.md](SECURITY.md) for the trust boundaries this covers (the model
as an oracle rather than an agent, prompt injection via model output, DNS
rebinding pinning) and what it does not.

## Installation

Replace `/path/to/mcp-ollama-vllm` below with the directory you extracted/cloned this
project into.

Python 3.10 or newer is required. There are two equivalent ways to set up a
virtual environment; use whichever tool you have.

With the standard `venv` module and `pip`:

```bash
cd /path/to/mcp-ollama-vllm
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt
```

If `python3 -m venv` fails (on some distributions the `python3-venv` package is
missing), either install that package or take the route with
[uv](https://github.com/astral-sh/uv):

```bash
cd /path/to/mcp-ollama-vllm
uv venv .venv --seed
.venv/bin/pip install -r requirements.txt
```

(`--seed` makes sure `pip` lands in the environment. Alternatively, without
`--seed`, install with `uv pip install --python .venv/bin/python -r requirements.txt`.)

The modules live under `src/mcp_ollama_vllm/` (a src layout, so the package
name does not collide with other MCP servers' top-level modules). Either of
the above gives you a checkout that runs via `PYTHONPATH=src` (see "Entry for
the MCP configuration" below), or install the package itself with
`.venv/bin/pip install .`, which additionally provides the `mcp-ollama-vllm`
console script and needs no `PYTHONPATH`.

A reachable **Ollama** or **vLLM** backend is a prerequisite for running:

- Ollama: install locally (see <https://ollama.com>), start it with
  `ollama serve` and pull at least one model (e.g. `ollama pull llama3.2:3b`);
  check with `ollama list`. For `local_embed` additionally pull an embedding
  model, e.g. `ollama pull nomic-embed-text`.
- vLLM: an OpenAI-compatible vLLM endpoint on this machine or one on your local
  network. The backend is switched only through the environment variables
  described below; the server itself does not change.

## Entry for the MCP configuration

In `~/.claude.json`, or the `mcpServers` section of your Claude Code config:

```json
{
  "mcpServers": {
    "local-models": {
      "command": "/path/to/mcp-ollama-vllm/.venv/bin/python",
      "args": ["-m", "mcp_ollama_vllm.server"],
      "env": {
        "PYTHONPATH": "/path/to/mcp-ollama-vllm/src",
        "LOCAL_BACKEND": "ollama",
        "LOCAL_HOST": "http://localhost:11434",
        "LOCAL_TIMEOUT": "600"
      }
    }
  }
}
```

Replace `/path/to/mcp-ollama-vllm` with the absolute path you extracted this project
into. Alternatively via CLI:

```bash
claude mcp add local-models --env PYTHONPATH=/path/to/mcp-ollama-vllm/src -- \
  /path/to/mcp-ollama-vllm/.venv/bin/python -m mcp_ollama_vllm.server
```

By default, `claude mcp add` binds the server to the current working directory;
for global visibility across all directories, append `--scope user`.

Alternatively, after `.venv/bin/pip install .` (see "Installation" above), the
`PYTHONPATH` entry is not needed and `args` can be `["-m",
"mcp_ollama_vllm.server"]` as is, or the installed console script can be used
directly as `command`: `/path/to/mcp-ollama-vllm/.venv/bin/mcp-ollama-vllm`
with no `args`.

### First call

Restart your MCP client so it picks up the new server, then call `list_models`
with no arguments. A response with your models proves the server starts, finds
the backend and talks to it correctly, all in one step.

### Environment variables

| Variable | Default | Meaning |
|---|---|---|
| `LOCAL_BACKEND` | `ollama` | Backend: `ollama`, `vllm` or `openai` |
| `LOCAL_HOST` | per backend | Endpoint; must be local or on your own network |
| `LOCAL_TIMEOUT` | `600` | Read timeout in seconds |
| `LOCAL_EMBED_MODEL` | `nomic-embed-text` (Ollama) | Default embedding model |
| `LOCAL_API_KEY` | - | Optional; `openai` only. Sent as a `Bearer` header, never logged |
| `LOCAL_EMBED_PATH` | `/v1/embeddings` | Optional; `openai` only. Embedding endpoint path |
| `LOCAL_SCHEMA_MODE` | `auto` | Optional; `openai` only. `auto`, `response_format` or `none` (see `local_structured` below) |

The default for `LOCAL_HOST` is `http://localhost:11434` for Ollama and
`http://localhost:8000` for vLLM. For `openai` there is no default: the
generic backend has no sensible port to guess, so `LOCAL_HOST` is **required**
and the server refuses to start without it.

The old names `OLLAMA_HOST`, `OLLAMA_TIMEOUT` and `OLLAMA_EMBED_MODEL` are still
read so existing configurations do not break. If both are set, the `LOCAL_` name
wins.

### Switching to a remote vLLM endpoint

The vLLM box on the local network is addressed through only two variables; the
server itself does not change:

```json
{
  "mcpServers": {
    "local-models": {
      "command": "/path/to/mcp-ollama-vllm/.venv/bin/python",
      "args": ["-m", "mcp_ollama_vllm.server"],
      "env": {
        "PYTHONPATH": "/path/to/mcp-ollama-vllm/src",
        "LOCAL_BACKEND": "vllm",
        "LOCAL_HOST": "http://192.168.x.x:8000",
        "LOCAL_TIMEOUT": "600",
        "LOCAL_EMBED_MODEL": "BAAI/bge-m3"
      }
    }
  }
}
```

Both backends can also be registered in parallel, under different names (e.g.
`local-models` and `local-models-server`), so both are available in the same
session.

### Registering the same server multiple times

There is nothing wrong with registering the same server under several names at
once, each with its own `LOCAL_BACKEND`/`LOCAL_HOST`. That way different
backends can be addressed in the same session, for example:

- one entry for a local Ollama (`http://localhost:11434`),
- another for a vLLM on a different machine on your network,
- a third for an Ollama on that same remote machine.

All three are then instances of one and the same server, just configured
differently. Give them descriptive names and check before use which entry hits
the intended machine and backend, rather than relying on the name alone.

The `openai` backend works the same way, no new routing mechanism: two
generic-server entries on different ports, for example LM Studio on
`http://localhost:1234` and a llama.cpp server on `http://localhost:8080`:

```json
{
  "mcpServers": {
    "local-models-lmstudio": {
      "command": "/path/to/mcp-ollama-vllm/.venv/bin/python",
      "args": ["-m", "mcp_ollama_vllm.server"],
      "env": {
        "PYTHONPATH": "/path/to/mcp-ollama-vllm/src",
        "LOCAL_BACKEND": "openai",
        "LOCAL_HOST": "http://localhost:1234"
      }
    },
    "local-models-llamacpp": {
      "command": "/path/to/mcp-ollama-vllm/.venv/bin/python",
      "args": ["-m", "mcp_ollama_vllm.server"],
      "env": {
        "PYTHONPATH": "/path/to/mcp-ollama-vllm/src",
        "LOCAL_BACKEND": "openai",
        "LOCAL_HOST": "http://localhost:8080"
      }
    }
  }
}
```

If the endpoint used for chat does not also serve embeddings (see
`local_embed` below), register a second entry with a matching
`LOCAL_EMBED_PATH`, or one pointing at a separate, embedding-capable backend
entirely, rather than expecting one endpoint to do both.

The timeout is intentionally generous: local models can take minutes with a
long context or under load. The connection setup has its own short limit of
10 seconds, independent of that, so a switched-off Ollama service surfaces
immediately instead of hanging for ten minutes.

### Other MCP clients

The server speaks plain stdio MCP, so any client that supports a
`command`/`args`/`env` style server entry works the same way. The exact
config file and key names differ per client; check its docs for where an
`mcpServers`-style block (or equivalent) lives. Two generic examples:

```jsonc
// Checkout-based, PYTHONPATH points at src/:
{
  "mcpServers": {
    "local-models": {
      "command": "/path/to/mcp-ollama-vllm/.venv/bin/python",
      "args": ["-m", "mcp_ollama_vllm.server"],
      "env": {
        "PYTHONPATH": "/path/to/mcp-ollama-vllm/src",
        "LOCAL_BACKEND": "ollama"
      }
    }
  }
}
```

```jsonc
// After `pip install .`, using the installed console script directly:
{
  "mcpServers": {
    "local-models": {
      "command": "/path/to/mcp-ollama-vllm/.venv/bin/mcp-ollama-vllm",
      "env": { "LOCAL_BACKEND": "ollama" }
    }
  }
}
```

## Tools

### `list_models`

No parameters. Returns per model the name, parameter size, quantization,
on-disk size (GB and bytes), family and capabilities. The capabilities show,
among other things, which models can do `embedding`.

With **vLLM** the backend does not report size and quantization; those fields
then read `unknown` and `0` respectively, instead of inventing a value. In return,
three fields are populated there:

- `is_adapter`: whether the entry is a LoRA adapter (vLLM exposes adapters as
  their own entries in the model list). Ollama's entries carry this field too,
  always `False`, since Ollama has no adapter concept.
- `base_model`: which base model the adapter belongs to (from the `parent`
  field). Ollama's entries carry this field too, always empty.
- `max_context`: the model's context length (`max_model_len`), where vLLM
  reports it. This one is exclusive to vLLM; Ollama's entries do not carry it
  at all.

At the top level there is additionally `adapters` as a count, `backend` as the
identifier of the active backend, and `metadata_reported` (`True`/`False`):
whether the backend actually reports the metadata fields above, so a caller
can tell "not reported by this backend" (`openai`, generally `False`) from
"reported, but happens to be empty".

### `local_ask`

| Parameter | Required | Default | Meaning |
|---|---|---|---|
| `model` | yes | - | Model name, e.g. `llama3.2:3b` |
| `prompt` | yes | - | The instruction |
| `system` | no | - | Role/behavior instruction |
| `temperature` | no | `0` | 0 = as deterministic as possible |
| `max_tokens` | no | - | Cap on generated tokens |
| `context` | no | - | Text prepended to the prompt |
| `think` | no | `false` | Enable the model's reasoning trace (see below) |

Return: `answer`, `model`, `model_digest`, `backend`, `duration_seconds`,
`tokens` (`input`/`output`/`total`), `tokens_per_second`, `finish_reason`.
Model name plus digest make a run traceable, even if the same label later points
at different weights.

#### The `think` parameter

Thinking models (capability `thinking` in `list_models`, e.g. `qwen3:4b`) reason
before the actual answer. This reasoning trace counts against the same token
budget as the answer. With a tight `max_tokens` the answer is therefore
**empty**, even though the call technically succeeded; measured were ~3000 to
4000 reasoning tokens per call.

That is why `think` defaults to `false`, and the bridge sends `think: false`
with every Ollama call. Whoever explicitly wants the reasoning trace sets
`think: true`.

The `think` field is deliberately **always** sent, even as `false`. Measured
against Ollama 0.31.2: `think: false` is harmless on a model without the
`thinking` capability, only `think: true` is rejected there with
`HTTP 400 "... does not support thinking"`. A preliminary capability lookup via
`/api/tags` is thus unnecessary, and the error message in the single problem
case names the cause clearly.

With **vLLM** the bridge accepts the parameter but does not act on it: the
thinking control there is not yet wired up. The call form is kept identical for
both backends on purpose. The generic **`openai`** backend accepts and ignores
`think` the same way as vLLM, for the same reason: no generic reasoning
control (`reasoning_effort`, `chat_template_kwargs`, ...) is in scope here.

### `local_structured`

| Parameter | Required | Meaning |
|---|---|---|
| `model` | yes | Model name |
| `prompt` | yes | Instruction on what to extract from which text |
| `schema` | yes | JSON schema of the desired result |
| `system` | no | Role/behavior instruction |
| `think` | no | Enable the model's reasoning trace, default `false` (see `local_ask`) |

Passes the schema to the backend's schema enforcement and **additionally
validates the answer against the schema itself**. With Ollama this goes through
the `format` field, with vLLM through the OpenAI-compatible `response_format`
with `{"type": "json_schema", "json_schema": {...}}` (as the current vLLM docs
recommend). If an older vLLM build rejects that with HTTP 400, it falls back
once to the older `guided_json`, so the bridge stays usable there too. With the
generic **`openai`** backend the default is `response_format` with
`json_schema` as well; if the server rejects that with HTTP 400, it falls back
(unlike vLLM, there is no `guided_json` to try) to a schemaless call, whose
result the built-in validator checks and, on a violation, retries with the
concrete violations fed back, same as any other unvalidated answer. Which path
was accepted is remembered process-wide, so later calls skip the doomed
`response_format` roundtrip. `LOCAL_SCHEMA_MODE` overrides this detection:
`response_format` always requires it (a rejection is then a hard error, not a
fallback), `none` never sends it. On a violation, up to two retries, passing
the model the concrete violations. Without
that feedback a retry at temperature 0 would be word-for-word identical and thus
pointless. Only then an error, but then with the violations and the invalid raw
output.

Return: `outcome` (parsed object), `attempts`, `valid_on_first_try`, `model`,
`model_digest`, `backend`, `duration_seconds`, `tokens`.

The validation covers the subset that `format` uses in practice: `type` (also
lists of types), `properties`, `required`, `additionalProperties`, `items`
(also tuple form), `enum`, `const`, `minimum`/`maximum`,
`minLength`/`maxLength`, `minItems`/`maxItems`, `anyOf`/`oneOf`/`allOf`. Not a
full draft: `$ref`, `pattern` and `format` are **not** validated. Any keyword
not listed above is silently ignored rather than enforced, among others
`exclusiveMinimum`/`exclusiveMaximum`, `multipleOf`, `uniqueItems`,
`patternProperties`, `if`/`then`/`else` and `contains`.

### `local_embed`

| Parameter | Required | Default | Meaning |
|---|---|---|---|
| `texts` | yes | - | List of texts to embed |
| `model` | no | `nomic-embed-text` (Ollama) | Embedding model |

Return: `vectors`, `count`, `dimensions`, `model`, `backend`, `duration_seconds`.

## Self-tests

Two separate tests. `selftest.py` runs against the real Ollama; `selftest_vllm.py`
is a fast offline mock gate that needs no backend at all. Beyond these two, the
vLLM paths were in addition confirmed once against a real vLLM (see "What was
measured against a real vLLM" below).

```bash
.venv/bin/python selftest.py --rounds 10 --model llama3.2:3b   # against real Ollama
.venv/bin/python selftest_vllm.py                              # against a mock
```

Both run without an MCP client. `selftest.py` calls the tools through the real
FastMCP dispatch (`mcp.call_tool`), i.e. the same path a connected client takes,
and pulls no models itself.

The Ollama test first checks the schema validator itself against a battery of
known violations (the exact count grows as the validator's keyword coverage
grows). Without this step a high schema hit rate would be worthless: it
could also mean the validator simply flags nothing. Then the retry and abort
logic is checked deterministically with replaced HTTP access, because a schema
violation cannot be produced on command at the real model.

`selftest_vllm.py` starts its own tiny HTTP server that mimics the three
OpenAI-compatible endpoints and shuts it down again afterwards. It checks
**both**: that the answers are read correctly and that the sent requests are
formed correctly (that `response_format` with `json_schema` is actually sent and
not Ollama's `format`). It stays useful as a fast, offline, deterministic gate
even now that the vLLM paths have also been confirmed against a real vLLM (see
"What was measured against a real vLLM" below): the mock needs no backend and
always runs the same. Your own vLLM build may still behave differently, so a
check against it stays worthwhile.

## Measured limits

The following numbers come from a test run on a single laptop with tight video
memory (about 4 GB VRAM), Ollama with `-np 1`, i.e. **a single processing
slot**. Read them as an order of magnitude, not a promise; on other hardware the
runtimes come out differently.

### What is fully green

| Area | Result |
|---|---|
| Schema validator against known bad schemas | each correctly detected |
| Retry and abort logic (without a model) | 8/8 |
| `list_models` (Ollama) | 8 models, fields complete |
| `local_ask` answer, tokens, time | correct; reported duration matches externally measured to within 0.04 s |
| `local_ask` `context` and `max_tokens` | both effective (`max_tokens: 16` yielded exactly 16 output tokens) |
| `local_embed` | 3 vectors, 768 dimensions, 2.05 s |
| meaning of the vectors | similar sentences 0.815, dissimilar 0.574 |
| error paths (5 cases) | each an understandable message, no stacktrace |
| stdio handshake through a real MCP client | all four tools visible with correct required fields |
| network guard | LAN and loopback allowed, public host and public IP rejected |
| **vLLM paths against the mock** | **70/70** |

### Schema fidelity, the decisive metric

Collected with `qwen3:4b` over 30 different inputs, 10 per schema. Measured was
how often the output was schema-conform **on the first try**, i.e. without the
built-in retry:

| Schema | n | conform on 1st try | failed | Duration s (min/median/max) |
|---|---|---|---|---|
| A flat (`enum`, numeric bounds, `additionalProperties: false`) | 10 | **10/10** | 0 | 35 / 56 / 516 |
| B nested object (2 levels) | 10 | **10/10** | 0 | 56 / 93 / 832 |
| C list of objects (`minItems`, `enum` per element) | 10 | **10/10** | 0 | 63 / 107 / 827 |
| **Total** | **30** | **30/30 (100 %)** | **0** | 35 / 93 / 832 |

Every delivered object additionally passed the independent re-validation against
its schema, and the contents were factually correct (example B: sender, net
amount, currency and reminder flag extracted correctly from prose).

**How to read these 100 %.** The backend's grammar binding already forces the
JSON form at generation time; the rate therefore mainly measures whether the
schemata are tight enough, not whether the model is clever. It holds for these
three schemata, for this model, and for short German texts. It weakens as
expected for schemata full of free strings without `enum` and without bounds.
The retry loop was **not needed a single time** in this series, and it was not
triggered in the real vLLM run either (`local_structured` came back valid on the
first try there too, `attempts: 1`); its correctness is instead proven
deterministically with replaced HTTP access.

### On the runtime, and what skewed it

The spread from 35 s to 832 s does not come from the server. During the first
half of the measurement a second, unrelated test ran on the same machine (at
times a Python process at 1011 % CPU, load average 21 to 28). Ollama runs here
with `-np 1`, i.e. **a single processing slot** for the whole machine. Measured
consequences:

- Under foreign load a call took 309 to 832 s, without it 35 to 107 s.
- One call ran into the default timeout of 600 s and failed.
- A switch to `llama3.2:3b` was **impossible at all** under foreign load: as
  long as another model holds the slot, the request for a not-yet-loaded model
  waits indefinitely. Two attempts aborted after 400 s with no answer.
- The obvious way out, `num_gpu: 0` (compute purely on the CPU to sidestep the
  4-GB-VRAM contention), did **not** help, because the bottleneck was the CPU,
  not the GPU.

None of this is a fault of this server, but it is the practical limit of the
approach on a single machine, and that is why it is here.

### What was measured against a real vLLM

The vLLM paths are backed against a mock server: correct endpoints, correctly
formed requests (`response_format` with `json_schema`), fallback to
`guided_json`, LoRA detection, unified metadata, understandable errors. Beyond
that, the tools were run once against a **real** vLLM: an OpenAI-compatible
endpoint serving `Qwen2.5-1.5B-Instruct` (`max_model_len` 8192), driven through
the real FastMCP dispatch (`mcp.call_tool`), the same path a connected client
takes.

- `list_models` -> `GET /v1/models`: backend reported as `vllm`, the model
  recognized, `max_context` 8192, `adapters` 0.
- `local_ask` -> `POST /v1/chat/completions`: answer text, token counts
  (input/output/total) and `tokens_per_second`, `finish_reason` `stop`.
- `local_structured` -> `POST /v1/chat/completions` with `response_format`
  `json_schema`: the schema-conform object back, valid on the first try
  (`valid_on_first_try: true`, `attempts: 1`). This is exactly the
  schema-enforcement path, now confirmed against a real vLLM and not only the
  mock.
- `local_embed` -> `POST /v1/embeddings`: 404 from the backend, which the server
  caught cleanly as a clear message ("backend reports: Not Found ...; with vLLM
  the model is fixed when the service starts"). This is correct behavior, not a
  fault: the test vLLM served a chat model, not an embedding model, so there was
  no embedding endpoint to hit, and the server said so plainly.

What genuinely stays open, therefore, is the vLLM **embedding** path: it was
measured against a real Ollama, but not against a vLLM deployment that serves an
embedding model (the test vLLM was chat-only). Whoever uses vLLM embeddings must
run an embedding-capable vLLM; against a chat-only vLLM the server deliberately
returns a clear error. Beyond that, your own build may still differ in whether
it takes `response_format` or the legacy `guided_json` path, and in schema
fidelity with a larger model and a LoRA adapter.

### What follows for operation

- **One model at a time.** With `-np 1` the local Ollama instance is a resource
  two parallel agents cannot share. Before a longer run, a glance at
  `curl localhost:11434/api/ps` pays off.
- **The 600 s timeout is tight, not generous.** Under foreign load it was
  exceeded. For measurement runs set `LOCAL_TIMEOUT` higher.
- **Thinking models stay more expensive, but are no longer dangerous.**
  `qwen3:4b` produced 186 output tokens for the answer "Paris", because it
  reasons before answering. For classifying and extracting, a model without a
  thinking step (`llama3.2:3b`, `granite4:micro`) remains faster and cheaper.
  For qwen3 variants the `think` parameter (default `false`) controls the
  reasoning trace: the bridge switches it off by itself, and a tight
  `max_tokens` then yields an answer instead of an empty string.
- **Build long runs to be resumable.** Otherwise every abort under foreign load
  destroys all values collected so far.

## Pitfalls

- **Ollama's `format` does not guarantee a valid result.** The grammar binding
  forces the JSON form, but not reliably `required`, `enum` or numeric bounds.
  That is exactly why this server additionally validates itself. Whoever uses
  `format` without their own validation gets silently incomplete objects.
- **Small models need a tight schema.** The more `enum`, `required` and bounds
  are set, the more usable the result. A schema of nothing but free strings
  invites the model to make things up.
- **VRAM contention is the biggest time sink.** If two requested models do not
  fit into video memory at once, Ollama reloads on every switch. A call then
  takes minutes instead of seconds, without anything being wrong at the server.
  Work with a single model where possible.

## License

MIT, see [LICENSE](https://github.com/setheerwagen/local-ai-mcp-servers/blob/main/LICENSE).
