Metadata-Version: 2.4
Name: llm-llamacpp-plugin
Version: 0.1.2
Summary: LLM plugin providing access to llama.cpp server models
Author: Sukhbinder Singh
License: Apache-2.0
Project-URL: Homepage, https://github.com/sukhbinder/llm-llamacpp
Project-URL: Changelog, https://github.com/sukhbinder/llm-llamacpp/releases
Project-URL: Issues, https://github.com/sukhbinder/llm-llamacpp/issues
Classifier: License :: OSI Approved :: Apache Software License
Description-Content-Type: text/markdown
Requires-Dist: llm>=0.26
Requires-Dist: httpx
Requires-Dist: httpx-sse
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Requires-Dist: pytest-httpx; extra == "test"
Requires-Dist: pytest-asyncio; extra == "test"

# llm-llamacpp

[![PyPI](https://img.shields.io/pypi/v/llm-llamacpp-plugin.svg)](https://pypi.org/project/llm-llamacpp-plugin/)
[![Changelog](https://img.shields.io/github/v/release/sukhbinder/llm-llamacpp-plugin?include_prereleases&label=changelog)](https://github.com/sukhbinder/llm-llamacpp-plugin/releases)
[![Tests](https://github.com/sukhbinder/llm-llamacpp-plugin/workflows/Test/badge.svg)](https://github.com/sukhbinder/llm-llamacpp-plugin/actions?query=workflow%3ATest)
[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/sukhbinder/llm-llamacpp-plugin/blob/main/LICENSE)


A plugin for [LLM](https://llm.datasette.io/) providing access to models running on a [llama.cpp](https://github.com/ggerganov/llama.cpp) server.

## Installation

Install this plugin in the same environment as LLM:

```bash
llm install llm-llamacpp-plugin
```

## Setup

First, you need to have a llama.cpp server running. You can start one using the llama.cpp server binary:

```bash
# Download and build llama.cpp if you haven't already
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp

# Build the server
make

# Start the server with your model
./llama-server -m models/your-model.gguf -c 4096
```

The server will start on `http://localhost:8080` by default.

## Usage

Once the plugin is installed and your llama.cpp server is running, you can use it like any other LLM model:

```bash
llm -m llamacpp "Your prompt here"
```

![llamacpp demo](https://raw.githubusercontent.com/sukhbinder/llm-llamacpp-plugin/refs/heads/main/llamacpp-cli-help-lossless-o3.gif)

### Using a different server URL

If your llama.cpp server is running on a different host or port, you can set the `LLM_LLAMACPP_SERVER` environment variable:

```bash
export LLM_LLAMACPP_SERVER=http://your-server:port
```

in windows

```cmd
setx LLM_LLAMACPP_SERVER http://your-server:port
```

### Conversations

You can use conversations just like with other models:

```bash
llm -m llamacpp "First message"
llm -c "Follow-up question"
```

### JSON Schema

You can request JSON output using LLM's schema feature:

```bash
llm -m llamacpp-tools "Generate a person" --schema '{"name": "string", "age": "integer"}'
```

### Vision Models

If you're running a vision-capable llama.cpp model with multimodal support, the plugin can handle image attachments.

```bash
llm -m llamacpp-vision -a /tmp/screen.png "What is the user doing in this screenshot"
```

### Embedding Models

The plugin also supports embedding models running on llama.cpp server. To use embeddings:

```bash
# Start the server with embedding support
./llama-server -m models/embedding-model.gguf --embedding
```

Then use it with LLM:

```bash
# Get embeddings for text
llm embed -m llamacpp-embed "Hello world"

# Get embeddings for multiple items
llm embed -m llamacpp-embed "First text" "Second text"
```

## Development

To install the plugin for development:

```bash
cd llm-llamacpp
pip install -e .
```

Run the tests:

```bash
pytest
```

## License

Apache 2.0
