Metadata-Version: 2.1
Name: geistr
Version: 0.3.4
Summary: Run language models in your Python process (geist-runtime)
Description-Content-Type: text/markdown
Home-page: https://github.com/geisten/geist-runtime
License: see LICENSE
Requires-Python: >=3.9

# geistr

Run language models on your own computer: download, chat, serve. One small
binary, no Python, no Docker, no account. The model runs locally, on the CPU
or the GPU.

> [!WARNING]
> **geistr is at an early stage (v0.3.4).** It works and every change is tested
> on Linux and macOS, but expect rough edges: commands, settings and file formats may
> still change between versions, and the model list is small. Please
> [report what breaks or confuses you](https://github.com/geisten/geist-runtime/issues);
> early feedback shapes what comes next ([roadmap](ROADMAP.md)).

**Full manual:** [docs/GUIDE.md](docs/GUIDE.md): every command, chat
command, setting and file, the HTTP API, and what to do when something fails.

![geistr chat in action: qwen3-0.6b answers on the CPU, /model switches to gemma4-e2b on the GPU for a table, and a new session continues the conversation](docs/demo.gif)

## Install

```sh
curl -fsSL https://www.geisten.net/download/geistr/install.sh | sh
```

Or with [Homebrew](https://brew.sh):

```sh
brew install geisten/tap/geistr
```

Both work on Linux (x86_64, arm64) and macOS (Apple Silicon). The script
downloads the latest release, checks it against `SHA256SUMS` and puts
`geistr` in `/usr/local/bin`.

Without sudo, or a specific release (the server keeps the last 5, from 0.3.2
on; another one is reported as "no release …"):

```sh
curl -fsSL https://www.geisten.net/download/geistr/install.sh | PREFIX=~/.local sh
curl -fsSL https://www.geisten.net/download/geistr/install.sh | GEISTR_VERSION=v0.3.4 sh
```

<details>
<summary>Requirements</summary>

The Linux binaries are fully static and run on any distribution. The CPU
needs x86-64-v3 (AVX2: Intel since 2013, AMD since 2015) or ARMv8.2 with
dotprod (Raspberry Pi 5, AWS Graviton 2 and newer); on an older CPU geistr
says so and exits. To build from source instead, see
[docs/DEVELOPMENT.md](docs/DEVELOPMENT.md#build-geistr-from-source).
</details>

## Quick start

```sh
geistr chat                    # the first time: suggests a model for this computer,
                               # downloads and verifies it on Enter, then chats
```

Or choose yourself:

```sh
geistr catalog                 # which models fit this computer
geistr pull qwen3-0.6b         # download and verify one (640 MB)
geistr chat qwen3-0.6b         # talk to it
```

Or a single answer, for scripts:

```sh
geistr run qwen3-0.6b "Name three prime numbers."
echo "Summarize: …" | geistr run qwen3-0.6b
```

## Models

`geistr catalog` lists the models geistr knows, the best suited to this
computer first (installed or not does not move a row), marks what is installed
(✓), what is available (↓) and whether each fits your RAM (⚠/✗), with the speed
measured here on ⚙ CPU and ⚡ GPU.

| model | download | RAM |
|---|---|---|
| `smollm2-360m` | 0.4 GB | 2 GB |
| `qwen3-0.6b` | 0.6 GB | 4 GB |
| `qwen35-0.8b` | 0.8 GB | 4 GB |
| `bitnet-2b` | 1.2 GB | 4 GB |
| `gemma4-e2b` | 3.1 GB | 8 GB |
| `gemma4-e4b` | 5.0 GB | 16 GB |
| `bonsai2-27b-pq2` | 7.2 GB | 24 GB |
| `qwen38-27b-q4` | 16 GB | 32 GB |
| `qwen38-27b-q8` | 29 GB | 48 GB |

Every download is checked against its SHA-256 and resumes where it stopped.
Any other `.gguf` file works too: `geistr chat path/to/model.gguf`.

## Chat

In a terminal, `geistr chat` renders Markdown, tables, math and clickable
links, shows the
speed after each answer and continues your last conversation (see below).
Once the context is half full, the prompt says how full (`⚙ 62% >`, yellow
from 80 %, red from 95 %); `/info` always shows it. Type `/` for the commands:

```
/model gemma4-e2b    switch the model, keep the conversation (best fit first)
/gpu /cpu /auto      switch the processor (Metal on macOS, Vulkan on Linux)
/temp 0.7            sampling temperature
/system Be brief.    system prompt (/system off removes it)
/info                what runs now, and how full the context is
/files ~/Notes       answers from your files (/files off: none)
/retry               the last answer again (at temperature 0: once at 0.7)
/copy  /copy code    the last answer, or its last code block, to the clipboard
/save                keep model, processor, temperature and system prompt
/clear  /exit
```

| key | does |
|---|---|
| Ctrl-J, Alt-Enter, `\` + Enter | a new line in the message (Enter sends) |
| Esc | stop the answer |
| Ctrl-C | clear the line; twice on an empty line: exit |
| ↑ ↓ | earlier lines, also from earlier chats; → takes the hint |
| Ctrl-R | search earlier lines (type to narrow, Ctrl-R for older, Enter takes it) |
| `?` | the shortcuts |

### Your conversation stays

Close the terminal, reboot, come back tomorrow: `geistr chat` picks up where
you left off, and the model still knows what you talked about.

```
↻ 6 · „Compare Mars and Venus in a small table…“ · /clear new
⚡ >
```

That is what makes a local model useful for real work: you can think a
problem through over several sessions, refer back to an earlier answer, or
switch to a bigger model mid-conversation without explaining everything
again. It is also private by design: each conversation is a plain file on
your computer (readable only by you, one JSON message per line), never sent
anywhere.

A long conversation stays fast to resume: the model re-reads only the
newest part (about 2000 tokens, `geistr config resume_tokens`), and the
`↻` line says how much. The whole conversation stays in the file. When a
conversation outgrows the model's context, the oldest messages are left
out, and the chat says so (`↥ 12 oldest messages left out …`).

- `/clear` starts a new conversation and keeps the old file.
- `geistr chat --new` starts fresh once; `geistr config resume off` always.
- Piped chats (`echo … | geistr chat`) keep nothing, so scripts stay reproducible.

### Ask your own files

```
geistr pull bitnet-embed-0.6b      # the embedding model, 428 MB, once
geistr index ~/Notes               # text, Markdown, PDF (with pdftotext)
geistr chat --files ~/Notes        # or /files ~/Notes in a chat
```

Each question gets the most relevant passages of your files put in front of
it, and the answer shows where they came from (`sources: garden/shed.md`).
With the reranker installed (`geistr pull qwen3-reranker-0.6b`, 639 MB) only
the passages that answer the question go in front of it and only their files
are named; for a question your files do not answer it says `nothing relevant
in your files`. Without it, the four nearest passages are used and named.
`geistr run --files ~/Notes "…"` does the same once. The index is updated by
file hash whenever you use the folder, so only new and changed files are
read again. It stays on your computer, readable only by you, in the data
folder (`index/`); hidden files and folders are left out.

## Use it from other tools

Keep a model loaded and talk to it over the OpenAI or Ollama API, e.g. from
Open WebUI, editor plugins or the OpenAI SDK:

```sh
geistr serve gemma4-e2b --http          # http://127.0.0.1:11434
```

```sh
curl http://127.0.0.1:11434/v1/chat/completions \
  -d '{"messages":[{"role":"user","content":"Hi"}]}'
```

```python
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:11434/v1", api_key="unused")
reply = client.chat.completions.create(model="gemma4-e2b",
                                       messages=[{"role": "user", "content": "Hi"}])
print(reply.choices[0].message.content)
```

In Open WebUI, add `http://127.0.0.1:11434` as an Ollama connection (or
`…/v1` as OpenAI). Clients that resend the whole conversation pay only for
what is new.

<details>
<summary>Endpoints, the socket, and security</summary>

| endpoint | |
|---|---|
| `POST /v1/chat/completions` | OpenAI: `messages`, `temperature`, `max_tokens` / `max_completion_tokens`, `stop`, `stream` (SSE, `stream_options.include_usage`), `tools` / `tool_choice` |
| `POST /v1/embeddings` | OpenAI: `input` (a text or a list), `encoding_format` (`float`, `base64`); with an embedding model |
| `GET /v1/models` | the one model |
| `POST /api/chat` | Ollama: `messages`, `stream` (NDJSON, default), `options.temperature`, `num_predict`, `stop`, `tools` |
| `POST /api/embed` | Ollama: `input` (a text or a list); with an embedding model |
| `GET /api/tags`, `/api/version`, `/` | the model, the version, a health check |

There is no authentication. By default geistr listens on loopback only and
answers only requests addressed to this computer (`Host` check, against DNS
rebinding). A web page in a browser may use it only if it comes from this
computer too (`Origin` check: another site's page cannot start answers);
`GEISTR_ORIGINS=https://ui.example,…` (or `*`) allows more, like Ollama's
`OLLAMA_ORIGINS`. `--http=0.0.0.0:11434` makes it reachable for everyone who can
reach the computer; geistr warns about that. One request is answered at a
time; a client that disconnects stops its answer.

**Tool calling** works with Qwen3 models (e.g. `qwen3-0.6b`): send `tools`,
get `tool_calls` back (`finish_reason: tool_calls`; streamed too), send the
results as `role: tool` messages, and the model answers with them, as in the
OpenAI SDK's function-calling example. `tool_choice: "none"` turns tools off
for a request. A model without a tool format refuses `tools` (400).

**Images** work with Gemma 4 E2B (`gemma4-e2b`): `geistr pull gemma4-e2b`
also fetches its vision tower (the vision tensors of the original model, a
337 MB `vision_tower.safetensors` next to it, SHA-256 verified). Send an
`image_url` part with a `data:` URL (OpenAI) or `images` (Ollama), PNG, JPEG
or BMP, one per message; the image goes with the last message, and the
conversation keeps it for the follow-up questions. Other models refuse images
(400).

**Embeddings**: `geistr serve bitnet-embed-0.6b --http` answers
`/v1/embeddings` (the OpenAI SDK's `embeddings.create` works) and Ollama's
`/api/embed`: unit-length vectors of 1024 dimensions, multilingual. For
retrieval, embed the question and the documents as they are, with nothing
before them, as `--files` does: on `scripts/retrieval-bench.py` (#157) that
put the right file among the four nearest passages most often (97 %), ahead of
an `Instruct: …\nQuery: ` task before the question (92 %) or `query: ` /
`passage: ` (89–94 %). An embedding model does not chat, and a chat model does not
embed (400). Not supported yet: more than one model per service.

Without `--http`, `geistr serve` listens on a Unix socket only (owner only,
0600), and `geistr chat --socket` chats through it. `--chats N` sets how many
conversations the service keeps (default 2), `--max-request=MiB` the size of one
request (default 16). The options and the socket protocol:
[docs/GUIDE.md](docs/GUIDE.md#serve-use-the-model-from-other-programs).
</details>

## Settings

Settings are remembered between runs; `geistr config` shows them and the
file they live in.

```sh
geistr config processor gpu                # auto (default), cpu, gpu
geistr config temperature 0.7              # 0 to 2
geistr config system "Answer in German."   # system prompt for every new chat
geistr config markdown off                 # plain text
geistr config stats off                    # no speed line
geistr config resume off                   # always start new, keep nothing
geistr config history off                  # don't keep typed lines between chats
geistr config threads 8                    # CPU threads (default: one per physical core)
```

`--cpu` / `--gpu` choose the processor for one run, `--threads N` the CPU
threads, `--models DIR` another model folder.

<details>
<summary>All commands</summary>

| command | does |
|---|---|
| `geistr catalog [--installed \| --available] [--json]` | the models, their state, fit and speed |
| `geistr pull <id>` | download, resume, verify |
| `geistr pull` | update the installed models after a geistr update |
| `geistr run <model> [prompt…]` | one answer to stdout; the prompt from stdin if none |
| `geistr chat [<model>] [--new]` | interactive; without a model the last one |
| `geistr serve <model> [--http[=ADDR:PORT]] [--socket=PATH] [--chats N] [--max-request=MiB]` | the model as a service |
| `geistr chat --socket[=PATH]` | chat with that service |
| `geistr config [key [value]]` | settings |
| `geistr bench [model…]` | tokens/s on CPU and GPU (default: every installed model) |
| `geistr bench --compare [A [B]]` | two engine versions' speeds side by side |
| `geistr --version` | geistr, catalog revision, engine commit |

`<model>` is a catalog id or a path to a `.gguf` file. Options:
`--cpu` / `--gpu`, `--models DIR`, `--catalog FILE`. Exit codes: 0 ok,
1 error, 2 usage, 130 cancelled.

Every complete answer records its speed in `speed.tsv` in the data folder;
`geistr catalog` shows the median of the last ten.
</details>

## Update

`brew upgrade geistr`, or run the install command again. The model list
ships inside geistr, so new models come with a new geistr. Afterwards `geistr catalog` marks installed
models whose file changed with ⟳ and a file of the right size with another
checksum (damaged?) with ✗, and `geistr pull` fetches them (the old file stays
until the new one is verified).

## Python and C

geistr is built on **geist-runtime**, an embeddable model runner on
[geistlib](https://github.com/geisten/geistlib) with one C API
([`include/geistr.h`](include/geistr.h)) and a Python package
(`pip install geistr`; every release also has its wheels at
`https://www.geisten.net/download/geistr/vX.Y.Z/`):

```python
import geistr

with geistr.chat("gemma4-e2b", system="Answer briefly.") as chat:
    for piece in chat.send("What is the capital of France?"):   # streamed
        print(piece, end="", flush=True)
    print(chat.ask("And of Italy?"))           # only the new message is processed
```

Using geistr: [docs/GUIDE.md](docs/GUIDE.md).
Building, the library, tests and CI: [docs/DEVELOPMENT.md](docs/DEVELOPMENT.md).
API reference and design: [docs/API.md](docs/API.md).
