Metadata-Version: 2.5
Name: rishi
Version: 0.1.45
Summary: fast local llm driver with tools
Project-URL: Repository, https://github.com/vedicreader/rishi
Project-URL: Documentation, https://vedicreader.github.io/rishi/
Author-email: Karthik <karthik.rajgopal@hotmail.com>
License: Apache-2.0
License-File: LICENSE
Keywords: nbdev
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Requires-Python: >=3.12
Requires-Dist: aidialog>=0.0.10
Requires-Dist: claude-agent-sdk>=0.2
Requires-Dist: fastcore>=2.0.0
Requires-Dist: httpx
Requires-Dist: huggingface-hub>=1.23.0
Requires-Dist: liteparse<2.13
Requires-Dist: litert-lm-api>=0.14.0
Requires-Dist: llmsurgery>=0.0.13
Requires-Dist: python-fastllm>=0.0.45
Requires-Dist: safepyrun>=0.2.3
Requires-Dist: truststore>=0.10
Requires-Dist: uraiyadal>=0.0.12
Provides-Extra: all
Requires-Dist: llama-cpp-python==0.3.30; extra == 'all'
Requires-Dist: mlx-lm>=0.31.3; (sys_platform == 'darwin' and platform_machine == 'arm64') and extra == 'all'
Requires-Dist: mlx-vlm>=0.6.8; (sys_platform == 'darwin' and platform_machine == 'arm64') and extra == 'all'
Requires-Dist: numpy; extra == 'all'
Requires-Dist: soundfile>=0.14.0; extra == 'all'
Requires-Dist: zstandard>=0.22; (sys_platform == 'linux' and python_version < '3.14') and extra == 'all'
Provides-Extra: dev
Requires-Dist: dhrishti>=0.0.5; extra == 'dev'
Requires-Dist: diskcache>=5.6.3; extra == 'dev'
Requires-Dist: ipykernel>=7.3.0; extra == 'dev'
Requires-Dist: koshas>=0.0.31; extra == 'dev'
Requires-Dist: llama-cpp-python==0.3.30; extra == 'dev'
Requires-Dist: mlx-lm>=0.31.3; (sys_platform == 'darwin' and platform_machine == 'arm64') and extra == 'dev'
Requires-Dist: mlx-vlm>=0.6.8; (sys_platform == 'darwin' and platform_machine == 'arm64') and extra == 'dev'
Requires-Dist: nbdev>=3.1.0; extra == 'dev'
Requires-Dist: notebook>=7.6.0; extra == 'dev'
Requires-Dist: numpy; extra == 'dev'
Requires-Dist: pyskills>=0.0.21; extra == 'dev'
Requires-Dist: soundfile>=0.14.0; extra == 'dev'
Requires-Dist: zstandard>=0.22; (sys_platform == 'linux' and python_version < '3.14') and extra == 'dev'
Provides-Extra: llama
Requires-Dist: llama-cpp-python==0.3.30; extra == 'llama'
Requires-Dist: numpy; extra == 'llama'
Requires-Dist: soundfile>=0.14.0; extra == 'llama'
Provides-Extra: mlx
Requires-Dist: mlx-lm>=0.31.3; (sys_platform == 'darwin' and platform_machine == 'arm64') and extra == 'mlx'
Provides-Extra: mlx-vlm
Requires-Dist: mlx-vlm>=0.6.8; (sys_platform == 'darwin' and platform_machine == 'arm64') and extra == 'mlx-vlm'
Provides-Extra: ollama
Requires-Dist: zstandard>=0.22; (sys_platform == 'linux' and python_version < '3.14') and extra == 'ollama'
Description-Content-Type: text/markdown

# rishi


<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->

rishi lets you talk to a language model from Python with the same few calls, whether the model runs on your own machine or on someone else’s server. Each way of running a model is a backend. Four run models locally: LiteRT-LM, llama.cpp, MLX and Ollama. Three send the conversation elsewhere: to a hosted API such as OpenAI’s or Anthropic’s, to your Claude Code installation, or to GitHub Copilot. Each of these has its own message format, its own way of passing images and tool calls, and its own idea of what a conversation is.

rishi keeps one conversation format for all of them. The conversation, called the history, is a plain Python list of message dicts in the OpenAI shape (`{'role': 'user', 'content': 'hi'}`), stored on `chat.hist`. Every backend reads and writes that list, so you can start a conversation on a small local model and hand the list to a hosted one, which carries on as though it had been there from the start.

The parts every backend shares come from [Urai](https://vedicreader.github.io/urai), published on PyPI as `uraiyadal`: the `Chat` class, tool calling, streaming, options and recorded replies. rishi adds the seven backends and registers them with Urai. `from rishi import *` gives you Urai’s names as well as rishi’s.

## Install

The base package includes LiteRT-LM and the three backends that send the conversation elsewhere. The other local runtimes are extras, so you install only the ones you use:

``` sh
pip install rishi                # LiteRT-LM, hosted APIs, Claude Code and GitHub Copilot
pip install 'rishi[llama]'       # llama.cpp, for GGUF model files
pip install 'rishi[mlx]'         # MLX, on Apple Silicon Macs
pip install 'rishi[mlx-vlm]'     # MLX vision and audio models
pip install 'rishi[ollama]'      # Ollama; on Linux before Python 3.14 this adds zstandard to unpack Ollama
pip install 'rishi[all]'         # every extra above
```

Extras combine, as in `rishi[llama,mlx]`. A backend module is imported only when a chat first needs it, so an extra you never installed costs nothing until you ask for that backend. The hosted backends use their usual credentials: an API key in the environment for hosted APIs, a signed-in `claude` command for Claude Code, and a GitHub sign-in for Copilot.

## Chat

`Chat(model)` opens one conversation. The model name decides the backend. rishi recognises a `.litertlm` build, a `-GGUF` repo, an `mlx-community` repo, an Ollama `hf.co/` id, or a hosted name such as `gpt-5.1`. A prefix such as `claude/` or `copilot/` names the backend outright. With no model at all you get LiteRT-LM and its default model, Gemma 4 E2B. The [core](core.html) page lists the exact rules.

Calling the chat with a prompt sends one turn and returns the reply, a dict Urai calls a `Resp`. Each call appends the prompt and the reply to `chat.hist`, so the second question below is asked with the first exchange in view. `resp_text` pulls the answer text out of a reply, and `close()` frees the model’s memory when you are done.

``` python
chat = Chat(gemma4_e2b)
print(resp_text(chat('Give one fact about lobsters.')))
print(resp_text(chat('Give one more.')))
chat.close()
```

    Lobsters are crustaceans, which means they have a hard exoskeleton and eight legs.
    Lobsters are known for their ability to change their color and texture to blend in with their surroundings, a behavior called camouflage.

## Streaming

A long reply is easier to follow as it arrives. Pass `stream=True` and the call returns an iterator of markdown strings instead of one reply. Urai renders every backend’s events into the same markdown, so thinking and tool calls look alike whichever backend produced them. Pass `stream='raw'` if you want the chunk dicts that markdown is built from.

``` python
chat = Chat(gemma4_e2b)
for markdown in chat('Write a two-line poem about rain.', stream=True):
    print(markdown, end='', flush=True)
chat.close()
```

    Soft drops tap on the pane,
    Washing the world clean again.

## Images

To ask about a picture, pass a list instead of a string, with the prompt text and the image side by side. The image can be a `Path`, raw bytes or a `PIL.Image`, and every backend that takes images accepts the same list. The model has to be one that can see. Gemma 4 on LiteRT-LM can, and each backend’s page names others.

``` python
chat = Chat(gemma4_e2b)
image = repo_root()/'nbs/images.jpeg'
display(Image.open(image))
print(resp_text(chat(['Describe this image in one sentence.', image])))
chat.close()
```

![](index_files/figure-commonmark/cell-4-output-1.png)

    A medium-sized, reddish-brown German Shepherd is looking attentively off to the side with its mouth slightly open.

## Audio

Audio goes in the same way, as a file path or bytes next to the prompt. The model needs an audio encoder. Gemma 4 on LiteRT-LM has one, and so do some llama.cpp and MLX models. Claude Code, Ollama and text-only models raise an error instead of dropping the clip.

``` python
chat = Chat(gemma4_e2b)
audio = repo_root()/'nbs/speech.wav'
print(resp_text(chat(['Transcribe this clip.', audio])))
chat.close()
```

    Dancing in the masquerade, idol truth in plain sight jaded, pop, roll, click, dot, who will I be today or not? But such a tide as moving seems asleep, too full for sound and foam, when that drew from out the boundless deep turns again home, twilight and evening bell and after that.

## One conversation across seven backends

Because every backend keeps the same history format, a conversation can move between them. The example below gives a code word to Gemma on LiteRT-LM, then passes the history along a chain: Qwen3 on Ollama, on MLX and on llama.cpp, GPT-4.1 through the hosted API, Claude Haiku through Claude Code, and GPT-4.1 through Copilot. Each leg opens a new `Chat` with `messages=history`, asks for the code word, and hands the longer history to the next leg. Every leg answers “amber”, which it could only know from the history it was given.

Running it downloads four local models, and the last three legs need their credentials.

``` python
legs = [
    ('LiteRT', gemma4_e2b, {}),
    ('Ollama', f'ollama/{ollama_qwen}', {'think': True}),
    ('MLX', mlx_qwen, {}),
    ('llama.cpp', llama_qwen, {'n_ctx': 4096}),
    ('cloud', 'gpt-4.1', {}),
    ('Claude', 'claude/haiku', {}),
    ('Copilot', 'copilot/gpt-4.1', {}),
]

history = []
for i, (name, model, options) in enumerate(legs):
    chat = Chat(model, messages=history, **options)
    prompt = (
        'The code word is amber. Start a travel log with one short sentence.'
        if i == 0 else
        f'What is the code word? Add one short sentence saying {name} received this conversation.'
    )
    reply = chat(prompt)
    print(f'{name}: {resp_text(reply)}')
    history = list(chat.hist)
    chat.close()

print([m['role'] for m in history])
```

    LiteRT: Amber awaits.
    Ollama: The code word is **amber**. Ollama received this conversation.

    Fetching 9 files:   0%|                                                                                                                 | 0/9 [00:00<?, ?it/s]Fetching 9 files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 9/9 [00:00<00:00, 1912.59it/s]

    MLX: The code word is **amber**. MLX received this conversation.

    llama_context: n_ctx_seq (4096) < n_ctx_train (40960) -- the full capacity of the model will not be utilized

    llama.cpp: The code word is **amber**. llama.cpp received this conversation.
    cloud: The code word is amber. Cloud received this conversation.
    Claude: The code word is amber. Claude received this conversation.
    Copilot: The code word is amber. Copilot received this conversation.
    ['user', 'assistant', 'user', 'assistant', 'user', 'assistant', 'user', 'assistant', 'user', 'assistant', 'user', 'assistant', 'user', 'assistant']

## Backend guides

Each backend has a page for what is particular to it.

| notebook | what it covers |
|----|----|
| [`00_core.ipynb`](core.html) | how backends are registered and chosen, old names kept for compatibility, installing the agent skill |
| [`01_llama.ipynb`](llama.html) | choosing a GGUF file, GPU offload, KV cache reuse, image and audio input |
| [`02_litert.ipynb`](litert.html) | `.litertlm` builds, GPU or CPU, the engine’s own tool loop, [`bench()`](https://vedicreader.github.io/rishi/litert.html#bench) |
| [`03_mlx.ipynb`](mlx.html) | MLX text and vision models, the prompt cache, speculative decoding |
| [`04_remote.ipynb`](remote.html) | hosted providers, generated images and provider-side tools |
| [`06_claude.ipynb`](claude.html) | Claude Code sessions, transcripts, tools on locked-down machines |
| [`07_copilot.ipynb`](copilot.html) | signing in, Copilot’s tokens and headers, listing models |
| [`08_ollama.ipynb`](ollama.html) | installing and running the daemon, thinking levels, model capabilities |
