Metadata-Version: 2.5
Name: rishi
Version: 0.1.63
Summary: fast local llm driver with tools
Project-URL: Repository, https://github.com/vedicreader/rishi
Project-URL: Documentation, https://vedicreader.github.io/rishi/
Author-email: Karthik <karthik.rajgopal@hotmail.com>
License: Apache-2.0
License-File: LICENSE
Keywords: nbdev
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Requires-Python: >=3.12
Requires-Dist: aidialog>=0.0.10
Requires-Dist: claude-agent-sdk>=0.2
Requires-Dist: fastcore>=2.0.0
Requires-Dist: httpx
Requires-Dist: huggingface-hub>=1.23.0
Requires-Dist: liteparse<2.13
Requires-Dist: litert-lm-api>=0.14.0
Requires-Dist: llmsurgery>=0.0.13
Requires-Dist: python-fastllm>=0.0.45
Requires-Dist: safepyrun>=0.2.3
Requires-Dist: truststore>=0.10
Requires-Dist: uraiyadal==0.0.20
Provides-Extra: all
Requires-Dist: llama-cpp-python==0.3.30; extra == 'all'
Requires-Dist: mlx-lm>=0.31.3; (sys_platform == 'darwin' and platform_machine == 'arm64') and extra == 'all'
Requires-Dist: mlx-vlm>=0.6.8; (sys_platform == 'darwin' and platform_machine == 'arm64') and extra == 'all'
Requires-Dist: numpy; extra == 'all'
Requires-Dist: soundfile>=0.14.0; extra == 'all'
Requires-Dist: zstandard>=0.22; (sys_platform == 'linux' and python_version < '3.14') and extra == 'all'
Provides-Extra: dev
Requires-Dist: dhrishti>=0.0.5; extra == 'dev'
Requires-Dist: diskcache>=5.6.3; extra == 'dev'
Requires-Dist: ipykernel>=7.3.0; extra == 'dev'
Requires-Dist: koshas>=0.0.31; extra == 'dev'
Requires-Dist: llama-cpp-python==0.3.30; extra == 'dev'
Requires-Dist: mlx-lm>=0.31.3; (sys_platform == 'darwin' and platform_machine == 'arm64') and extra == 'dev'
Requires-Dist: mlx-vlm>=0.6.8; (sys_platform == 'darwin' and platform_machine == 'arm64') and extra == 'dev'
Requires-Dist: nbdev>=3.1.0; extra == 'dev'
Requires-Dist: notebook>=7.6.0; extra == 'dev'
Requires-Dist: numpy; extra == 'dev'
Requires-Dist: pyskills>=0.0.21; extra == 'dev'
Requires-Dist: soundfile>=0.14.0; extra == 'dev'
Requires-Dist: zstandard>=0.22; (sys_platform == 'linux' and python_version < '3.14') and extra == 'dev'
Provides-Extra: llama
Requires-Dist: llama-cpp-python==0.3.30; extra == 'llama'
Requires-Dist: numpy; extra == 'llama'
Requires-Dist: soundfile>=0.14.0; extra == 'llama'
Provides-Extra: mlx
Requires-Dist: mlx-lm>=0.31.3; (sys_platform == 'darwin' and platform_machine == 'arm64') and extra == 'mlx'
Provides-Extra: mlx-vlm
Requires-Dist: mlx-vlm>=0.6.8; (sys_platform == 'darwin' and platform_machine == 'arm64') and extra == 'mlx-vlm'
Provides-Extra: ollama
Requires-Dist: zstandard>=0.22; (sys_platform == 'linux' and python_version < '3.14') and extra == 'ollama'
Description-Content-Type: text/markdown

# rishi


<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->

rishi lets you use a language model from Python with the same calls, whether the model runs on your machine or on a remote server. Each way of running a model is a backend. Four backends run models locally: LiteRT-LM, llama.cpp, MLX and Ollama. Three backends send the conversation to another program or server: a hosted API such as OpenAI or Anthropic, your Claude Code installation, and GitHub Copilot. Each of these seven has its own message format and its own way to pass images and tool calls.

rishi uses one conversation format for all of them. The conversation is called the history. The history is a Python list of message dicts in the OpenAI format, such as `{'role': 'user', 'content': 'hi'}`. It is stored in `chat.hist`. Every backend reads and writes that list. You can start a conversation on a small local model, give the list to a hosted model, and the hosted model continues the conversation.

The parts that all backends share come from [Urai](https://vedicreader.github.io/urai), which is published on PyPI as `uraiyadal`. These are the `Chat` class, tool calling, streaming, options and recorded replies. rishi adds the seven backends and registers them with Urai. `from rishi import *` gives you the names from Urai and from rishi.

## Install

The base package includes LiteRT-LM and the three backends that send the conversation to another program or server. The other local backends need optional extras. Install only the extras you use:

``` sh
pip install rishi                # LiteRT-LM, hosted APIs, Claude Code and GitHub Copilot
pip install 'rishi[llama]'       # llama.cpp, for GGUF model files
pip install 'rishi[mlx]'         # MLX, on Apple Silicon Macs
pip install 'rishi[mlx-vlm]'     # MLX vision and audio models
pip install 'rishi[ollama]'      # Ollama; on Linux before Python 3.14 this also installs zstandard
pip install 'rishi[all]'         # every extra above
```

You can combine extras, as in `rishi[llama,mlx]`. rishi imports a backend module only when a chat first needs it. A missing extra causes no error until you ask for that backend. The hosted backends use their usual credentials. Hosted APIs need an API key in the environment. Claude Code needs a signed-in `claude` command. Copilot needs a GitHub sign-in.

## Chat

`Chat(model)` opens one conversation. The model name selects the backend. rishi recognizes a `.litertlm` build, a `-GGUF` repo, an `mlx-community` repo, an Ollama `hf.co/` id, and a hosted name such as `gpt-5.1`. A prefix such as `claude/` or `copilot/` selects the backend directly. With no model, `Chat` uses LiteRT-LM and its default model, Gemma 4 E2B. The [core](core.html) page lists the exact rules.

Call the chat with a prompt to send one turn. The call returns the reply, a dict that Urai calls a `Resp`. Each call appends the prompt and the reply to `chat.hist`. The second question below is therefore asked with the first exchange in the history. `resp_text` returns the answer text from a reply. `close()` frees the model memory.

``` python
chat = Chat(gemma4_e2b)
print(resp_text(chat('Give one fact about lobsters.')))
print(resp_text(chat('Give one more.')))
chat.close()
```

    Lobsters are crustaceans, which means they have a hard exoskeleton and eight legs.
    Lobsters are known for their ability to change their color and texture to blend in with their surroundings, a behavior called camouflage.

## Streaming

Pass `stream=True` to receive a reply as it is generated. The call then returns an iterator of markdown strings instead of one reply. Urai renders the events of every backend into the same markdown. Thinking and tool calls look the same on every backend. Pass `stream='raw'` to receive the chunk dicts that the markdown is built from.

``` python
chat = Chat(gemma4_e2b)
for markdown in chat('Write a two-line poem about rain.', stream=True):
    print(markdown, end='', flush=True)
chat.close()
```

    Soft drops tap on the pane,
    Washing the world clean again.

## Images

To ask about a picture, pass a list instead of a string. The list holds the prompt text and the image. The image can be a `Path`, raw bytes or a `PIL.Image`. Every backend that accepts images accepts the same list. The model must accept image input. Gemma 4 on LiteRT-LM does. The page of each backend names other models.

``` python
chat = Chat(gemma4_e2b)
image = repo_root()/'nbs/images.jpeg'
display(Image.open(image))
print(resp_text(chat(['Describe this image in one sentence.', image])))
chat.close()
```

![](index_files/figure-commonmark/cell-4-output-1.png)

    A medium-sized, reddish-brown German Shepherd is looking attentively off to the side with its mouth slightly open.

## Audio

Pass audio in the same list as the prompt, as a file path or as bytes. The model needs an audio encoder. Gemma 4 on LiteRT-LM has one. Some llama.cpp and MLX models have one. Claude Code, Ollama and text-only models raise an error. They do not drop the clip.

``` python
chat = Chat(gemma4_e2b)
audio = repo_root()/'nbs/speech.wav'
print(resp_text(chat(['Transcribe this clip.', audio])))
chat.close()
```

    Dancing in the masquerade, idol truth in plain sight jaded, pop, roll, click, dot, who will I be today or not? But such a tide as moving seems asleep, too full for sound and foam, when that drew from out the boundless deep turns again home, twilight and evening bell and after that.

## One conversation across seven backends

Every backend uses the same history format. A conversation can move between backends. The example below has seven legs. Each leg opens a new `Chat` with `messages=history`, sends one prompt, and passes the longer history to the next leg. The first leg uses Gemma on LiteRT-LM. Its prompt gives the code word “amber”. The next six legs use Qwen3 on Ollama, Qwen3 on MLX, Qwen3 on llama.cpp, GPT-4.1 through the hosted API, Claude Haiku through Claude Code, and GPT-4.1 through Copilot. Each of them is asked for the code word and answers “amber”. It can know this only from the history.

The example downloads four local models. The last three legs need their credentials.

``` python
legs = [
    ('LiteRT', gemma4_e2b, {}),
    ('Ollama', f'ollama/{ollama_qwen}', {'think': True}),
    ('MLX', mlx_qwen, {}),
    ('llama.cpp', llama_qwen, {'n_ctx': 4096}),
    ('cloud', 'gpt-4.1', {}),
    ('Claude', 'claude/haiku', {}),
    ('Copilot', 'copilot/gpt-4.1', {}),
]

history = []
for i, (name, model, options) in enumerate(legs):
    chat = Chat(model, messages=history, **options)
    prompt = (
        'The code word is amber. Start a travel log with one short sentence.'
        if i == 0 else
        f'What is the code word? Add one short sentence saying {name} received this conversation.'
    )
    reply = chat(prompt)
    print(f'{name}: {resp_text(reply)}')
    history = list(chat.hist)
    chat.close()

print([m['role'] for m in history])
```

    LiteRT: Amber awaits.
    Ollama: The code word is **amber**. Ollama received this conversation.

    Fetching 9 files:   0%|                                                                                                                 | 0/9 [00:00<?, ?it/s]Fetching 9 files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 9/9 [00:00<00:00, 1912.59it/s]

    MLX: The code word is **amber**. MLX received this conversation.

    llama_context: n_ctx_seq (4096) < n_ctx_train (40960) -- the full capacity of the model will not be utilized

    llama.cpp: The code word is **amber**. llama.cpp received this conversation.
    cloud: The code word is amber. Cloud received this conversation.
    Claude: The code word is amber. Claude received this conversation.
    Copilot: The code word is amber. Copilot received this conversation.
    ['user', 'assistant', 'user', 'assistant', 'user', 'assistant', 'user', 'assistant', 'user', 'assistant', 'user', 'assistant', 'user', 'assistant']

## Backend guides

Each backend has a page for the details that apply only to it.

| notebook | what it covers |
|----|----|
| [`00_core.ipynb`](core.html) | how backends are registered and chosen, old names kept for compatibility, installing the agent skill |
| [`01_llama.ipynb`](llama.html) | choosing a GGUF file, GPU offload, KV cache reuse, image and audio input |
| [`02_litert.ipynb`](litert.html) | `.litertlm` builds, GPU or CPU, the engine’s own tool loop, [`bench()`](https://vedicreader.github.io/rishi/litert.html#bench) |
| [`03_mlx.ipynb`](mlx.html) | MLX text and vision models, the prompt cache, speculative decoding |
| [`04_remote.ipynb`](remote.html) | hosted providers, generated images and provider-side tools |
| [`06_claude.ipynb`](claude.html) | Claude Code sessions, transcripts, tools on locked-down machines |
| [`07_copilot.ipynb`](copilot.html) | signing in, Copilot’s tokens and headers, listing models |
| [`08_ollama.ipynb`](ollama.html) | installing and running the daemon, thinking levels, model capabilities |
