Metadata-Version: 2.4
Name: pylolm
Version: 0.1.5
Summary: LoLm: run GGUF language models on-device on Android from Python (llama.cpp bindings for Kivy, Flet and Termux).
Author-email: AuraWay <auraway.dev@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/aura-way/pylolm
Project-URL: Issues, https://github.com/aura-way/pylolm/issues
Project-URL: Changelog, https://github.com/aura-way/pylolm/blob/main/CHANGELOG.md
Keywords: llm,llama.cpp,gguf,android,termux,kivy,flet,on-device,local-llm
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: Android
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: C
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Typing :: Typed
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: jinja2>=3.0
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: twine; extra == "dev"
Dynamic: license-file

# pylolm (LoLm)

**Local LLMs on Android, from Python.** An Android-friendly take on `llama-cpp-python`, built on
[llama.cpp](https://github.com/ggml-org/llama.cpp) and designed for **Kivy**, **Flet** and **Termux**.

- Tiny footprint: a thin `ctypes` layer over a small C shim; the only Python dependency is `jinja2`.
- `llama-cpp-python`-style API: `Llama(...)`, `create_chat_completion(...)`, `create_completion(...)`.
- Streaming generators, stop strings, a `stop()` button for your UI.
- Multi-turn chat that reuses the KV cache, so long conversations stay fast.
- Aimed at small GGUF models (0.5B-2B) on phones.

> **Status: alpha (0.1.0).** Text generation only. TTS / STT / OCR are on the roadmap.

## Install

```bash
pip install pylolm
```

That is all. It brings everything with it:

- **Chat-template support** (`jinja2`) is installed as a normal dependency.
- **The native libraries** come inside the Android arm64 wheel, which pip selects automatically
  on Android (including Termux with Python 3.13+).
- If your Python ends up with the library-free wheel instead (older Termux Pythons, building from
  source), pylolm downloads the prebuilt libraries **once, on first use** (checksum-verified, from
  the matching GitHub Release) and tells you it is doing so. This never happens inside an app
  (APK), and you can switch it off with `PYLOLM_AUTO_DOWNLOAD=0`.

Check the result any time with `pylolm doctor`.

Other ways to get the libraries: `pylolm get-libs` (download now), or compile for your exact
device (slow, ~30 min):

```bash
pkg install cmake clang git
pylolm build-native
```

Libraries are searched in `$PYLOLM_LIB_DIR`, `pylolm/lib/`, `site-packages/opt/lib/`, the app's
native-library folder, then `~/.pylolm/lib/`.

## Quick start

```python
from pylolm import Llama

llm = Llama("/sdcard/models/Qwen2.5-0.5B-Instruct-Q4_K_M.gguf", n_ctx=2048)

messages = [{"role": "user", "content": "Explain GGUF in two sentences."}]

# whole answer
out = llm.create_chat_completion(messages, max_tokens=200)
print(out["choices"][0]["message"]["content"])

# streaming
for chunk in llm.create_chat_completion(messages, max_tokens=200, stream=True):
    piece = chunk["choices"][0]["delta"].get("content")
    if piece:
        print(piece, end="", flush=True)
```

Try it from the terminal: `pylolm chat /path/to/model.gguf`. Use `pylolm info /path/to/model.gguf`
to see a model's details and chat template, and `--chat-template NAME` if its template is not
recognised.

## API

| Call | Notes |
|---|---|
| `Llama(model_path, n_ctx=2048, n_threads=None, verbose=False)` | `n_threads` defaults to `min(4, cpu_count)` |
| `create_chat_completion(messages, max_tokens=256, temperature=0.7, top_p=0.95, top_k=40, min_p=0.05, repeat_penalty=1.0, seed=None, stop=None, stream=False, chat_template=None)` | OpenAI-style dict, or chunk iterator with `stream=True` |
| `create_completion(prompt, ...)` / `llm(prompt, ...)` | raw text completion |
| `stop()` | stop the running generation from another thread |
| `reset()` | clear the KV cache (forget the conversation) |
| `count_tokens(text)` | tokens used by a string |
| `format_chat(messages, chat_template=None)` | the prompt text for this model (debugging) |
| `chat_template` (property) | the model's own Jinja chat template, or `None` |
| `close()` / `with Llama(...) as llm:` | free the model |

Errors: `LibraryNotFoundError`, `ModelLoadError`, `ContextOverflowError`, `GenerationError`
(all subclass `PyLoLmError`).

`usage` in a non-streaming result includes `cached_prompt_tokens`: how many prompt tokens were
reused from the previous turn.

**Chat templates.** Every chat model expects its own prompt format. pylolm handles this for you:
it first tries llama.cpp's built-in templates (ChatML/Qwen, Llama 3, Gemma 2/3, ...), and if the
model's template is not one of those (typical for brand-new models) it renders the template stored
inside the GGUF file itself, in Python (using `jinja2`, which is installed with pylolm).

Use the `chat_template=` argument to override: `"jinja"` (always use the model's own template),
a built-in name such as `"chatml"` or `"gemma"`, or your own Jinja string.
`llm.format_chat(messages)` shows the exact prompt text that will be sent.

Debugging a model: `pylolm info model.gguf` (no native libraries needed) prints its metadata and chat
template, and `pylolm template model.gguf` loads it and shows how prompts get formatted.

## Using it in apps

**UI threads.** Generation blocks, so run it off the UI thread. A streaming generator is easiest:

```python
import threading

def worker():
    for chunk in llm.create_chat_completion(messages, stream=True):
        text = chunk["choices"][0]["delta"].get("content")
        if text:
            Clock.schedule_once(lambda dt, t=text: append_to_label(t))  # Kivy
            # page.run_thread / page.update(...) for Flet

threading.Thread(target=worker, daemon=True).start()
```

Call `llm.stop()` from your Stop button; breaking out of the loop also stops generation.

**Flet.** `flet build apk` installs your dependencies for Android from PyPI, so the
`android_24_arm64_v8a` wheel is picked automatically. Its libraries sit in `opt/lib/`, which is
the folder Flet's packager copies into the app's native-library directory. Add this to your
app's `pyproject.toml`:

```toml
[project]
dependencies = ["flet", "pylolm"]

[tool.flet.android]
legacy_packaging = true      # extract the .so files so llama.cpp can find its CPU backends
target_arch = ["arm64-v8a"]  # pylolm ships arm64 libraries only
```

A complete streaming chat app is in `examples/flet_chat`. Check with the **Diagnostics** button
(or `pylolm.doctor`) that the native libraries loaded.

**Kivy / python-for-android.** p4a does not consume Android wheels yet. For now, bundle the
`.so` files with your app and point `PYLOLM_LIB_DIR` at them before importing pylolm:

```python
import os
os.environ["PYLOLM_LIB_DIR"] = os.path.join(os.path.dirname(__file__), "pylolm_libs")
import pylolm
```

A proper p4a recipe is on the roadmap.

## Building from source

```bash
git clone https://github.com/aura-way/pylolm && cd pylolm
pip install -e ".[dev]"
python -m pytest                                   # pure-Python tests, no model needed
PYLOLM_MODEL=/path/model.gguf python -m pytest     # + integration tests
```

Native libraries: `src/pylolm/native/build_termux.sh` (on a phone) or `build_android_ndk.sh`
(cross-compile). The llama.cpp version is pinned in `src/pylolm/native/LLAMA_CPP_REF`.

## Releasing to PyPI

1. Set `LLAMA_CPP_REF` to a llama.cpp tag/commit you have tested.
2. Bump `version` in `pyproject.toml` and add a `CHANGELOG.md` entry.
3. On PyPI, add a [trusted publisher](https://docs.pypi.org/trusted-publishers/) for this repo
   (workflow `release.yml`, environment `pypi`).
4. `git tag v0.1.0 && git push --tags`. CI builds the sdist, a pure wheel, and the Android arm64
   wheel, publishes them to PyPI, and attaches `pylolm-libs-android-arm64-v8a.tar.gz` (+ `.sha256`)
   to the GitHub Release for `pylolm get-libs`. The tag must match the version in `pyproject.toml`.

## Roadmap

- p4a recipe for Kivy; verified Flet build
- whisper.cpp (speech-to-text) module
- TTS and OCR modules
- GPU backends (Vulkan/OpenCL) as an opt-in

## License

MIT. llama.cpp is MIT-licensed too; model licenses are your responsibility.
