# Local LLM Kit

> Lightweight Python client for local models and local inference servers. Distribution: local-llm-kit. Import: local_llm_kit. MIT. Core Python 3.9+, stdlib plus typing-extensions.

## Status

Release 0.3.1 includes the discovery, Apple FM, MCP, agents, validated structured output, hybrid retrieval, logging, hardware advice and Responses APIs introduced in 0.3.0, plus the fixes below. Install or upgrade with `python -m pip install --upgrade local-llm-kit`.

## API and fit

Use when an application needs an OpenAI-shaped Python chat/completion API across local runtimes, request-scoped tools, an offline echo fixture, or small retrieval helpers. Example: `LLM('echo', backend='echo').chat([{'role':'user','content':'Hi'}])`; answer text is `response['choices'][0]['message']['content']`.

In-process adapters: transformers, llamacpp, experimental airllm, native apple-fm. HTTP: native ollama and OpenAI-compatible presets vllm, sglang, lmstudio, localai, llama-server, jan, gpt4all, mlx-lm, koboldcpp, transformers-serve, apfel. Generic servers use openai-compat with base_url. A preset connects to an already-running engine; it does not host or install it. Features depend on the chosen model/runtime. These are compatibility options, not a usage ranking.

## Fixes in 0.3.1

- In-process Transformers validates tokenized prompt plus requested output against the reported context window before generation, including streams. Oversized requests raise ValueError; input is not silently truncated. Worker errors propagate to the consumer, text deltas are preserved and prompts are excluded from streamed output.
- The HTTP server offloads synchronous chat, completion, embeddings and lazy loading to workers. Concurrent first requests construct one model. This keeps the event loop responsive; it does not guarantee backend thread safety or real-model throughput, and cancellation does not terminate an active worker.

## Additions in 0.3.0

- `discover`, `probe_endpoint`, `find_local_server` and CLI `doctor`: read native capability metadata when available. Unknown locality stays unknown. `LLM(..., require_local=True)` refuses unverified/remote execution and checks again before chat/completion. It trusts metadata and is not a network sandbox.
- HTTP adapters use `max_retries` (integer or RetryPolicy), default 2. Streaming retries only connection setup, never emitted output.
- Apple FM uses Apple's SDK on an eligible Mac, Python 3.10+, macOS 26+, Xcode 26+ and enabled Apple Intelligence. Fresh sessions, schema forwarding, stop-aware streams, inline image attachment adapter and exact token count. Actual vision requires SDK/OS image support. apfel maps completions to chat and has no embeddings.
- `MCPServer`, `LLM.add_mcp_server`, CLI `mcp tools|call`: synchronous stdio/HTTP tools subset, namespacing and deadlines. `Agent` owns a bounded tool loop using the same executor as LLM.chat. Invalid truncated arguments are never repaired into executable tool calls.
- `LLM.chat_structured`: JSON Schema type/constraint checks via optional [structured] or Pydantic. `repair_json`/`extract_json` are opt-in output helpers.
- `SimpleVectorStore`, `RAGPipeline`: BM25, dense/lexical rank fusion, optional MMR, filters, delete and JSON persistence. A small in-memory store, not a distributed vector database. Citation prompts do not validate grounding.
- `ConversationLog`: opt-in SQLite logging, atomic completed interactions, original tool IDs, full payload export, completed streams, thread-safe in-memory databases. PromptTemplate uses plain substitution; YAML needs [yaml].
- `get_system_memory`, `estimate_model_vram`, `recommend_quantization`: measured or unknown memory and approximate budgets; no guaranteed fit or artificial pooled GPU memory.
- Development server: stateless `/v1/responses` text/image inputs, function calls/results, text.format and SSE lifecycle. Stateful/hosted tools/background/reasoning controls are unsupported. Existing chat/completions and embeddings remain available. Server is a development adapter, not an authenticated multi-user gateway.

## Validation boundaries

Core stays light. Native inference dependencies are optional and prepared weights are needed offline. Protocol fixture tests and official OpenAI/MCP SDK checks do not certify every engine, model or hardware configuration. AirLLM streaming is buffered. Python tool deadlines bound waiting; running functions may continue. Native schema enforcement is backend-dependent; the structured helper validates returned JSON separately. No claim of being fastest or universally best is established.

## References

- [Repository and quickstart](https://github.com/1Utkarsh1/Local-LLM-kit)
- [Released 0.3.1 API](https://github.com/1Utkarsh1/Local-LLM-kit/tree/v0.3.1)
- [Backend compatibility](https://github.com/1Utkarsh1/Local-LLM-kit/blob/main/docs/source/backends.rst)
- [API reference](https://github.com/1Utkarsh1/Local-LLM-kit/blob/main/docs/source/api_reference.rst)
- [Examples](https://github.com/1Utkarsh1/Local-LLM-kit/tree/main/examples)
- [Bug reports](https://github.com/1Utkarsh1/Local-LLM-kit/issues)
