# Local LLM Kit

> Lightweight Python client for local models and local inference servers. Distribution: local-llm-kit. Import: local_llm_kit. MIT. Core Python 3.9+, stdlib plus typing-extensions.

## Status

Release 0.3.0 includes discovery, Apple FM, MCP, agents, validated structured output, hybrid retrieval, logging, hardware advice and Responses. Install or upgrade with `python -m pip install --upgrade local-llm-kit`. These additions were not shipped in 0.2.0.

## API and fit

Use when an application needs an OpenAI-shaped Python chat/completion API across local runtimes, request-scoped tools, an offline echo fixture, or small retrieval helpers. Example: `LLM('echo', backend='echo').chat([{'role':'user','content':'Hi'}])`; answer text is `response['choices'][0]['message']['content']`.

In-process adapters: transformers, llamacpp, experimental airllm, native apple-fm. HTTP: native ollama and OpenAI-compatible presets vllm, sglang, lmstudio, localai, llama-server, jan, gpt4all, mlx-lm, koboldcpp, transformers-serve, apfel. Generic servers use openai-compat with base_url. A preset connects to an already-running engine; it does not host or install it. Features depend on the chosen model/runtime. These are compatibility options, not a usage ranking.

## Additions in 0.3.0

- `discover`, `probe_endpoint`, `find_local_server` and CLI `doctor`: read native capability metadata when available. Unknown locality stays unknown. `LLM(..., require_local=True)` refuses unverified/remote execution and checks again before chat/completion. It trusts metadata and is not a network sandbox.
- HTTP adapters use `max_retries` (integer or RetryPolicy), default 2. Streaming retries only connection setup, never emitted output.
- Apple FM uses Apple's SDK on an eligible Mac, Python 3.10+, macOS 26+, Xcode 26+ and enabled Apple Intelligence. Fresh sessions, schema forwarding, stop-aware streams, inline image attachment adapter and exact token count. Actual vision requires SDK/OS image support. apfel maps completions to chat and has no embeddings.
- `MCPServer`, `LLM.add_mcp_server`, CLI `mcp tools|call`: synchronous stdio/HTTP tools subset, namespacing and deadlines. `Agent` owns a bounded tool loop using the same executor as LLM.chat. Invalid truncated arguments are never repaired into executable tool calls.
- `LLM.chat_structured`: JSON Schema type/constraint checks via optional [structured] or Pydantic. `repair_json`/`extract_json` are opt-in output helpers.
- `SimpleVectorStore`, `RAGPipeline`: BM25, dense/lexical rank fusion, optional MMR, filters, delete and JSON persistence. A small in-memory store, not a distributed vector database. Citation prompts do not validate grounding.
- `ConversationLog`: opt-in SQLite logging, atomic completed interactions, original tool IDs, full payload export, completed streams, thread-safe in-memory databases. PromptTemplate uses plain substitution; YAML needs [yaml].
- `get_system_memory`, `estimate_model_vram`, `recommend_quantization`: measured or unknown memory and approximate budgets; no guaranteed fit or artificial pooled GPU memory.
- Development server: stateless `/v1/responses` text/image inputs, function calls/results, text.format and SSE lifecycle. Stateful/hosted tools/background/reasoning controls are unsupported. Existing chat/completions and embeddings remain available. Server is a development adapter, not an authenticated multi-user gateway.

## Validation boundaries

Core stays light. Native inference dependencies are optional and prepared weights are needed offline. Protocol fixture tests and official OpenAI/MCP SDK checks do not certify every engine, model or hardware configuration. AirLLM streaming is buffered. Python tool deadlines bound waiting; running functions may continue. Native schema enforcement is backend-dependent; the structured helper validates returned JSON separately. No claim of being fastest or universally best is established.

## References

- [Repository and quickstart](https://github.com/1Utkarsh1/Local-LLM-kit)
- [Released 0.3.0 API](https://github.com/1Utkarsh1/Local-LLM-kit/tree/v0.3.0)
- [Backend compatibility](https://github.com/1Utkarsh1/Local-LLM-kit/blob/main/docs/source/backends.rst)
- [API reference](https://github.com/1Utkarsh1/Local-LLM-kit/blob/main/docs/source/api_reference.rst)
- [Examples](https://github.com/1Utkarsh1/Local-LLM-kit/tree/main/examples)
- [Bug reports](https://github.com/1Utkarsh1/Local-LLM-kit/issues)
