HFL — Complete Architecture

1. Overview and Purpose

HFL (HuggingFace Local) is a CLI tool and API server that allows downloading, managing, and running artificial intelligence models from HuggingFace Hub directly on the user's local machine. Its design aims to be a drop-in replacement for Ollama, but connected to the HuggingFace ecosystem with more than 500,000 models.

2. Technology Stack

3. Project File Structure

hfl/
├── pyproject.toml           — Package definition, deps, scripts, tools
├── hfl.spec                 — PyInstaller spec for standalone executable
├── LICENSE                  — Apache-2.0
├── PRIVACY.md               — Privacy policy
├── NOTICE-EU-AI-ACT.md     — EU AI Act compliance
├── DISCLAIMER.md            — Liability disclaimer
├── README.md                — Main documentation
├── README.es.md             — Main documentation (Spanish)
├── LICENSE-FAQ.md            — License notice
├── CONTRIBUTING.md           — Contributing guide
├── CODE_OF_CONDUCT.md        — Code of Conduct (Contributor Covenant 2.1)
├── CHANGELOG.md              — Changelog (Keep a Changelog format)
├── SECURITY.md               — Security policy
│
├── src/hfl/                 — MAIN SOURCE CODE
│   ├── __init__.py          — Package version (0.15.0)
│   ├── config.py            — HFLConfig dataclass — global configuration
│   ├── exceptions.py        — Complete exception hierarchy
│   ├── events.py            — EventBus for internal pub/sub
│   ├── metrics.py           — Performance metrics
│   ├── plugins.py           — Plugin system with entry_points
│   ├── security.py          — Security sanitization and validation
│   ├── validators.py        — Common data validators
│   ├── logging_config.py    — Centralized logging configuration
│   │
│   ├── core/                — SYSTEM CORE
│   │   ├── __init__.py
│   │   ├── container.py     — DI container with thread-safe Singleton
│   │   ├── observability_setup.py — Observability event listeners setup
│   │   └── tracing.py       — Request tracing with IDs
│   │
│   ├── cli/                 — COMMAND LINE INTERFACE
│   │   ├── __init__.py
│   │   ├── main.py          — 12 commands: pull, run, serve, list, search, rm, inspect, alias, login, logout, version, compliance-report
│   │   └── commands/        — Modularized commands
│   │       └── _utils.py    — Shared CLI utilities
│   │
│   ├── api/                 — REST API SERVER
│   │   ├── __init__.py
│   │   ├── server.py        — FastAPI app, CORS, disclaimer, lifespan
│   │   ├── state.py         — ServerState with async locks for LLM/TTS
│   │   ├── streaming.py     — SSE streaming helpers
│   │   ├── model_loader.py  — Dynamic model loading
│   │   ├── helpers.py       — ensure_llm_loaded, ensure_tts_loaded
│   │   ├── errors.py        — Centralized HTTP error handling
│   │   ├── middleware.py    — Privacy-safe logging
│   │   ├── rate_limit.py    — Rate limiting by IP/token
│   │   ├── routes_openai.py — /v1/chat/completions, /v1/completions, /v1/models
│   │   ├── routes_native.py — /api/generate, /api/chat, /api/tags (Ollama-compatible)
│   │   ├── routes_tts.py    — /v1/audio/speech, /api/tts (TTS endpoints)
│   │   ├── routes_health.py — /health, /ready, /live (healthchecks)
│   │   ├── routes_metrics.py — /metrics (Prometheus + JSON)
│   │   ├── exception_handlers.py — Global HFLError exception handling
│   │   └── timeout.py       — @with_timeout decorator + configurable timeout
│   │
│   ├── engine/              — INFERENCE ENGINES
│   │   ├── __init__.py
│   │   ├── base.py          — InferenceEngine + AudioEngine ABCs + dataclasses
│   │   ├── selector.py      — Automatic backend selection for LLM/TTS
│   │   ├── llama_cpp.py     — LlamaCppEngine (GGUF, CPU/GPU)
│   │   ├── transformers_engine.py — TransformersEngine (safetensors, GPU)
│   │   ├── vllm_engine.py   — VLLMEngine (production GPU)
│   │   ├── bark_engine.py   — BarkEngine (TTS via transformers)
│   │   ├── coqui_engine.py  — CoquiEngine (TTS XTTS-v2)
│   │   ├── async_wrapper.py — Sync→async wrapper for engines
│   │   ├── model_pool.py    — Model pool with LRU eviction
│   │   ├── dependency_check.py — Optional dependency verification
│   │   ├── failover.py      — FailoverEngine (multi-engine retry with sticky routing)
│   │   ├── memory.py        — Real-time RAM/GPU memory tracking
│   │   ├── observability.py — Engine performance metrics
│   │   └── prompt_builder.py — Prompt formats + delimiter escaping
│   │
│   ├── hub/                 — HUGGINGFACE INTEGRATION
│   │   ├── __init__.py
│   │   ├── auth.py          — Authentication and tokens
│   │   ├── client.py        — HTTP client for HF Hub
│   │   ├── downloader.py    — Download with resume and rate limiting
│   │   ├── license_checker.py — License classification (5 levels)
│   │   └── resolver.py      — Intelligent model resolution
│   │
│   ├── models/              — DATA MODELS
│   │   ├── __init__.py
│   │   ├── manifest.py      — ModelManifest — complete metadata
│   │   ├── provenance.py    — ConversionRecord + ProvenanceLog
│   │   ├── registry.py      — ModelRegistry — local JSON inventory
│   │   └── backends/        — Storage backends (File + SQLite)
│   │
│   ├── converter/           — FORMAT CONVERSION
│   │   ├── __init__.py
│   │   ├── formats.py       — ModelFormat + ModelType enums
│   │   └── gguf_converter.py — GGUFConverter — conversion + quantization
│   │
│   ├── utils/               — CROSS-CUTTING UTILITIES
│   │   ├── __init__.py
│   │   ├── circuit_breaker.py — Circuit breaker for resilience
│   │   └── retry.py         — Retry with exponential backoff
│   │
│   └── i18n/                — INTERNATIONALIZATION
│       ├── __init__.py      — t(), get_language(), set_language()
│       └── locales/         — Translation files
│           ├── en.json      — English translations
│           └── es.json      — Spanish translations
│
├── docs/                    — DOCUMENTATION
│   ├── adr/                 — Architecture Decision Records
│   │   ├── 0001-singleton-pattern.md
│   │   ├── 0002-async-api-sync-engines.md
│   │   ├── 0003-gguf-default-format.md
│   │   ├── 0004-ollama-compatibility.md
│   │   ├── 0005-license-classification.md
│   │   └── 0006-rate-limiting-strategy.md
│   └── *.html               — Architecture documentation
│
└── tests/                   — TEST SUITE (~200 files, ~89% coverage)
    ├── conftest.py          — Shared fixtures
    ├── test_api*.py         — API tests (5 files)
    ├── test_cli*.py         — CLI tests (4 files)
    ├── test_engine*.py      — Engine tests (8 files)
    ├── test_hub*.py         — HF Hub tests (5 files)
    ├── test_i18n*.py        — i18n tests (4 files)
    └── test_*.py            — Unit and integration tests

├── .github/                 — GITHUB INFRASTRUCTURE
│   ├── workflows/
│   │   ├── ci.yml           — CI: lint + test + type-check (Python 3.10/3.11/3.12)
│   │   ├── pages.yml        — Deploy docs to GitHub Pages
│   │   ├── build-executables.yml — Cross-platform executable builds
│   │   ├── lint.yml         — Linting with ruff
│   │   ├── test.yml         — Tests with pytest
│   │   ├── security.yml     — Security audit
│   │   └── license-check.yml — License verification
│   ├── ISSUE_TEMPLATE/
│   │   ├── bug_report.yml   — Bug report template
│   │   └── feature_request.yml — Feature request template
│   └── PULL_REQUEST_TEMPLATE.md — PR template with compliance checklist

4. General Architecture Diagram

5. Module config — Central Configuration

New Environment Knobs (since v0.1.0)

Resolution rule for every row: HFL-specific name → HFL alias → Ollama-equivalent → default. So a host that already exports Ollama vars works as a drop-in replacement. An invalid value is logged once and ignored — boot never fails on operator misconfiguration. Full reference: docs/env-vars.md.

6. Module cli — Command Line Interface

CLI Helper Functions

_format_size() converts bytes to readable format. _get_key() reads a key without Enter (raw terminal). _extract_params_from_name() extracts parameters from name (regex: "70b", "7b", "1.5b"). _estimate_model_size() estimates disk size based on parameters and quantization. _display_model_row() renders a search result row. _get_params_value() extracts numeric value for filtering.

New Commands (since v0.1.0)

File grew to ~2,350 lines in src/hfl/cli/main.py. Two families were added: drop-in Ollama-parity commands and the Hub-native discovery/ops commands. Lazy imports keep startup cheap; most of these commands run an asyncio.run() coroutine internally.

The serve command also gained --tray/--gui (system tray, see the tray section), --log-level, --json-logs and --ctx.

7. Module hub — HuggingFace Integration

7. Hub Discovery, Recommendation, Smart Pull & Push

Beyond plain pull, the Hub layer now exposes the full HuggingFace catalogue (1.5M+ models with structured metadata) as a queryable, hardware-aware surface. Four read/write endpoints sit on top of pure, network-free planning modules so the heuristics are unit-testable without the network.

Hardware probing & VRAM budgets

hfl/hub/hw_profile.py builds a cheap, metadata-only HardwareProfile (no GPU work). Memory budgets used by both recommend and smart_pull:

hfl/hub/quant_table.py turns (params_b, quantization) into a conservative FitEstimate = (weights + 4k-context KV cache + 1.0 GB overhead) × 1.2 safety. Bits-per-weight are tabulated per quant (q4_k_m→4.85, q8_0→8.5, mlx-4bit→4.5, …); layer counts and KV hidden dims are bucketed by size, with Qwen 3, Gemma 2 and Mixtral-8x7B (effective) added to close the original Llama-only gap.

Smart-pull candidate order

_candidate_repos() generates, most-specific-first:

The MLX legs only appear when has_mlx is true. The quant ladder is ["mlx-8bit","mlx-4bit","q5_k_m","q4_k_m"] on Apple Silicon, else ["q5_k_m","q4_k_m","q4_0","q3_k_m"]; rejected probes are recorded in fallback_chain and surfaced verbatim. When nothing fits, build_smart_plan() raises ValueError (mapped to a 400 with a "try --max-vram-gb or a smaller repo" hint).

# POST /api/pull/smart — Apple Silicon, 16 GB unified
{"status": "planned", "target_repo_id": "mlx-community/Llama-3.1-8B-Instruct-4bit",
 "quantization": "mlx-4bit", "estimated_vram_gb": 6.3,
 "reason": "picked mlx-community/... @ mlx-4bit (6.3 GB / 11.2 GB budget)"}

Discovery cache & security

DiscoveryCache (HFL_HOME/cache/discovery.json) keys on the JSON-serialised query, TTL 300s, bounded to the 32 most-recent keys, with atomic temp-file writes and corrupt-file reset — it exists to dodge the Hub's unauthenticated 429s on chained CLI calls. On the push side, redact_secrets() strips anything matching hf_[A-Za-z0-9]{20,} from both logs and client-facing error events, and the upload is scoped to allow_patterns = exactly the planned files — critical because a blob-backed model (FROM sha256:<digest>) has its local_dir pointing at the shared blobs/ store, so an unscoped upload_folder would exfiltrate every other model's bytes.

8. Module models — Data and Registry

ModelManifest (manifest.py)

Dataclass that stores complete metadata for each downloaded model. It is the fundamental unit of information in the system.

ModelRegistry (registry.py)

Manages local inventory. Persists to ~/.hfl/models.json as JSON array. Operations: add() (avoids duplicates), get() (searches by name, alias, or repo_id), set_alias(), list_all() (sorted by date), remove().

ProvenanceLog (provenance.py)

Immutable conversion log in ~/.hfl/provenance.json. Each ConversionRecord documents: source (repo, format, revision), destination (format, path, quantization), tool used (llama.cpp + version), original license, and timestamps. Serves for legal traceability and compliance audit (R3).

8. Registry, Manifest, Provenance & Capabilities (current state)

The model layer hardened and widened considerably. The on-disk registry gained concurrency safety; the manifest absorbed Modelfile-derived and legal fields; and a dedicated module now computes Ollama-style capabilities.

Capability detection (Ollama-compatible)

New module hfl/models/capabilities.py. detect_capabilities() returns the list POST /api/show emits, which Ollama SDK clients (ollama-python, Open WebUI, LangChain) use to gate tool-calling, vision, embeddings and reasoning. Detection is permissive substring-matching over name + repo_id + architecture:

Order is deterministic: completion/embedding first, the rest alphabetical.

Provenance log

hfl/models/provenance.py records the source→conversion→result chain (~/.hfl/provenance.json) for the legal-audit safeguard: source repo/format/revision, target path, quantization, tool + version, original license and acceptance timestamp. log_conversion() is the convenience entrypoint, called from the GGUF converter after a successful quantize.

9. Module engine — Inference Engines

Automatic Selector (selector.py)

Decision logic in select_engine(model_path, backend):

All imports are lazy (_get_llama_cpp_engine(), etc.) to avoid requiring all dependencies to be installed.

MLX Backend — Apple Silicon (Phase 13)

mlx_engine.MLXEngine is a fourth InferenceEngine implementation that wraps Apple's mlx-lm and hits Metal directly. The module docstring claims a 3-10% edge on prompt processing and 15-25% on decode at fp16 versus llama-cpp's Metal path for Llama-family architectures, plus native q4/q5/q8 mixed precision with no GGUF conversion. The dependency lives behind the [mlx] extra.

mlx-lm exposes no per-call seed or stop-string support, so the engine fills both gaps itself: _maybe_seed() seeds mlx's global RNG (mx.random.seed) when cfg.seed >= 0 (ENG-10/11 parity with vLLM/diffusers), and _stop_strings() + _earliest_stop() truncate the completion at the first caller stop string. In generate_stream() the stop enforcement is a buffered char-level filter that holds back a tail (max_stop) that could begin a stop marker so output never runs past it. Token counts for the response envelope are recomputed via the tokenizer because mlx-lm returns only the completion text.

Backend Selection — MLX path & force-override

select_engine() in selector.py gained an Apple-Silicon branch and an operator-level override that the old flow doesn't show.

GGUF stays on llama-cpp regardless — MLX cannot ingest GGUF. _create_engine("mlx") wires the explicit --backend mlx request.

KV-Cache Snapshots — warm-start (snapshot.py)

Most servers re-process the system prompt / few-shot prefix on every new conversation. snapshot.py exposes llama-cpp-python's Llama.save_state() / load_state() as a server feature so an operator can capture the entire KV cache (processed tokens + cache tensors) and restore it for zero TTFT on the cached prefix, even across server restarts. It backs POST /api/snapshot/save, /load, GET /api/snapshot and DELETE /api/snapshot/{name}.

def save_snapshot(engine, *, name, model_name) -> SnapshotMeta: ...
def load_snapshot(engine, *, name, model_name) -> SnapshotMeta: ...

State is a pickle of save_state() plus a <name>.meta.json sidecar (model, tokens, created_at, bytes, version, mac) under HFL_HOME/snapshots/. Safety properties built into the module:

save_snapshot/load_snapshot resolve save_state/load_state either on the engine directly or on its inner _model; engines without that API raise (KV snapshots are llama-cpp-python-only).

Speculative Decoding (llama_cpp.py)

The llama-cpp backend can run a draft model to propose tokens the target verifies in a single pass. Selected at load time through the draft_model_path kwarg (surfaced from a Modelfile); GenerationConfig also carries a draft_model field. Two modes share the one kwarg:

_LlamaModelDraftAdapter bridges a plain Llama into the LlamaDraftModel protocol (a bare Llama returns a completion dict, not a token-ids array). It is deliberately incremental: _align_and_eval_suffix() finds the longest common prefix with the previously-processed ids and evaluates only the suffix, preserving the draft's KV cache across steps. A naive reset-per-call would pay the full prefill N times and make speculation 2-3× slower; on divergence (fresh request / rejected predictions) it resets and replays. The draft Llama is tracked as self._draft_model and freed in unload() so a subsequent load doesn't double up VRAM.

LoRA Hot-Swap & Multi-LoRA (lora.py)

lora.py applies / removes LoRA adapters on a running model without reloading base weights, backing POST /api/lora/apply, /remove, GET /api/lora and /api/lora/{model}. The public surface:

def apply_lora(engine, *, lora_path, scale=1.0, name=None) -> AdapterInfo
def remove_lora(engine, adapter_id) -> bool
def list_loras(engine=None) -> list[AdapterInfo]

remove_lora detaches from the live engine first and only drops the registry entry on success — if the engine refuses the unset, the entry stays tracked so the client's retry can succeed instead of leaving a ghost adapter mixed into every generation but unremovable. At load time, LlamaCppEngine.load() also accepts a lora_paths list but wires only the first into llama-cpp's single lora_path kwarg, logging that any extras are ignored (post-load stacking is the multi-LoRA route).

Flash Attention — architecture-aware gating

The doc lists flash_attn as a load parameter; the backend now resolves it through a three-level precedence in LlamaCppEngine.load(), because llama-cpp-python's flash-attn path is historically crash-prone on new architectures:

The resolved value is passed into the Llama constructor alongside the architecture-aware context cap (_ARCHITECTURE_CTX_CAP, e.g. Gemma 3/4 capped at 8192 to avoid pinning unified memory on Apple Silicon) and optional KV-cache quantisation (kv_cache_type → type_k/type_v q4_0/q8_0/f16).

Engine-Side Model Lifetime — keep-alive & eviction

HFL has no literal pin()/unpin() on the engine; a model's residency is governed by the ModelPool (model_pool.py) and keep-alive deadlines, so "pinning" is an emergent property rather than an API call. The pool keeps multiple loaded engines and evicts on three signals: LRU (capacity), idle timeout (default idle_timeout_seconds=3600), and memory pressure (estimates). A background loop (background_eviction_interval=60s) sweeps idle models via start_background_eviction().

Other Engines in the Package

Beyond the three text backends, engine/ ships audio, diffusion and embedding engines the original section never mentioned. They are loaded lazily behind their own extras.

The base module defines the parallel AudioEngine ABC (TTSConfig / AudioResult), and select_tts_engine() in selector.py auto-routes a TTS model to Bark or Coqui (name/config.json architecture sniffing, Bark as the transformers-pipeline default).

10. Module core — Concurrency, Dispatcher & Sessions

The hfl.core package is the cross-cutting infrastructure layer added after the v0.1.0 freeze. It holds the dependency-injection Container, request tracing, optional process sandboxing, the events→metrics bridge, and chat-session persistence. Its single most important companion is the InferenceDispatcher — the bounded-concurrency primitive that serialises every inference call against the shared, non-reentrant model.

Where it lives. The dispatcher class itself is defined in hfl/engine/dispatcher.py, but it is wired as a singleton inside the core container and always reached through hfl.core.get_dispatcher(). Treat it as part of the core concurrency story.

10.1 InferenceDispatcher — a bounded Bulkhead

InferenceDispatcher (hfl/engine/dispatcher.py) is async-aware and backed by an asyncio.Semaphore whose capacity tracks max_inflight exactly. All counters mutate under a single _counter_lock so fast-path and slow-path decisions observe one consistent view.

The factory build_default_dispatcher() reads these from hfl.config; the container's _create_dispatcher() wraps it in the lazy singleton.

The acquire lifecycle — fast path, slow path, rejection

The async context manager slot() drives a four-phase protocol. Under the counter lock it decides between the fast path (capacity now) and the slow path (must wait); a full wait queue is rejected before queueing, so the client gets an immediate 429 rather than blocking.

async with self._counter_lock:
    if self._in_flight < self._max_inflight:
        self._in_flight += 1          # fast path: reserve a slot
        fast_path = True
    else:
        if self._depth >= self._max_queued:
            self._rejected_full_total += 1
            raise QueueFullError(...)  # → HTTP 429 + Retry-After
        self._depth += 1            # slow path: enqueue and wait
        fast_path = False

Phase 2 acquires the semaphore outside the counter lock (so releasing callers never deadlock). Slow-path callers wrap the acquire in asyncio.wait_for(..., timeout=acquire_timeout); a timeout decrements depth, bumps rejected_timeout_total, and raises QueueTimeoutError. Cancellation safety is explicit: a cancel while waiting unwinds depth; a cancel after reserving the fast-path slot rolls back in_flight; a slot held during execution is always released in the finally of phase 4 (decrement in_flight, then sem.release() — chosen so a snapshot taken mid-release over-counts load rather than under-counting it).

snapshot() — the metrics surface

snapshot() returns an immutable DispatcherSnapshot (six ints: max_inflight, max_queued, in_flight, depth, accepted_total, rejected_full_total, rejected_timeout_total). It is read-only and cheap, so it is exported on every surface:

How routes consume it: run_dispatched() in api/helpers.py acquires a slot, runs the sync engine call on a thread (asyncio.to_thread), and applies the generation timeout. Critically it holds the slot until the worker thread actually exits — a sync engine call cannot be cancelled, so on a 504/client-cancel the dispatcher slot is kept (_release_when_worker_exits) to stop a queued request starting a second inference that would corrupt the model mid-generation. Streaming routes pre-acquire via acquire_stream_slot(), which returns either an entered context manager or a ready 429/503 JSONResponse.

10.2 exclusive() — the drain barrier for admin work

It guards three operations against the shared model:

10.3 Engine refcount pin/unpin — the hot-swap use-after-free guard

exclusive() only covers paths that hold a dispatcher slot. The WebSocket chat turn (routes_ws.py) reads the engine directly and does not hold a slot, so a concurrent swap could unload() the model mid-stream — a use-after-free of the non-reentrant model. ServerState closes that gap with a reference count.

The swap logic in set_llm_engine() performs an atomic retire-then-assign: it retires the old engine before assigning the new one, so an unload failure leaves the old engine in place (callers observe "replacement failed", not a half-updated state). Inside the exclusive() drain it checks _engine_inuse: if the displaced engine is still pinned it goes into _engine_retired and the unload is deferred to the last unpin_engine; otherwise it unloads immediately off-loop. The WS producer pins for the lifetime of the producer thread and unpins in its finally (which can outlive a client cancel, since the engine call cannot be preempted).

10.4 Container — dependency injection & singletons

core/container.py centralises global state. Singleton[T] is a thread-safe lazy holder (double-checked locking around a factory, with reset() for tests). The Container dataclass holds one Singleton per shared service and exposes convenience accessors:

from hfl.core import get_config, get_registry, get_state, get_dispatcher

reset_container() rebuilds everything from scratch — the standard test fixture for clean state.

10.5 Tracing, Sessions, Sandbox, Observability

11. Module converter — Format Conversion

formats.py — Format Detection

Enum ModelFormat: GGUF, SAFETENSORS, PYTORCH, UNKNOWN. The detect_format(path) function inspects extensions (.gguf, .safetensors, .pt/.pth/.bin) both in individual files and directories (rglob). find_model_file() locates the main model file.

gguf_converter.py — GGUF Conversion

Class GGUFConverter with two-step pipeline:

ensure_tools() auto-installs llama.cpp if not present: git clone → cmake build → pip install requirements. check_model_convertibility() validates that the model is convertible (rejects LoRA adapters, image models, models without config.json).

11. Architecture coverage, Modelfile parser & Go-template renderer (current state)

The converter cluster spans format/type detection (formats.py), the GGUF pipeline (gguf_converter.py), a full bidirectional Modelfile parser/renderer (modelfile_parser.py) and a minimal Go-template engine (go_template.py).

30+ LLM architectures + MLX detection

hfl/converter/formats.py recognises 33 causal-LM architectures in LLM_ARCHITECTURES (Llama, Mistral, Mixtral, Qwen2/Qwen2Moe, Gemma/Gemma2, Phi/Phi3, Falcon, Cohere/CommandR, DeepseekV2, InternLM/InternLM2, Yi, StarCoder2, CodeLlama, Olmo/Olmo2, Mamba, Jamba, Arctic, StableLm, OPT, Bloom, GPT2/NeoX/J, …), plus 16 other architecture families (TTS, STT, embeddings, vision-QA, seq2seq, depth, …) used by detect_model_type() to classify any HF config.json. is_mlx_quantized_repo() flags MLX-packed repos (mlx in the repo id, or a quantization object with group_size+bits in config.json) so they're routed to the MLX backend instead of llama.cpp's converter, which rejects MLX-packed graphs.

GGUF conversion pipeline

hfl/converter/gguf_converter.py runs safetensors → convert_hf_to_gguf.py (F16) → llama-quantize → final GGUF. check_model_convertibility() rejects LoRA adapters, diffusion/image, and audio/TTS configs up front. convert_with_cache() normalises the quant token before building the lock key (so q4_k_m and Q4_K_M can't race onto the same output file) and serialises duplicate conversions with a per-key lock; _gguf_variant_path() deliberately avoids with_suffix so version dots in repo names (Qwen2.5-7B) aren't truncated into colliding filenames. The F16 intermediate is resumable, and _check_conversion_environment() probes the host Python for a compatible transformers/huggingface_hub combo before launching the real conversion.

Modelfile parser — escapes, INCLUDE guards, round-trip

hfl/converter/modelfile_parser.py closes the loop for POST /api/create: parse_modelfile() handles FROM, PARAMETER, TEMPLATE, SYSTEM, ADAPTER, LICENSE, MESSAGE, REQUIRES, ENV, CAPABILITIES, DRAFT and INCLUDE; render_modelfile_document() serialises back with a tested round-trip invariant. Hardening:

• Escape decoding — _decode_quoted() interprets \n \t \r \" \\ inside single- and triple-quoted values; the encoder escapes backslashes (and embedded """) symmetrically so a Jinja template carrying a literal \n survives parse→render→parse unchanged.
• INCLUDE path-injection guard — _expand_includes() walks the base dir once into an allow-list dict; every downstream open() consumes a dictionary-indexed string, never raw user input. _is_safe_include_rel() refuses .., absolute paths, backslashes and NUL; cycles and a depth-16 cap are enforced.
• Hand-written INCLUDE matcher (no regex) so CodeQL's polynomial-ReDoS pass has nothing to flag.

Go-template renderer with recursion guard

hfl/converter/go_template.py implements the Go text/template subset real Modelfiles use — {{ .Field }}, nested paths, {{ range }}, {{ if }}/{{ else }}/{{ end }}, string literals, and {{- -}} whitespace trimming. render_go_template() is a convenience layer, never a gate: on GoTemplateError it falls back to the literal source with a warning, and it catches RecursionError explicitly so a crafted, deeply-nested {{ if }}/{{ range }} can't exhaust the Python stack and surface as an unhandled 500.

12. Module api — REST Server

ServerState

Class with engine (active InferenceEngine), current_model (loaded ModelManifest) and api_key (str|None for authentication). Instantiated as global singleton. Server lifespan handles cleanup on close.

Middleware Stack

Execution order: RequestLogger → APIKey → RateLimit (conditional) → Disclaimer → CORS. APIKey runs BEFORE RateLimit, so unauthenticated requests are rejected without consuming rate limit tokens.

RequestLogger — Privacy-safe logging. NEVER logs: bodies (prompts/outputs), auth headers, User-Agent. Only: method, path, status, duration.

APIKeyMiddleware — Optional authentication via --api-key flag. Supports Authorization: Bearer <key> and X-API-Key: <key> headers. Public endpoints (/health, /) exempt.

RateLimitMiddleware — IP-based rate limiting, configurable via env vars. Supports per-model rate limiting.

DisclaimerMiddleware — Adds X-AI-Disclaimer header to AI generation endpoint responses (R9).

CORSMiddleware — Origin allow-list configurable via cors_origins / cors_allow_all (not unconditionally all origins). WebSocket upgrades bypass this middleware, so /ws/chat re-checks the same API-key and Origin policy inline in _check_ws_auth_and_origin().

Exception Handlers

Centralized exception handling via register_exception_handlers(app). Maps the entire HFLError hierarchy to HTTP responses, and additionally registers a RequestValidationError handler (per-dialect 400 on /v1/*) and a catch-all Exception handler. All three reshape the body per API dialect via render_envelope() (see "Per-Dialect Error Envelopes" below).

OpenAPI Documentation

All endpoints include tags, summary, and responses in their decorators for auto-generated OpenAPI documentation. Tags include: OpenAI, Anthropic, Ollama, TTS, Health, Metrics, and "HFL Beyond" (the WebSocket /ws/chat surface).

Structured Logging

All logging uses %-style format strings (logger.info('Model loaded: %s', name)) instead of f-strings, avoiding unnecessary evaluation when the log level is disabled.

Health & Metrics Endpoints

Shared Chat Core (chat_core.py)

Every chat surface — OpenAI /v1/chat/completions, Ollama /api/chat and Anthropic /v1/messages — has to make the same decision once the engine has produced text: did the model emit a tool call, and if so what is it? Historically the OpenAI route drifted behind the Ollama route and silently dropped tool calling. That decision now lives in a single place so routes can only differ in their wire-format translation, never in this logic.

The canonical tool-call shape is {"function": {"name": str, "arguments": dict}}; each route maps it to its own wire format. arguments is always a parsed object, never a string (rule C3). When a turn is a tool call, content is "" (rule C4) and tool_calls is always a list — empty when none (rule C7).

def resolve_chat_output(raw_text, model_name, tools, engine_tool_calls=None, *, tools_disabled=False) -> ChatOutput:
    if tools_disabled:
        return ChatOutput(content=raw_text)
    if isinstance(engine_tool_calls, list) and engine_tool_calls:
        return ChatOutput(content="", tool_calls=engine_tool_calls)
    cleaned, parsed = parse_tool_calls(raw_text, model_name, tools)
    return ChatOutput(content="", tool_calls=parsed) if parsed else ChatOutput(content=cleaned)

Tool Calling (3 surfaces)

Function/tool calling is implemented uniformly across all three chat dialects. Each route maps into the engine's OpenAI-function payload on the way in, and maps the canonical ChatOutput.tool_calls out to its own wire shape on the way back.

Per-family marker parsers (tool_parsers.py)

When a backend does not surface structured tool_calls itself, HFL parses the model's native tool-call markers out of the generated text. dispatch(text, model_name, tools) selects a parser by model-name substring, runs it first, and only falls through to the generic JSON-envelope fallback when tools are declared (so ordinary chat that happens to contain JSON is never misread as a tool call).

Streaming with declared tools is buffered: every dialect accumulates the full generation and emits the structured tool_calls in one terminal delta, so a raw <tool_call> marker never leaks as a content token (API-7).

Native agent loop (agent_loop.py)

On /api/chat, opting in with agent_loop=true drives the tool-use cycle server-side: call the model, dispatch any emitted tool_calls (MCP tools only — the model must not hallucinate external names), append role=tool results, and re-call until a tool-call-free reply or max_iterations (default 6). Tool calls within one turn fire concurrently via asyncio.gather. run_agent_loop() returns the final message plus a tool_trace for replay/debugging.

Responses API (POST /v1/responses)

routes_openai_responses.py. OpenAI's higher-level wrapper (the path client.responses.create(...) hits) is implemented on top of the existing chat-completion machinery — no new engine path. It is not stateful: every request is self-contained and the server does not persist a response_id chain.

The non-stream response is built by _render_response() as a heterogeneous ordered output list: reasoning summaries first, then the assistant message (output_text blocks), then any function_call items — plus a usage block with input_tokens/output_tokens/total_tokens. Streaming buffers tool-aware turns and only emits structured function_call items in the final response.completed event; both modes run through the inference dispatcher (a slot is held for the whole stream).

WebSocket /ws/chat (cancellable)

routes_ws.py. The streaming HTTP endpoints are one-shots: cancellation relies on TCP close and carries no actionable signal back. The WebSocket endpoint adds a persistent connection across multiple chat turns, a cancel frame that interrupts the in-flight generation, and token-level server frames so a client renders UI without parsing NDJSON or SSE.

Frame grammar (JSON, one per WS message)

The receive loop reads frames continuously while one turn runs in a background asyncio.Task (_drive_chat), so a cancel can arrive mid-flight. The driver races a per-turn asyncio.Event against each produced token via asyncio.wait(FIRST_COMPLETED) and exits cleanly, leaving the connection open for the next prompt.

Pre-accept gate

WebSocket upgrades bypass the HTTP APIKeyMiddleware and CORSMiddleware, so _check_ws_auth_and_origin() mirrors both inline: API key via ?api_key=, Authorization: Bearer or X-API-Key (constant-time compare), and an Origin allow-list check (strict same-origin by default). Rejections accept-then-close with code 1008 so the browser surfaces the reason.

Per-Dialect Error Envelopes

HFL's native (flat) {"error": str, "code": …, "category", "retryable", "details", "request_id"} shape matches no real provider. errors.py:render_envelope(path, status_code, flat) reshapes it by request-path prefix so real OpenAI/Anthropic SDKs (which branch on error.type) parse it correctly, keeping HFL's rich diagnostic fields as SDK-ignored extensions inside the error object.

Every error code carries a (category, retryable) policy from _ERROR_POLICY (e.g. RATE_LIMIT_EXCEEDED→retryable, VALIDATION_ERROR→not), so a client's retry / circuit-breaker logic decides without parsing prose. Factory helpers (service_unavailable, model_loading, queue_full, queue_timeout, …) all take an optional path for per-dialect rendering.

Validation handler (400)

exception_handlers.py registers three handlers via register_exception_handlers(app). The RequestValidationError handler emits a per-dialect 400 on /v1/* surfaces (real OpenAI/Anthropic return 400, not FastAPI's default 422 {"detail":[…]}), flattening the first error to "loc: msg"; /api/* and everything else keep the native 422. The HFLError handler and a catch-all Exception handler both run their bodies through render_envelope too, so even an unhandled 500 reaches an OpenAI/Anthropic SDK as a parseable error object.

Dispatcher Integration (spec §5.3)

Every chat surface routes its engine call through the shared inference dispatcher via helpers.py, serialising concurrent requests against the non-reentrant llama.cpp / transformers model (concurrent use corrupts the KV cache).

Metrics & Health (updated)

routes_metrics.py exposes GET /metrics (Prometheus text via get_metrics().export_prometheus()) and GET /metrics/json (structured JSON). The RequestLogger middleware (middleware.py) feeds every request into Metrics.record_request(), accumulating per-endpoint / per-status / per-method counters plus all-time latency histograms (_bucket/_sum/_count for Prometheus).

New counters since v0.1.0

The orphan counters exist because a sync engine call run via asyncio.to_thread cannot be cancelled — they let capacity planning see how often a cancel/disconnect leaves an inference running to completion in the background.

GET /healthz (orchestrator, spec §5.5)

Beyond the existing /health* probes, routes_health.py adds /healthz: returns 200 status=ok when an LLM engine is loaded and ready, else 503 status=degraded. The body carries models_loaded, live queue_depth / queue_in_flight from the dispatcher snapshot, and uptime_seconds — one scrape for orchestration.

13. Module i18n — Internationalization

Main Functions

Translation Files

Each language has a JSON file (~441 lines, ~294 keys) in src/hfl/i18n/locales/:

en.json (English) and es.json (Spanish). Keys follow the structure module.action.message, for example: commands.pull.downloading, commands.search.no_results, errors.model_not_found.

Current State (v0.15.0)

Still exactly two locales — en.json and es.json — but each has grown to 441 lines (~294 leaf keys) as every new command (discover, recommend, bench, snapshot, lora, …) registers its commands.<name>.description / option help strings through t() at import time. Language resolution is HFL_LANG → system locale (LC_ALL/LC_MESSAGES/LANG) → en. A missing key falls back to English and, failing that, returns the key verbatim, so an untranslated string degrades gracefully rather than crashing the CLI.

14. Module tray — System Tray Icon

15. Module observability — Metrics, Signing & Audit

Two distinct concerns live here, at two distinct maturity levels. Metrics (src/hfl/metrics.py) is the production telemetry path: it backs /metrics and the SLI view, and now emits aggregatable Prometheus histograms plus dispatcher saturation counters. The src/hfl/observability/ package — ed25519 manifest signing, a structured audit log, and OpenTelemetry tracing — is fully implemented and test-covered but dormant: no module under src/hfl/ imports hfl.observability yet, so these helpers ship as opt-in plumbing with zero production call sites. We document what the code does and flag the wiring gap explicitly.

Prometheus histograms

Metrics (a thread-safe @dataclass guarded by a single threading.Lock) keeps two latency families as all-time, monotonic histograms in addition to the bounded deque(maxlen=1000) ring buffers used for the local JSON percentile view. The deques silently shrink under load (their window is the last 1000 samples), so the exported series is the cumulative bucket array instead — that is what makes histogram_quantile() and rate(..._count) correct across any scrape window and across replicas.

Bucket boundaries (_LATENCY_BUCKETS_MS) are 13 fixed millisecond edges from 5 to 60000, plus a +Inf overflow bucket. _bucket_index() finds the first boundary >= value; record_request() and record_generation() increment the matching bucket and the running _sum_ms. _append_histogram() renders the cumulative _bucket{le=...} / _sum / _count triple.

def _bucket_index(value_ms: float) -> int:
    for i, boundary in enumerate(_LATENCY_BUCKETS_MS):
        if value_ms <= boundary:
            return i
    return len(_LATENCY_BUCKETS_MS)  # +Inf overflow

Two histogram series are exported: hfl_request_latency_ms and hfl_generation_latency_ms, each with the canonical _bucket/_sum/_count lines. The JSON view (export_json()) and get_sli() still compute p50/p95/p99 locally over the deque via linear-interpolation _percentile().

Dispatcher saturation metrics

On every scrape, export_prometheus() pulls a DispatcherSnapshot (from src/hfl/engine/dispatcher.py, a frozen six-int dataclass) via get_dispatcher().snapshot() and exports the load-shedding signals that were previously un-alertable. Reading the snapshot is cheap, so it happens inline under a best-effort try/except (config may not be loaded in some tests).

Alongside these, the exporter publishes streaming-cancel diagnostics: hfl_ws_cancels_total / hfl_ws_cancel_orphans_total (WebSocket cancel frames, and the subset that left an engine call running in the background with the dispatcher slot still busy) and the SSE/HTTP parity counter hfl_stream_cancel_orphans_total.

ed25519 manifest signing dormant

src/hfl/observability/signing.py implements opt-in provenance over model manifests. manifest_digest() carves a stable subset of the envelope (name, repo_id, file_hash, hash_algorithm, size_bytes, quantization, architecture, adapter_paths, parent_digest) — deliberately dropping mutable metadata like last_used so a hfl show never invalidates a signature — and SHA-256s its canonical JSON. sign_manifest_envelope() attaches an {alg:"ed25519", key_id, digest, sig} block; verify_manifest_envelope() checks it against a TrustRoot keyring the operator curates at ~/.hfl/trusted-publishers.json (key_id → base64url public key).

The crypto backend is selected at runtime: it tries pynacl first, then cryptography, raising SignatureUnavailableError if neither is importable; a bad/unauthorized block raises SignatureInvalidError (intended to become a 400 in strict mode). Unsigned manifests stay valid — verify_manifest_envelope() simply returns False.

Audit log dormant

src/hfl/observability/audit.py writes one JSON object per line to HFL_AUDIT_LOG_PATH via a RotatingFileHandler (default 100 MB × 5 backups, tunable through HFL_AUDIT_LOG_MAX_BYTES / HFL_AUDIT_LOG_BACKUPS). It is opt-in: with no path configured, audit_event() is a zero-cost no-op. The event envelope carries ts (UTC, microseconds, Z-suffixed), event, actor (an SHA-256 prefix, never the raw API key), resource, metadata, and outcome. The catalogue is a frozen set AUDIT_EVENTS — model.create/delete/copy/pull/stop/unload/load, blob.upload, mcp.connect/disconnect, api_key.mint/revoke. The emitter never raises; an unknown event name logs a warning but still emits.

OpenTelemetry tracing dormant

src/hfl/observability/tracing.py wraps the OTEL SDK behind a trace_span(name, attributes=...) context manager that becomes a nullcontext() no-op when tracing is off — so call sites never have to check is_enabled(). configure_tracing() is gated on HFL_OTEL_ENABLED; when on it builds a TracerProvider with a BatchSpanProcessor + OTLP/HTTP exporter (endpoint HFL_OTEL_EXPORTER_ENDPOINT, default http://localhost:4318/v1/traces; service name HFL_OTEL_SERVICE_NAME, default hfl). The SDK ships behind the [otel] extra; if it's missing, configuration logs a warning and stays disabled.

16. Modules tools & mcp

The agentic surface. src/hfl/tools/ provides the two built-in tools — web search and URL fetch — exposed Ollama-compatibly at /api/web_search and /api/web_fetch (src/hfl/api/routes_web.py). src/hfl/mcp/ implements both halves of the Model Context Protocol: HFL as a host that connects to external MCP servers and folds their tools into chat, and HFL as a server that publishes its own tools to other hosts. Both are wired into production; MCP rides behind the optional [mcp] extra.

Built-in tools (src/hfl/tools/)

The search backends form a small registry behind an abstract WebSearchBackend:

get_backend() resolves the name (falling back to DuckDuckGo with a warning for unknown values, so the server never 500s on startup). DDG's /l/?uddg= redirect wrappers are unwrapped to the real target URL.

SSRF / DNS-rebinding hardening

web_fetch is hardened against server-side request forgery in depth. The flow:

• Scheme allowlist — only http/https (_ALLOWED_SCHEMES).
• Pre-flight resolution — _resolve_and_validate() calls socket.getaddrinfo() and rejects the URL if any resolved address is private, loopback, link-local, multicast, reserved or unspecified (_is_private_ip(); an unparseable address is treated as unsafe). This blocks the cloud-metadata endpoint 169.254.169.254 and internal services.
• IP pinning (closes the DNS-rebinding TOCTOU window) — the validated IP is pinned for the actual connection. _pinned_request() rewrites the URL host to the vetted IP literal while preserving the original authority in the Host header and, for HTTPS, the TLS sni_hostname extension — so httpx cannot re-resolve at connect time and an attacker with TTL-0 authoritative DNS cannot answer public-here / private-to-httpx. Userinfo (user:pass@) is preserved so Basic-auth fetches keep working.
• Manual redirect following — follow_redirects=False; up to _MAX_REDIRECTS (3) hops are followed by hand, each re-run through the full scheme + private-IP guard and re-pinned, so a public URL cannot 302 onto an internal/metadata address.
• Streamed byte budget — the body is read incrementally via resp.aiter_bytes() with a hard max_bytes cap (default 5 MiB). A non-streaming get would buffer the whole (model-controlled) URL into RAM and could OOM the process; streaming caps actual memory, not just parser input.

def _is_private_ip(ip_str: str) -> bool:
    try:
        ip = ipaddress.ip_address(ip_str)
    except ValueError:
        return True  # unparseable == unsafe
    return (ip.is_private or ip.is_loopback or ip.is_link_local
            or ip.is_multicast or ip.is_reserved or ip.is_unspecified)

The route layer (routes_web.py) validates request bodies with Pydantic (query 1–2048 chars; max_results 1–10; url 1–2048 chars), maps WebSearchError/WebFetchError to HTTP 400, and a crash to 500 with a non-leaking message.

Model Context Protocol (src/hfl/mcp/)

The official mcp Python SDK is an optional dependency. Both modules import unconditionally and degrade gracefully: a missing SDK raises MCPClientUnavailableError / MCPServerUnavailableError only at the point of use, never at import, so installations without the [mcp] extra keep working.

HFL as host — mcp/client.py

MCPClient (a singleton via get_client()) holds live sessions to every configured server. connect(server_id, target) accepts two transports — stdio://<cmd> <args> (launches a subprocess speaking JSON-RPC) and sse://<url> (HTTP + Server-Sent Events) — calls session.initialize() + list_tools(), and wraps each result as an MCPTool. Tool names are namespaced as server_id__tool (qualified_name) to prevent collisions across servers, and to_ollama_tool() converts them to the {type:"function", function:{...}} schema the model sees. autoload_servers() connects every entry in the JSON file pointed to by HFL_MCP_AUTOLOAD; a broken entry is logged but never aborts hfl serve.

Chat integration (src/hfl/api/routes_native.py): _merge_mcp_tools() folds every connected MCP tool into the tools array of an /api/chat request that didn't already declare them (best-effort — a failed read is logged and ignored). When agent_loop=true, the route passes mcp_client.call_tool as the mcp_caller into run_agent_loop(), which dispatches tool calls and re-submits results until a tool-free reply or max_iterations (default 6).

HFL as server — mcp/server.py

HFLMCPServer publishes a minimal, safe-by-default registry HFL_TOOLS: web_search, web_fetch, model_list and model_show. The first two delegate to hfl.tools; the latter two read hfl.models.registry. --capabilities narrows the published set (_filter_tools()). build_server() wires the SDK's @server.list_tools() / @server.call_tool() handlers; a handler ValueError becomes an MCP IsError TextContent response, any other exception is logged and surfaced as a generic internal error. Two transports are served: serve_stdio() (for subprocess hosts) and serve_sse(host, port) (a Starlette app under uvicorn, /sse + /messages/).

CLI surface — hfl mcp

The mcp command in src/hfl/cli/main.py drives both halves:

17. Exception Hierarchy

All inherit from HFLError(message, details) with __str__ combining both. Each exception includes specific contextual data (model_name, repo_id, required_gb, etc.) for detailed diagnostics.

18. Complete Flow: hfl pull

19. Complete Flow: hfl run

20. Complete Flow: hfl serve

21. Legal Compliance and Audit

HFL implements a comprehensive legal compliance system documented with audit references (R1-R9):

Project Legal Files

22. Design Patterns

Architecture Decision Records (ADRs)

Important architectural decisions are documented in docs/adr/ following the standard ADR format:

New Patterns (added since v0.1.0)

23. Dependencies and Extras

New Optional Extras (since v0.1.0)

The extras matrix expanded well beyond the original four. Heavy native deps now carry upper bounds (cap at the next major) because they are the largest vuln/breakage surface. [all] = llama,transformers,vllm,convert,tts,coqui,audio,tray,mcp,mlx,stt,imagegen.

Pin bumps: core now targets transformers>=5.0,<6.0 + torch>=2.5,<3.0 (the 5.x / hub-1.x line), huggingface-hub>=1.5,<2.0, fastapi~=0.135, starlette>=1.3.1 and python-multipart>=0.0.31 (both floors set by CVE fixes). llama-cpp-python>=0.3.20 (gemma4 support). [dev] added mypy>=1.10 and numpy>=1.24.

24. Testing

Test Categories

Integration & Edge Case Tests

Main fixtures (conftest.py): tmp_hfl_home (temp directory), mock_hf_api (HfApi mock), sample_manifest (sample model), populated_registry (pre-populated registry), mock_engine (mocked inference engine), test_client (FastAPI TestClient).

# Run tests with coverage
pytest --cov=hfl --cov-report=html --cov-report=term-missing  # fail_under=75 (pyproject)

# Tests by category
pytest tests/test_api*.py -v          # API only
pytest tests/test_engine*.py -v       # Engines only
pytest -k "not slow" -v               # Exclude slow tests

25. Build and Distribution

Local Storage

~/.hfl/
├── models/               # Downloaded model files
│   └── org--model/       # Directory per model (org--model format)
├── cache/                # HuggingFace Hub cache
├── tools/
│   └── llama.cpp/        # Compiled conversion tools
│       └── build/bin/    # Binaries: llama-quantize, etc.
├── models.json           # Model registry (array of ModelManifest)
└── provenance.json       # Immutable conversion log

26. CI/CD & GitHub

GitHub Actions Workflows

GitHub Templates

Issue Templates: bug_report.yml (structured form with system info, reproduction steps, logs) and feature_request.yml (problem statement, proposed solution, affected component).

PR Template: Checklist including verification of all 5 Compliance Modules (license, provenance, disclaimer, privacy, gating) as project responsible-use norms.

Community files: CONTRIBUTING.md (dev setup, code style, PR process), CODE_OF_CONDUCT.md (Contributor Covenant v2.1), SECURITY.md (vulnerability reporting policy).

Manual-Only Workflows (v0.15.0)

Every one of the 14 workflows under .github/workflows/ is now declared on: workflow_dispatch — the automatic push / pull_request / schedule / tag triggers were intentionally removed. The owner runs each pipeline by hand from the Actions tab. There is no automatic publish: publish-pypi.yml has its tag trigger removed and runs only on manual dispatch (with a testpypi input), using PyPI Trusted Publisher (OIDC, no stored token) plus a Sigstore attestation. build-executables.yml additionally keeps a release: trigger for asset uploads.