lllm2 / local models
Memory details
Available RAM unknown.System usage includes all applications. RAM usage excludes reclaimable cache; engine RSS in results is a separate measurement.Choose a model
Set up an engine
Run in a terminal on the model workstation:
lllm2 engines install cudaDownloads a prebuilt CUDA engine with runtime libraries included. Requires an NVIDIA driver supporting a published engine bundle; the installer checks compatibility. No CUDA toolkit or build tools are needed. After installation, choose Check again. An existing llama-server can be selected under Customize settings.
Downloads
Select by path · all installed variants
Exact paths distinguish custom checkpoints and copies of the same variant.
Preparing recommended settings…
Settings source & evidence
Recommendation evidence
Discovering models and engines…
Exact error details
Connect a coding agent
Ready to uselllm2 piRun this command in a terminal on the model workstation, in your project directory. The agent CLI must be installed.
Context for coding agents
Agent instructions, tools and conversation history all consume context. If a request exceeds the window, increase context under Customize and restart the model, or shorten the agent session. Relaunch the agent after changing server context.
One observed Claude session needed more than 32K tokens and worked at 65,536. Your workload may need a different allocation.
Engine & device
Engine device probe
Conversation & acceleration
Advanced · independent cache precision
Inherit uses the common choice above. Selecting a common precision explicitly links both again. Mixed pairs are experimental; a CUDA library can advertise these types while lacking a mixed attention kernel. Launch may fail or fall back. Context capacity and quality need measurement; the inherited q8 planner does not estimate new pairs.
Advanced · prompt processing
Batch controls change how much prompt is processed at once. Larger values may read prompts faster but need more working memory. Leave blank to keep engine defaults; compare before changing them.
Advanced · experimental CUDA execution
Leave engine defaults unless a comparison shows a benefit. Support and compatibility reasons appear below.
Advanced · speculative methods
The explicit combined method is draft-mtp,ngram-simple. Build 662a0b0 tries lookup first and MTP on misses. MTP draft length and lookup M are independent widths. Blank lookup controls preserve old launches; the combined mode explicitly uses N3/M3 when blank. Set both together, with N ≤ M. No copying profile is recommended without measured benefit.
Advanced · conversation reuse
Finite controls for saved conversation state. Zero disables each component; blank preserves normal engine defaults. These limits exclude model weights and other working memory.
Feature availability · support details
Support comes from engine and checkpoint checks. “Available to try” does not establish a speedup or a successful launch. Form choices below are separate from the running model. Measured benefit: see comparable runs in Saved comparisons; this availability check does not assess it.
Model locations & engine setup
To download an engine, run lllm2 engines install cuda in a terminal on the model workstation. Requires a compatible NVIDIA driver; no CUDA toolkit or build tools are needed. Then rescan.
Experiment configuration
This draft is separate from Launch. Experiments use the same GPU and stop the serving model when explicitly run.
Experiments
Experiment guide ↗Quick sweep: 1K, 16K and 64K input tokens, with 256 output tokens by default. Sizes are capped to fit the experiment context per slot and duplicates are removed. Cold speed samples each start a fresh engine with prompt caching disabled. The separate warm conversation experiment is described below.
Source copy, single edit and long-context retrieval/edit check the complete returned source. Retrieval/edit places three fixed facts early, middle and late in the input; it is a narrow check, not general accuracy grading. These workloads use an output cap of at least 2048 tokens (or your larger requested budget, up to 4096), including thinking, and stop naturally. Other workloads retain the requested budget. Failed adherence is shown separately and cannot establish a copy speedup. Mixed selections share input sizes capped to fit the larger source output budget.
Warm conversation · separate experiment
Runs six controlled turns: cold start, append, suffix edit, switch away, switch back and early history edit; then replays the exact prompts with reuse disabled. Uses one slot, the fixed prompt size, output budget and repetition count above, with the long-code workload. Sweeps and context search do not apply. Twelve requests per repetition; allow several minutes. Blank Advanced conversation controls become 2048 MiB host cache and four checkpoints for this experiment only; explicit zero is respected. Reuse at larger contexts remains unverified. Results report measured reuse, client first-token event timing and sampled process RAM separately from cold baselines.
Compare selected combinations
0 selectedAdjust experiment settings above, then add each combination. Keep model, engine, backend, device, context and slots the same. Each combination uses the workloads and budgets above; total time grows with the number selected.
No combinations selected. Add the current settings to begin.
Queue & engine
Saved comparisons
One row per sample. Compare matching models, builds, workloads and budgets. Expand a row for context, timing and evidence.
How to read and export these measurements
Rows stay separate by coding workload. Compare identical budgets and engine builds. VRAM is sampled total device use, including the desktop. Ordinary cold workloads force the requested output budget, ignoring early end-of-sequence. Source copy, single edit and retrieval/edit stop naturally and report exact source adherence; compare actual output counts and total elapsed time too. Adherence is a narrow text check, not general answer-quality grading.
CSV and clipboard export the displayed samples in their current order, including partial and failed runs. Runs without samples are retained as empty rows. Cold prefill and warm processed-prefill rates have separate columns. Full JSON retains the complete evidence.
| No experiments yet. | |||||
Engine logs & exact command
No engine started.