Metadata-Version: 2.4
Name: skylar
Version: 0.7.11
Summary: Skylar — local, sovereign, from-scratch LLMs. CLI, loader and OpenAI-compatible server for the Skylar model family: chat with tool calling, a sandboxed agent, embeddings, typed decisions and COBOL.
Author: A. Ivanovitch
License: Apache-2.0
Project-URL: Models, https://huggingface.co/Skyl4r-Ai
Keywords: llm,cobol,code-generation,sovereign-ai,from-scratch,tool-calling,rag,agent
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: torch>=2.1
Requires-Dist: transformers>=4.40
Requires-Dist: tokenizers>=0.15
Requires-Dist: huggingface_hub>=0.20
Requires-Dist: rich>=13.0
Provides-Extra: serve
Requires-Dist: fastapi>=0.100; extra == "serve"
Requires-Dist: uvicorn>=0.23; extra == "serve"
Requires-Dist: pydantic>=2.0; extra == "serve"
Provides-Extra: hybrid
Requires-Dist: flash-linear-attention==0.5.2; extra == "hybrid"
Requires-Dist: fla-core==0.5.2; extra == "hybrid"
Requires-Dist: einops>=0.8; extra == "hybrid"
Requires-Dist: torch>=2.7; extra == "hybrid"
Provides-Extra: grammar
Requires-Dist: llguidance<2,>=1.9.1; extra == "grammar"
Provides-Extra: xgrammar
Requires-Dist: xgrammar<0.3,>=0.2.8; extra == "xgrammar"

# skylar

A tiny **runtime + CLI for the Skylar model family** — local, sovereign, from-scratch LLMs you
load, run, and serve with one `pip install`. It covers **generative chat with tool calling**, an
**agent in a sandbox** (beta), **embeddings / retrieval**, **typed, calibrated decisions** ("System One",
beta) and **COBOL code specialists** — 236M–1B class, runnable on a single GPU or CPU, no data leaving
your machine.

Models live under [`Skyl4r-Ai` on HuggingFace](https://huggingface.co/Skyl4r-Ai):
`Skylar-980M-Cobol` · `Skylar-980M-Cobol-Base` · `Skylar-980M-Cobol-Decider` ·
`Skylar-390M-Cobol` · `Skylar-236M-Base` · `Skylar-236M-Chat` · `Skylar-236M-Embed`.
Framework and training code: [github.com/skyl4r-ai/skylar](https://github.com/skyl4r-ai/skylar).

**New in 0.7.11:** `--yarn FACTOR` for a context longer than the trained one (static YaRN, see *Longer context*); the loop-guard becomes an optional guard, **off by default** (until 0.7.10 it was always on): `--loop-guard on` or a per-request `loop_guard: true` switches it on (see *The loop-guard*; if you see the model loop, please tell us); the default engine's prefill computes the logits of the last position only (up to 0.7.10 it computed them for the whole prompt, which was the memory peak of a long request: on a 24 GB GPU a 128K-token prompt now fits, same tokens); an out-of-memory error inside the fused SiTU kernel is raised as it is (up to 0.7.10 one request too long for the GPU switched the whole process to the slower eager kernel for good).

**New in 0.7.10:** documentation only (docstrings of the decision API); the code is that of 0.7.9.

**New in 0.7.9:** a server whose grammar engine is installed but fails to start (another version than the pin, an
error at its warm-up) now starts anyway with structured output off and says why in its start line (until 0.7.8 it did
not start); a tool marked `"strict": true` on a server without an engine is served unconstrained, with one line in the
log, instead of a 400 (many clients send `strict` by default); on a constrained request `<|im_end|>` can no longer end
the turn inside the `<think>` block; a schema the engine rejects, a malformed named `tool_choice` and concurrent
compilations give a 400 instead of a 500. An answer cut by the loop-guard now ends with `finish_reason: "length"` both
streaming and not (the stream said `stop`), as an answer cut by `max_tokens`. Importing `skylar` no longer touches the
GPU (0.7.8: the hybrid kernels' library is loaded at the model's first use on CUDA).

**New in 0.7.8:** structured output in `skylar serve` (below): JSON mode, JSON Schema, strict tool calls, `tool_choice`
`required` or a named function, Anthropic `output_config.format`, with llguidance or xgrammar. Every other request
decodes as in 0.7.7.

**New in 0.7.7:** an opt-in fast engine for `skylar serve` on a GPU. `--cuda-graph` replays the decode step as a CUDA graph: on a 24 GB consumer GPU, a ~1B hybrid model in bf16 with a 4K-token context goes from 30 to 132 tokens/s with one request and from 204 to 748 tokens/s with eight. It uses a preallocated cache and other attention kernels, so in bf16 its answers are not token for token those of the default (the quality is the same), and past ~48K tokens of context it is slower than the default (2× at 128K): use it up to ~32K. `--prefill-chunk N` picks the same engine without the graph and reads long prompts in chunks of N tokens, so a 128K-token context fits a 24 GB GPU (~6 GiB for a ~1B model; with `--cuda-graph` the chunk is 8,192 unless given). Without either flag nothing changes: the default engine is 0.7.6's, token for token. The same in Python: the model's `generate` / `generate_streaming` (so `Skylar.generate`, `Skylar.stream` and `skylar agent`) keep the 0.7.6 decode unless you pass `static_cache=True`. The hybrid layers' short-convolution cache no longer keeps its whole input alive (less memory after a long prompt, same values). The default engine no longer builds the (prompt × prompt) prefill mask for a hybrid model like Skylar2, which never read it (it took gigabytes on a long prompt; same tokens). `--fp32-head` now works on CUDA: up to 0.7.6 it ran under the server's bf16 autocast, so on a GPU the head came out in bf16 and the option had no effect (on CPU it worked).

**New in 0.7.6:** `skylar serve` on CPU never goes through `torch.compile` (it needed a C++ compiler at run time: in a slim image every answer was a 500, while `/health` said ok) — eager was as fast; the server starts listening only after a test generation succeeded (else it exits with code 1), so `/health` ok means the model answers; `--dtype auto` on CPU now loads bf16 where the CPU computes it natively (AVX512-BF16/AMX: ~2× faster, same greedy text on our prompts), fp32 elsewhere, and the chosen dtype is always explicit (a bf16 checkpoint loads in fp32 with `--dtype fp32`); new `--fp32-head`: final logits in float32 over bf16 weights (off by default). Our model repos now keep bf16 as `model.safetensors` and the original fp32 as `model.fp32.safetensors`: `--dtype fp32` (or `Skylar.load(dtype=torch.float32)`) loads that original, `--dtype auto` the bf16 main file, `--variant` picks one by hand. `Skylar.load()` without a dtype now loads the checkpoint as stored on every transformers version (4.x used to cast it to fp32).

**New in 0.7.5:** streaming decodes each new token on its own (a 32K-token answer reaches the client ~50 s sooner per
row, with the same text byte for byte); a tool sent without `description` or `parameters` reaches the model in its
training format; the embedder is the framework's own, ready for Skylar 2 embedders (the published 236M vectors are
unchanged). Answers can be up to 32,768 tokens long (8,192 up to 0.7.3).

## Install

```bash
pip install skylar                      # CLI + Python API
pip install "skylar[serve]"             # + the OpenAI- and Anthropic-compatible server
pip install "skylar[serve,hybrid]"      # Skylar 2 hybrid models on a CUDA GPU (Triton kernels of their linear-attention layers)
```

Python ≥ 3.10. Without a GPU the Skylar 2 hybrid layers run in plain PyTorch: correct, but slow on long inputs.

## Use it — CLI

```bash
# chat with any Skylar generative model (no forced persona — steer it with --system)
skylar chat --model Skyl4r-Ai/Skylar-236M-Chat --system "Sei un assistente che risponde dal contesto."

# embeddings / retrieval (any SkylarEmbedder model)
skylar embed --model Skyl4r-Ai/Skylar-236M-Embed --query "prestito per comprare casa" \
  --docs "Il mutuo ipotecario è un finanziamento per l'acquisto di un immobile." "Domani il meteo prevede pioggia su tutto il nord Italia."

# typed, calibrated decisions on a state: what does this COBOL program print on this input?
#   state.json     {"cobol": "<program>", "stdin": "12\n30\n"}
#   questions.json {"output": {"type": "choice", "instructions": "What does the COBOL program in `cobol` print
#                   on the input in `stdin`?", "criteria": {"A": "0042", "B": "0041"}}}
skylar decide --model Skyl4r-Ai/Skylar-980M-Cobol-Decider --state-file state.json --questions questions.json

# one-shot generation (HF repo id or a local checkpoint dir)
skylar generate --model Skyl4r-Ai/Skylar-236M-Chat --prompt "..."

# the COBOL specialist — completes a COBOL stub into a full, compilable program
#   (auto-downloads Skylar-390M-Cobol; it's a stub completer, not a chatbot)
skylar cobol --example
skylar cobol --stub-file my_task.cbl --compile        # your own stub + GnuCOBOL check

# multi-user OpenAI-compatible server (needs the [serve] extra) — full details in "Serve it" below
skylar serve --model <any-skylar-model> --port 8000     # interactive docs at http://localhost:8000/docs

# an agent (beta): bash, read_file and write_file on a folder, inside a Docker sandbox
skylar agent --model <a tool-calling Skylar model> --workdir ./project "Write fizzbuzz.py, run it and check the output."
```

Decoding is greedy by default (`--temperature 0.0`) with **no repetition penalty** (`1.0`, as vLLM and
🤗 Transformers: a penalty also hits the prompt's tokens, so it works against copying from a document or a
code skeleton); an optional loop-guard (off by default) can cut a runaway repetition. There is **no forced system prompt** — pass
`--system "..."` to steer a chat model. (The `skylar cobol` subcommand handles the COBOL prompt format for
you.)

### `skylar agent` (beta)

The model gets a task and three tools — `bash`, `read_file`, `write_file` — on the folder you pass with
`--workdir`. At every step it calls tools or answers; each call runs for real and its output goes back to the
model, until it answers or `--max-steps` (default 20) ends. Commands run in a Docker container with no
network, a read-only root and only your folder mounted on `/work`. The image (`skylar-agent-sandbox:1`: bash,
Python, gcc, Java, Node, GnuCOBOL) is built from the Dockerfile shipped in the package the first time you run
the agent. `--sandbox local` runs the commands on your machine instead, asking before each one (`--yes` to
skip asking). `--lang it|en` sets the language of its instructions; `--transcript run.json` saves the
conversation.

## Use it — Python

```python
import skylar

# generative chat — pass your own system prompt (no forced persona)
m = skylar.load("Skyl4r-Ai/Skylar-236M-Chat")          # HF repo id or a local dir
print(m.generate("Domanda: dove ha sede la Banca d'Italia?",
                 system="Rispondi solo dal contesto fornito."))
for delta in m.stream("..."):                          # streaming
    print(delta, end="", flush=True)

# embeddings / retrieval
e = skylar.load_embedder("Skyl4r-Ai/Skylar-236M-Embed")
ranked = e.rank("costo del denaro", ["la BCE alza i tassi", "ricetta pizza"])

# typed, calibrated decisions ("System One", beta): no text generated, probabilities you can trust
d = skylar.load_decider("Skyl4r-Ai/Skylar-980M-Cobol-Decider")
d.decide({"cobol": open("program.cbl").read(), "stdin": "12\n30\n"}, {
    "output": {"type": "choice", "instructions": "What does the COBOL program in `cobol` print on the input in `stdin`?",
               "criteria": {"A": "0042", "B": "0041", "C": "0084"}}})
# -> {"output": {"type": "choice", "probabilities": {"A": ..., "B": ..., "C": ...}, "choice": ..., "confidence": ...}}

# the COBOL specialist — a stub completer (not a chatbot)
c = skylar.load("Skyl4r-Ai/Skylar-390M-Cobol")
print(c.complete_cobol(my_stub))                       # -> full, compilable COBOL program
```

`skylar` also registers the architecture with 🤗 Transformers, so this works too:

```python
import skylar  # registers skylar2 (and the earlier nano-transformer)
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("Skyl4r-Ai/Skylar-236M-Chat")
```

## Serve it — multi-user HTTP API (`skylar serve`)

`skylar serve --model <id>` turns any Skylar generative model into an **OpenAI-compatible HTTP
server** built for concurrent users. Requests from many clients are fused into **dynamic
micro-batches** on a single worker that owns the model — so one GPU (or CPU) serves a whole demo
without per-request OOM or GPU contention — and each request can **stream** its tokens.

```bash
pip install "skylar[serve]"
skylar serve --model Skyl4r-Ai/Skylar-236M-Chat          # swap the id for ANY Skylar model
#  → http://127.0.0.1:8000   ·   interactive docs: http://127.0.0.1:8000/docs
```

Open **`/docs`** for the auto-generated Swagger UI — every endpoint, schema, and example is
described there (or **`/redoc`** for ReDoc). The model is whatever you pass to `--model` (an HF
repo id or a local checkpoint dir); an embedder model is auto-detected and served at
`/v1/embeddings` instead.

| Method & path | What it does |
|---|---|
| `POST /v1/chat/completions` | OpenAI chat format, **function calling** included (`tools` → `tool_calls`, parallel calls, `tool` turns). `"stream": true` → Server-Sent Events. |
| `POST /v1/messages` | **Anthropic-compatible** Messages API (same engine): `system`, content blocks, `stop_sequences`, `thinking` → `<think>` control, CoT returned as `thinking` blocks, native Anthropic SSE events. |
| `POST /generate` | One `prompt` → one `completion`. |
| `GET /health` | Liveness + which model/device is loaded. |
| `GET /metrics` | Throughput, batch sizes, queue depth. |

```bash
# one-shot completion
curl localhost:8000/generate -H 'content-type: application/json' \
  -d '{"prompt": "Dove ha sede la Banca d'\''Italia?", "max_new_tokens": 64}'

# OpenAI chat format (+ "stream": true for SSE)
curl -N localhost:8000/v1/chat/completions -H 'content-type: application/json' -d '{
  "messages": [{"role":"system","content":"Sei un esperto COBOL."},
               {"role":"user","content":"Somma due campi PIC 9(4)."}],
  "max_tokens": 256, "stream": true
}'
```

Drop-in with the official OpenAI client — just point `base_url` at the server:

```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
r = client.chat.completions.create(
    model="Skyl4r-Ai/Skylar-236M-Chat",
    messages=[{"role": "user", "content": "Spiega cosa fa questo COBOL ..."}],
    stream=True,
)
for chunk in r:
    print(chunk.choices[0].delta.content or "", end="", flush=True)
```

Function calling works as with OpenAI: pass `tools`, get `tool_calls` back (`finish_reason: "tool_calls"`),
run them, send the results as `tool` messages. The model sees the tools in the format it was trained on: every tool
written as `{"name", "description", "parameters"}`, with an empty `description` and an object without properties when
the request leaves them out (until 0.7.4 they were left out of the prompt too).

```python
tools = [{"type": "function", "function": {"name": "get_weather", "description": "Current weather of a city",
          "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}}]
r = client.chat.completions.create(model="skylar", tools=tools,
                                   messages=[{"role": "user", "content": "Che tempo fa a Torino?"}])
print(r.choices[0].message.tool_calls)  # [ChatCompletionMessageToolCall(function=Function(name='get_weather', arguments='{"city": "Torino"}'), ...)]
```

…or with the official Anthropic client (`/v1/messages` speaks the Anthropic wire format,
streaming events included; tool use is not supported and returns an explicit 400):

```python
import anthropic
client = anthropic.Anthropic(base_url="http://localhost:8000", api_key="not-needed")
msg = client.messages.create(
    model="Skyl4r-Ai/Skylar-236M-Chat",
    max_tokens=256,
    system="Sei un esperto programmatore COBOL.",
    messages=[{"role": "user", "content": "Somma due campi PIC 9(4)."}],
    stop_sequences=["\n\n\n"],
)
print(msg.content[-1].text)          # a <think> CoT, if any, arrives as a thinking block
```

### Structured output (0.7.8)

With a grammar engine installed, the server can **guarantee** the shape of an answer: while decoding, each token
is drawn only among the tokens the grammar allows, and `<|im_end|>` is allowed only once the JSON is complete.

```bash
pip install "skylar[serve,grammar]"     # llguidance (MIT); or skylar[xgrammar] for xgrammar (Apache-2.0)
```

| You send | You get |
|---|---|
| `response_format: {"type": "json_object"}` | a JSON object |
| `response_format: {"type": "json_schema", "json_schema": {"name", "schema"}}` | an object valid against the schema, keys in the schema's order |
| a tool with `"strict": true` | its calls carry arguments valid against its `parameters`; the name is always one of the tools |
| `tool_choice: "required"` | the turn is tool calls only (`parallel_tool_calls: false` = exactly one) |
| `tool_choice: {"type": "function", "function": {"name": …}}` | exactly one call of that tool |
| `/v1/messages` with `output_config: {"format": {"type": "json_schema", "schema"}}` | the text block is an object valid against the schema |

Every example in this section was run against `skylar serve` 0.7.8 (the outputs in the comments are the real ones:
a ~1B chat checkpoint, CPU, greedy). The formats are the ones the model was trained on: a call is `<tool_call>` + one JSON line + `</tool_call>`, and with
`json_schema` the schema is also appended to the last user turn after one fixed line, as in the training data (send
`"schema_in_prompt": false` for the grammar alone). An answer cut by `max_tokens` ends with `finish_reason: "length"`
(`stop_reason: "max_tokens"`) and is not a complete JSON: give it room. Without a grammar engine (or when the one
installed fails to start: the start line says why) the server starts as before: a JSON answer and a `required` or named
`tool_choice` get a 400, a tool marked `strict` is served unconstrained (strict needs `skylar[grammar]`); every other
request decodes exactly as in 0.7.7.

```bash
curl localhost:8000/v1/chat/completions -H 'content-type: application/json' -d '{
  "messages": [{"role": "user", "content": "Estrai i dati: Fattura FT-118 del 3 novembre 2026, totale 1.525,00 euro."}],
  "max_tokens": 200,
  "response_format": {"type": "json_schema", "json_schema": {"name": "fattura", "schema": {
    "type": "object", "additionalProperties": false, "required": ["numero", "data", "totale"],
    "properties": {"numero": {"type": "string"},
                   "data": {"type": "string", "pattern": "^[0-9]{4}-[0-9]{2}-[0-9]{2}$"},
                   "totale": {"type": "number"}}}}}
}'
# → "content": "{\"numero\": \"FT-118\", \"data\": \"2026-11-03\", \"totale\": 1525.0}", "finish_reason": "stop"
```

```python
from openai import OpenAI
from pydantic import BaseModel

class Invoice(BaseModel):
    number: str
    date: str
    total: float

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
r = client.chat.completions.parse(
    model="skylar", max_tokens=200, response_format=Invoice,
    messages=[{"role": "user", "content": "Extract the invoice: INV-7731 dated 2026-10-28, total $3,672.00."}])
print(r.choices[0].message.parsed)      # Invoice(number='INV-7731', date='2026-10-28', total=3672.0)

tools = [{"type": "function", "function": {"name": "get_weather", "strict": True, "parameters": {
    "type": "object", "additionalProperties": False, "required": ["city", "unit"],
    "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}}}}]
r = client.chat.completions.create(model="skylar", tools=tools, tool_choice="required",
                                   messages=[{"role": "user", "content": "Che tempo fa a Torino?"}])
print(r.choices[0].message.tool_calls[0].function.arguments)   # {"city": "Torino", "unit": "celsius"}
```

```python
import anthropic
client = anthropic.Anthropic(base_url="http://localhost:8000", api_key="not-needed")
msg = client.messages.create(
    model="skylar", max_tokens=200,
    messages=[{"role": "user", "content": "Extract the invoice: INV-7731 dated 2026-10-28, total $3,672.00."}],
    output_config={"format": {"type": "json_schema", "schema": {
        "type": "object", "additionalProperties": False, "required": ["number", "date", "total"],
        "properties": {"number": {"type": "string"}, "date": {"type": "string"}, "total": {"type": "number"}}}}})
print(msg.content[0].text)              # {"number": "INV-7731", "date": "2026-10-28", "total": 3672.0}, indented by 2
# messages.parse(output_format=Invoice) and messages.stream(..., output_config=...) work the same way
```

Not yet: `response_format` together with `tools` (400: send one or the other), tools on `/v1/messages`, OpenAI
`allowed_tools` / custom tools.

### The loop-guard (optional, off by default since 0.7.11)

Small models can fall into a loop and repeat the same text until `max_tokens`. With the loop-guard on, on
`/v1/chat/completions` (streaming and not) and `/v1/messages` the server watches the **tail** of the answer: when the shortest unit of up to 160
characters repeats exactly 4 or more times in a row (at least 30 characters; a unit made only of symbols such as `-`,
`=`, `*` or `|` only past 320, so a separator or a table row is not cut), it keeps the first copy, stops the generation
and answers with `finish_reason: "length"` (`stop_reason: "end_turn"` on `/v1/messages`). The comparison is exact:
`'A001'`, `'A002'`… is not a loop. `POST /generate` never cuts; a structured-output request (JSON schema) is bounded by
its grammar instead.

It is an **optional guard, off by default** since 0.7.11 (up to 0.7.10 it was always on): without it you get the raw
generation, as from other servers, and a repetition the user asked for («write *ciao* 50 times») is never cut. Switch
it on where a looping answer must not reach the user:

```bash
skylar serve --model <id> --loop-guard on            # the server's default for every request
```

```python
client.chat.completions.create(model="m", messages=msgs, extra_body={"loop_guard": True})    # one request
sk.generate("…", loop_guard=True)                                                             # Python API
```

A request can also say `"loop_guard": false` on a server started with `--loop-guard on`. **If you see the model loop,
please tell us**: open an issue at [github.com/skyl4r-ai/skylar/issues](https://github.com/skyl4r-ai/skylar/issues) with
the request (or the prompt) and the answer — that is how the next models stop doing it.

### Longer context: static YaRN (0.7.11)

`--yarn FACTOR` stretches the RoPE of the attention layers with static YaRN (Peng et al., arXiv 2309.00071): the context
grows from the length the model was trained at to that × FACTOR. For a model trained at 131,072 tokens, `--yarn 2`
serves 262,144.

```bash
skylar serve --model <id> --yarn 2                   # 131072 → 262144 tokens; the startup line says rope=yarn x2
```

```python
import skylar
sk = skylar.load("<id>", yarn=2)                     # or skylar.apply_rope_scaling(model, 2) on a loaded model
```

- **Switch it on only when you need more than the trained length.** Static YaRN applies to every prompt, short ones
  too, so with the flag on a short prompt is not computed exactly as without it.
- The recurrent (KDA) layers have no positions and do not change; the attention layers get YaRN's frequencies
  (β_fast 32, β_slow 1) and its attention factor 0.1·ln(FACTOR) + 1 on cos and sin.
- A checkpoint can carry the same setting in `config.json`, in transformers' form:
  `"rope_scaling": {"rope_type": "yarn", "factor": 2.0, "original_max_position_embeddings": 131072}` (then set
  `max_seq_len` to the extended length).
- Memory, one request, a ~1B hybrid model in bf16, default engine, peak allocated: 16K tokens 4.4 GiB, 32K 5.9 GiB, 64K 8.7 GiB, 128K 14.3 GiB. At 256K the default engine does not fit a 24 GB GPU: add `--prefill-chunk 8192` (the fixed-cache engine, see 0.7.7), 9.4 GiB, prefill ~23 s, then ~32 tokens/s.
- For long prompts `--prefill-chunk 8192` is recommended: at 64K and 128K tokens (one request, measured on one prompt each, each pair in its own process) it used ~44 % and ~58 % less memory (4.8 vs 8.5 GiB, 5.9 vs 14.2 GiB) and decoded faster (28.1 vs 20.4 and 30.7 vs 10.7 tokens/s), with a prefill about as long (2.6 vs 2.8 s, 6.8 vs 7.0 s) and the same 512 tokens. In bf16 that engine is not guaranteed to give the default's tokens.

### Tuning concurrency

| Flag | Default | Meaning |
|---|---|---|
| `--max-batch` | `8` | Max requests fused into one forward pass. Raise for more throughput until VRAM/latency says stop. |
| `--max-wait-ms` | `15` | How long to wait for stragglers before launching a batch. Higher = bigger batches, slightly more latency. |
| `--max-queue` | `256` | Input-queue depth in **requests**; beyond it → **503**. |
| `--max-queue-tokens` | `auto` | Queue cap in **tokens** (prompt+max_new) — a few long requests can't saturate the queue. `0` = auto (~2 full batches). |
| `--dtype` | `auto` | Weight dtype: `auto` (bf16 on GPU; on CPU bf16 if the CPU has native bf16 — AVX512-BF16 or AMX — else fp32) · `bf16` · `fp16` · `fp32`. bf16 ≈ half the weight memory and ~2× decode; the rotary frequencies and the decay parameters of the hybrid layers stay fp32. For CPU measurements comparable with earlier fp32 runs pass `--dtype fp32`. |
| `--variant` | — | Weights file next to the main one: `fp32` = `model.fp32.safetensors`, `main` = always `model.safetensors`. Unset: the fp32 original when `--dtype fp32` is asked and the repo has it, else the main file. |
| `--grammar-backend` | `auto` | Structured output engine: `auto` (llguidance, then xgrammar, whichever is installed) · `llguidance` · `xgrammar` · `off`. Without one (or if it fails to start), JSON answers and a `required` / named `tool_choice` get a 400, `strict` tools are served unconstrained, nothing else changes. |
| `--cuda-graph` | off | GPU only: the decode step replayed as a CUDA graph, ~4× tokens/s up to ~32K of context, slower past ~48K (see 0.7.7). It is faster but its greedy text can differ from the default engine (other kernels in bf16). |
| `--prefill-chunk` | — | Tokens per forward pass in the prefill: a longer prompt is read in chunks, on the fixed-cache engine of `--cuda-graph` without the graph (with `--cuda-graph` the chunk is 8,192 unless given). Far less memory on long prompts (see *Longer context*); in bf16 its text is not guaranteed to be the default's. |
| `--fp32-head` | off | Final logits in float32 from a float32 copy of the head, over bf16/fp16 weights (+ vocab × d_model × 4 bytes). It can help exact copying at long context; off by default. On CUDA it has an effect only from 0.7.7 (up to 0.7.6 the bf16 autocast turned it back to bf16). |
| `--yarn` | off | Static YaRN on the attention layers' RoPE: the context becomes the trained length × FACTOR (see *Longer context*). Only when needed: it applies to every prompt. |
| `--loop-guard` | `off` | Optional guard: `on` = cut a looping tail on `/v1/chat/completions` and `/v1/messages` (see *The loop-guard*); a request can set `loop_guard` itself. Off by default since 0.7.11. |

### How it works (for implementers / a future maintainer)

The server is **`skylar/serve.py`** — the model code (`decoder.py` / `attention.py`) is left
untouched:

- **One worker owns the model.** Async routes enqueue requests; a single background thread pulls a
  micro-batch (up to `--max-batch`, waiting `--max-wait-ms`) and runs it. No two CUDA calls race,
  and there is exactly one set of KV-caches in flight. The input queue is bounded **in tokens**
  (`--max-queue-tokens`), so a flood of requests is shed with **503** rather than accepted blindly —
  though a single micro-batch of very long prompts can still pressure VRAM until the planned
  in-flight KV arena lands.
- **True batched decoding.** `generate_batch()` left-pads ragged prompts and builds a 4D additive
  mask (causal + pad) that `Skylar2ForCausalLM.forward` already accepts (its dense-mask SDPA path), so
  prompts of different lengths decode together with a shared KV-cache and per-row EOS stop. A Skylar 2
  hybrid is prefilled one row at a time (a pad token must never enter a recurrent state), then decodes in
  the same batch. Per-row sampling mirrors `Skylar2ForCausalLM.generate` exactly → **batched output is
  token-for-token identical to single-stream** (proven by `tests/test_batch_equiv.py`).
- **Batching + streaming coexist.** Each request carries its own queue; the worker pushes text
  deltas into it as tokens are produced, so every request in a batch streams independently.
- **Whole characters only.** An emoji (or any character split over several byte-level tokens) is sent
  when its last byte arrives, never as «�»; a character left unfinished at the stop is dropped. The
  vocabulary's tokens that *are* «�» (learned from broken web text) are never sampled (0.7.1), nor the byte pieces that
  would assemble one, as «ï» then «¿½» (0.7.2).
- **Streaming in linear time.** Each token decodes only the ids after the last complete character
  (`skylar.chatml.StreamDecoder`, the prefix offset of vLLM's detokenizer); until 0.7.4 the whole answer was decoded
  again at every token, ~3 s more for an 8K-token answer and ~50 s for a 32K one, per row. The streamed text is the
  one of the whole answer decoded at once, byte for byte (`tests/test_stream_decoder.py`). (0.7.5)
- **The loop-guard (when on) ignores separators.** A run of symbols only (`-----`, `=====`, `|---|`) counts as a loop past 320
  characters; words and phrases repeated 4 times past 30 characters are still cut. Until 0.7.2 a single 32-dash token
  (a COBOL comment line `*-----…`) stopped the stream. A cut keeps the first copy of the repeated unit. (0.7.3)
- **Long answers.** `max_tokens` (chat) and `max_new_tokens` (generate) accept up to 32,768 new tokens; until 0.7.3 the
  limit was 8,192. An agent that rewrites a whole COBOL member in one tool call (~7K tokens for a 500-line program) needs
  more than 8K. Nothing else changes: a request at or under 8,192 behaves as before. (0.7.4)
- **Current limits (PoC).** *Static* micro-batching (a batch starts and finishes together). For
  heavy, time-skewed load the next step is *continuous* batching (adding requests to an in-flight
  batch). One model per process; greedy is the default, sampling params are per-request.

```bash
python tests/test_batch_equiv.py     # run after touching batching/masking: batched == single-stream
```

## Serve decisions — `POST /v1/decide`

`skylar serve` detects a decider and serves it instead of chat:

```bash
skylar serve --model Skyl4r-Ai/Skylar-980M-Cobol-Decider --port 8000
curl -s localhost:8000/v1/decide -H 'content-type: application/json' -d '{
  "state": {"cobol": "<program>", "stdin": "12\n30\n"},
  "questions": {"output": {"type": "choice", "instructions": "What does the COBOL program in `cobol` print on the input in `stdin`?",
                           "criteria": {"A": "0042", "B": "0041"}}}}'
# -> {"model": ..., "answers": {"output": {"type": "choice", "probabilities": {...}, "choice": ..., "confidence": ...}}}
```

All the questions about one state run in one batch. A decider answers the kind of questions it was trained on:
the published one, on COBOL programs and their input (its card says which questions and how well). The confidence formulas: choice `(p_max − 1/n)/(1 − 1/n)`, score `1 − Σ pᵢ|i − m| / MAD of the uniform rubric`, noul `|2p − 1|`.

## What's inside

The Skylar models use a custom decoder (`Skylar2ForCausalLM`, formerly `NanoTransformer`, Qwen3-style:
RMSNorm + RoPE + GQA + QK-Norm + SwiGLU; Skylar 2 adds the KDA hybrid, Attention Residuals and
GatedNorm), trained 100% from scratch (no third-party pretrained weights). This package vendors the
framework's model code (kept identical to it) so the published weights load anywhere.
Skylar 2 hybrid checkpoints use the Triton kernels on a CUDA GPU (`pip install "skylar[hybrid]"`) and plain
PyTorch elsewhere.

## License

Apache-2.0 (code). Each model's license is on its model card. IP: A. Ivanovitch, CEO SKYL4R.
