Metadata-Version: 2.4
Name: blablador
Version: 0.2.0
Summary: Python client for JSC's Blablador LLM API, built on the OpenAI SDK
Author: Applied Machine Learning Lab at JSC
Author-email: Applied Machine Learning Lab at JSC <ai-support@fz-juelich.de>
License-Expression: MIT
License-File: LICENSE
Requires-Dist: openai>=3.42,<3.43
Requires-Dist: anyio>=4.10,<5
Requires-Dist: httpx2>=2.12,<3
Requires-Dist: typing-extensions>=4.14,<5
Requires-Python: >=3.10
Project-URL: Homepage, https://gitlab.jsc.fz-juelich.de/sdlaml/agents-on-hpc/blablador
Project-URL: Repository, https://gitlab.jsc.fz-juelich.de/sdlaml/agents-on-hpc/blablador.git
Description-Content-Type: text/markdown

# blablador

Python client and command-line tool for [Blablador](https://sdlaml.pages.jsc.fz-juelich.de/ai/guides/blablador_api_access), the LLM service of the Jülich Supercomputing Centre.

Blablador's API is OpenAI-compatible, so this package is a thin layer over the official [`openai` SDK](https://github.com/openai/openai-python): everything the SDK does works unchanged, and the package adds what Blablador needs on top:

- clients preconfigured for Blablador (`Blablador`, `AsyncBlablador`)
- typed model listing with Blablador's fields and the current `alias-*` names
- the model's reasoning, whichever backend serves it (`blablador.reasoning()`)
- the rerank endpoint, which the OpenAI API does not have (`client.rerank`)
- fixes for requests the gateway rejects but the OpenAI API allows
- rate-limit handling that does not extend Blablador's blocks: no retry of HTTP 429 by default, and an opt-in wait for the time the server states
- a `blablador` command: `models`, `chat`, `transcribe`

Status: early. Requires Python 3.10 or newer. The client overrides internals of the `openai` SDK, so each release pins one minor version of `openai` (0.2.0: `openai>=3.42,<3.43`); the pin is raised once the tests pass with a newer release.

## Install

```bash
pip install blablador
```

This installs the library and the `blablador` command. The development version installs from the [repository](https://gitlab.jsc.fz-juelich.de/sdlaml/agents-on-hpc/blablador): `pip install "git+https://gitlab.jsc.fz-juelich.de/sdlaml/agents-on-hpc/blablador.git"`.

If you use [uv](https://docs.astral.sh/uv/), add it to your project, or install only the command, available everywhere:

```bash
uv add blablador
uv tool install blablador
uv add "blablador @ git+https://gitlab.jsc.fz-juelich.de/sdlaml/agents-on-hpc/blablador.git"   # development version
```

## Getting started

You need an API key: a [Helmholtz Codebase token](https://codebase.helmholtz.cloud/-/user_settings/personal_access_tokens) with the `read_user` scope (sign in with your institution's account). Tokens are valid for at most a year. Put it in the environment:

```bash
export BLABLADOR_API_KEY=...
```

Then use `Blablador` like `openai.OpenAI`, which it is a subclass of:

```python
from blablador import Blablador

client = Blablador()  # reads BLABLADOR_API_KEY
completion = client.chat.completions.create(
    model="alias-fast",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(completion.choices[0].message.content)
```

Streaming, the Responses API, tool calls, embeddings, audio transcription and `with_options()` work as the [OpenAI SDK documentation](https://github.com/openai/openai-python) describes. `AsyncBlablador` is the same for `openai.AsyncOpenAI`.

From the shell:

```bash
blablador chat "What is the Jülich Supercomputing Centre?"
```

## What the package adds

### Models and aliases

Blablador's models change. The `alias-*` names (`alias-fast`, `alias-code`, `alias-large`, ...) stay, and are pointed at newer models as these replace older ones, so use aliases in code and read what is behind them at runtime rather than hardcoding a table.

`client.models.list()` returns `blablador.BlabladorModel`s, which add Blablador's fields to the OpenAI ones:

| Field | Meaning |
|---|---|
| `root` | the underlying model: a Hugging Face name, a path on the server, or `None` |
| `max_model_len` | context length in tokens, if reported |
| `client_count` | number of servers currently serving the model, if reported |
| `is_alias` | `True` for the `alias-*` names |

```python
for model in client.models.list():
    if model.is_alias:
        print(model.id, model.root, model.max_model_len)

client.models.aliases()  # {"alias-fast": "openai/gpt-oss-120b", ...}, one request
```

In chat completions, the response's `model` field holds the model's display name ("01 - GPT-OSS-120b - ..."), not the alias that was sent; embeddings responses name the alias. `client.models.retrieve()` does not work: Blablador answers 404 for a single model. The listing does not say what a model can do (chat, embeddings, rerank, speech); the names are the only hint.

### Reasoning

Reasoning models return their reasoning next to the answer, in a field the OpenAI API does not define and whose name depends on the backend (`reasoning` from vLLM, `reasoning_content` from sglang). `blablador.reasoning()` returns it from a message or a streamed delta, or `None`:

```python
import blablador

message = completion.choices[0].message
print(blablador.reasoning(message))  # how the model got there
print(message.content)  # the answer

for chunk in client.chat.completions.create(..., stream=True):
    if chunk.choices:
        print(blablador.reasoning(chunk.choices[0].delta) or "", end="")
```

### Rerank

`client.rerank.create()` scores documents by relevance to a query (Blablador's `/v1/rerank`), best first. Use it to pick the best few of many candidates, for example after an embedding search:

```python
texts = [
    "sbatch submits a batch script to Slurm",
    "JURECA has A100 GPUs",
    "The cafeteria opens at 11:30",
]
ranked = client.rerank.create(
    model="alias-qwen3-8b-reranking",
    query="How do I submit a job on JURECA?",
    documents=texts,
    top_n=2,  # optional: only the best two
)
for result in ranked.results:  # sorted by relevance_score, highest first
    print(f"{result.relevance_score:.3f}  {texts[result.index]}")
```

`return_documents=True` adds each document's text to its result. `AsyncBlablador` has the same method as a coroutine.

### Requests the gateway rejects

Where the gateway is stricter than the OpenAI API, the client adjusts the request so that code written for OpenAI keeps working:

- Function tools without a `description` (optional for OpenAI, HTTP 400 from Blablador) are sent with `"description": ""`, for chat completions and the Responses API.

### Rate limits

Blablador limits requests per model and user, and blocks a key that exceeds the limit: 60 s for the first violation, doubling with each further one up to an hour. The OpenAI SDK would retry an HTTP 429 at once, which risks a longer block, so this client never retries 429 by default (timeouts, 5xx and connection errors are still retried, `max_retries` times). Instead it raises `blablador.RateLimitError`, a subclass of `openai.RateLimitError`, whose `retry_after` holds the wait in seconds if the server stated one, in a `Retry-After` header or in the error message ("try again in 57 seconds"):

```python
import time
import blablador

try:
    completion = client.chat.completions.create(...)
except blablador.RateLimitError as err:
    time.sleep(err.retry_after or 60)  # then try again, once
```

For batch jobs, the client can do the waiting itself:

```python
client = blablador.Blablador(rate_limit_retries=3, max_rate_limit_wait=600)
```

A request that gets a 429 then waits the stated time (plus a second) and is sent again, up to `rate_limit_retries` times. It is not retried, and the error is raised, when the server did not say how long to wait, since a guessed wait that ends too early may extend the block, or when the wait is longer than `max_rate_limit_wait` seconds (default 3600). The client records every stated block, also one it does not wait out. While the key is blocked, other requests through the same client, from any thread, and through its `with_options()` copies wait for the block to end instead of being sent into it, or raise `RateLimitError` at once if the rest of the block is longer than `max_rate_limit_wait`. When the block ends, all waiting requests are sent together, so with many threads the client may exceed the limit again; it does not pace requests. A request whose upload cannot be read again (a stream that cannot seek back) is not retried. This applies only to clients with `rate_limit_retries` set: a copy made with `rate_limit_retries=0`, like a plain client, sends every request. Clients created separately, and other processes using the same key, do not know about the block.

How Blablador states the wait in a 429 has not been confirmed yet (no real 429 has been recorded, since that would block the key). If it uses a form the client does not recognise, `retry_after` is `None` and `rate_limit_retries` raises instead of waiting.

## Command line

The `blablador` command covers the quick uses: see which models are there, ask one question, transcribe a recording. It reads the API key from `BLABLADOR_API_KEY`, like the client, and every command sends one request. `python -m blablador` works too, `blablador COMMAND --help` lists the options, and `blablador --version` prints the versions of `blablador` and `openai` for bug reports.

```bash
blablador models                      # aliases, context length, the model behind each
blablador models --all --json         # every model, as JSON
blablador chat "What is JSC?"         # streams the answer
cat notes.txt | blablador chat "Summarize these notes"   # piped text follows the prompt
blablador chat --reasoning -m alias-code "Is 91 prime?"  # reasoning on stderr, answer on stdout
blablador transcribe meeting.m4a -l en                   # speech to text
```

### `blablador models`

Lists the `alias-*` names with their context length and the model behind each, aliases only unless `--all` is given.

| Option | Meaning |
|---|---|
| `--all` | also list the full model names, after the aliases |
| `--json` | print a JSON list with `id`, `root`, `max_model_len`, `owned_by` and `client_count` per model, for scripts |

### `blablador chat [PROMPT ...]`

Sends one prompt and streams the answer to stdout. The words of the prompt need no quotes. When stdin is not a terminal, it is read to its end and appended after the prompt, separated by a blank line, so `PROMPT` can be left out when the whole prompt is piped. There is no conversation: each call is a new chat.

Under `srun`, or from a script that starts `blablador` with an open stdin, nothing may ever close stdin, and the command would wait for it. After two seconds it says so on stderr; pass `--no-stdin` (or `< /dev/null`) there.

| Option | Meaning |
|---|---|
| `-m`, `--model MODEL` | model or alias; default `$BLABLADOR_MODEL`, else `alias-fast` |
| `-s`, `--system TEXT` | system prompt |
| `--reasoning` | also print the model's reasoning, to stderr, so that `> answer.txt` keeps only the answer |
| `--no-stream` | wait for the complete answer and print it at once |
| `--no-stdin` | do not read stdin, even when it is not a terminal |

### `blablador transcribe FILE`

Prints the text of an audio file (wav, mp3, m4a, ...). Blablador has no `alias-*` name for speech to text, so the default names the model itself; if it is renamed, `blablador models --all` shows the current one.

| Option | Meaning |
|---|---|
| `-m`, `--model MODEL` | speech-to-text model; default `faster-whisper-large-v3` |
| `-l`, `--language CODE` | language of the audio, e.g. `en` or `de`; optional |

### Exit status and errors

The exit status is 0 on success, 1 for a missing API key or an error from the API, 2 for wrong usage (unknown option, missing prompt or file) and 130 after Ctrl-C. Errors are printed to stderr without a traceback, with the same hints as the client (for example, that the key may have expired). The command does not retry rate-limited requests: a 429 is reported, with the wait if the server stated one.

## Examples

- [`examples/quickstart.py`](https://gitlab.jsc.fz-juelich.de/sdlaml/agents-on-hpc/blablador/-/blob/main/examples/quickstart.py): models, chat, streaming, tool calls, rerank and the async client, as sections you can run one at a time: `python examples/quickstart.py [models|chat|stream|tools|rerank|async ...]` (with uv: `uv run python examples/quickstart.py ...`)
- [`examples/agent_loop.py`](https://gitlab.jsc.fz-juelich.de/sdlaml/agents-on-hpc/blablador/-/blob/main/examples/agent_loop.py): a minimal coding agent, one `bash` tool and a loop. It runs the shell commands the model picks without any checks, so use a throwaway directory.

## Configuration

| Variable | Default |
|---|---|
| `BLABLADOR_API_KEY` | required: a [Helmholtz Codebase token](https://codebase.helmholtz.cloud/-/user_settings/personal_access_tokens) with `read_user` scope |
| `BLABLADOR_BASE_URL` | `https://api.blablador.fz-juelich.de/v1` |
| `BLABLADOR_MODEL` | `alias-fast`: the model `blablador chat` uses without `-m` |

Arguments (`Blablador(api_key=..., base_url=...)`) take precedence over the variables. `OPENAI_*` variables are ignored, so an OpenAI setup in the same shell does not send its key or headers to Blablador. When the key is missing or rejected, the error says where to get a new one.

## Development

In a virtual environment, with pip 25.1 or newer (for `--group`):

```bash
pip install -e . --group dev
pytest                     # offline tests, replaying recorded responses in tests/recordings/
pytest -m live             # a few requests against the real API (needs BLABLADOR_API_KEY)
ruff check . && ruff format --check .
python -m tests.record [SCENARIO ...]   # re-record tests/recordings/ from the live API
```

With uv, `uv sync --group dev` sets up the environment, and each command runs with `uv run` in front (`uv run pytest`).

GitLab CI runs ruff and the offline tests on Python 3.10 and 3.15, and installs the built wheel, for every push, merge request and tag; live tests run only locally. Blablador's rate limits are tight and repeated violations block the key for longer each time, so keep live tests and re-recordings to what you need. The recorder strips credentials and internal server paths; check the diff of `tests/recordings/` before committing.

Planned and finished work is tracked with [tatr](https://github.com/tsoding/tatr) in `tasks/`.

## License

MIT, see [LICENSE](https://gitlab.jsc.fz-juelich.de/sdlaml/agents-on-hpc/blablador/-/blob/main/LICENSE).
