Metadata-Version: 2.4
Name: flexinference
Version: 1.6.2
Summary: Official Python SDK for FlexInference - a deadline-aware, OpenAI-compatible inference router.
Project-URL: Homepage, https://flexinference.com
Project-URL: Documentation, https://flexinference.mintlify.app
Author-email: Aditya Perswal <adityaperswal@gmail.com>
License: MIT
License-File: LICENSE
Keywords: ai,flexinference,gpt,inference,llm,openai
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Typing :: Typed
Requires-Python: >=3.12
Requires-Dist: httpx>=0.27
Requires-Dist: typing-extensions>=4.12
Provides-Extra: dev
Requires-Dist: mypy>=1.13; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff>=0.7; extra == 'dev'
Description-Content-Type: text/markdown

# FlexInference (Python)

The official Python SDK for [FlexInference](https://flexinference.com). FlexInference is an inference router that works with **OpenAI, Google Gemini, and Anthropic**. You send the OpenAI-shaped requests you already send, you bring your own provider key, and you set one required field called `start_within`. That field is how long you will wait for the request to start. We try a cheaper tier first to lower your bill, and if it cannot start in time we escalate to your standard tier so the request still runs. The first 100,000 requests each month are free; a $10/month subscription raises the allowance to 10,000,000. The SDK speaks four caller formats: `responses`, `chat.completions`, `interactions` (Gemini shape), and `messages` (Anthropic shape). Any of them reaches any provider.

```bash
pip install flexinference
```

## Quickstart

```python
from flexinference import FlexInference, output_text

client = FlexInference(api_key="flex_live_...")

res = client.responses.create({
    "model": "gpt-5.5",
    "input": "Write a haiku about cheap GPUs.",
    "start_within": "00h-00m-30s",
})

print(output_text(res))
```

Responses come back as the **raw OpenAI JSON** and we never reshape the body. That means there is no `output_text` field on the wire, because OpenAI's own SDKs compute that field rather than the provider. `output_text(res)` pulls the assistant's text out of either a response or a chat completion for you.

`start_within` is **required** on every request. Set it to `"default"`, `"priority"`, `"auto"`, or a duration written as `"HHh-MMm-SSs"` from 5s to 10m. A duration is how long you will wait for the request to start running. We try OpenAI's cheaper flex tier first on a flex-capable model, and that is where your savings come from. If flex cannot start inside your window, we switch to your normal standard tier so the request still completes. The words `"default"`, `"priority"`, and `"auto"` map straight to those OpenAI service tiers and work with any model. See the [docs](https://flexinference.mintlify.app/deadline-routing).

This escalation is your safety net. You never lose a request just because the cheap tier was busy. Your standard tier always finishes the job. It runs the same model either way, so you trade a little waiting for a lower bill and keep the result you would have gotten anyway.

## Providers (OpenAI, Gemini, and Anthropic)

FlexInference routes to **OpenAI**, **Google Gemini**, and **Anthropic**. Send the same OpenAI-shaped request and pass whichever model id you want, such as `gpt-5.5`, `o4-mini`, `gemini-3.5-flash`, or `claude-opus-4-8`. We translate Gemini and Anthropic to and from the OpenAI shape, so your code is identical for all three.

- **OpenAI:** `default` (standard tier), `priority`, `auto`, and the flex race (a duration) on flex-capable models.
- **Gemini:** `default` maps to Gemini's **standard** tier, plus `priority` and the flex race on the Gemini flex models (`gemini-3.5-flash`, `gemini-3.1-flash-lite`, `gemini-3.1-pro-preview`, `gemini-3-flash-preview`, `gemini-2.5-pro`, `gemini-2.5-flash`, `gemini-2.5-flash-lite`). Gemini has no `auto` tier, so `start_within="auto"` on a Gemini model returns `400`.
- **Anthropic (Claude):** proxy-only. `default`, `priority`, and `auto` work; there is **no flex race**, so a duration `start_within` on a `claude-*` model returns `400 flex_unsupported_for_anthropic`. Anthropic requires an output-token field upstream. FlexInference forwards `max_output_tokens` (`max_completion_tokens` on Chat, `max_tokens` on Messages) when set and does not synthesize a default, so omitting it returns Anthropic's own request error. You keep the unified API and tier control, and draw down your own Anthropic credits.

Add the provider key you'll use (OpenAI, Gemini, and/or Anthropic) in the [dashboard](https://www.flexinference.com/dashboard). Text, streaming, structured outputs, function calling, image input, and web search work across the `openai`, `google`, and `anthropic` direct routes (send a Responses `web_search` tool; we map it to Gemini's `google_search`).

Pin the route or set a same-model fallback chain with the optional `provider` array (for example `["openai"]` or `["google", "vertex"]`): element 0 is the primary route and the sole flex gate, and later entries are tried in order. A malformed `provider` array raises a `ValueError` locally before any network call. All five routes serve: `google`, `openai`, and `anthropic` are direct routes, and the cloud routes `bedrock` (`claude-*`, via Amazon Bedrock) and `vertex` (`gemini-*`, via Google Vertex AI) serve the same model using your own cloud key. Cloud routes run at the standard tier only, with no flex. The `vertex` route serves text; keep cloud requests text-focused.

Don't send `service_tier`. The router picks the tier from `start_within`, so a request that sets its own `service_tier` fails fast with `400 service_tier_not_allowed`.

## Streaming

```python
stream = client.responses.create(
    {"model": "gpt-5-nano", "input": "Count to ten.", "start_within": "00h-00m-20s"},
    stream=True,
)
for event in stream:
    if event.get("type") == "response.output_text.delta":
        print(event["delta"], end="")
```

## Chat Completions

```python
res = client.chat.completions.create({
    "model": "gpt-5.5",
    "messages": [{"role": "user", "content": "Hello!"}],
    "start_within": "default",
})
print(res["choices"][0]["message"]["content"])
```

## Interactions (Gemini shape)

Speak Google's Interactions shape and reach any model. `interaction_output_text(res)` pulls the assistant text out of the interaction's `steps`.

```python
from flexinference import interaction_output_text

res = client.interactions.create({
    "model": "gemini-3.5-flash",
    "input": "Summarize this contract.",
    "start_within": "00h-01m-00s",
})
print(interaction_output_text(res))
```

## Messages (Anthropic shape)

Speak Anthropic's Messages shape and reach any model. Anthropic requires `max_tokens` upstream. FlexInference forwards it when set and does not synthesize a default, so omitting it returns Anthropic's own request error. `message_output_text(res)` pulls the assistant text out of the message `content`.

```python
from flexinference import message_output_text

res = client.messages.create({
    "model": "claude-opus-4-8",
    "max_tokens": 1024,
    "messages": [{"role": "user", "content": "Summarize this contract."}],
    "start_within": "default",
})
print(message_output_text(res))
```

## Closing the client

The client holds a pooled `httpx.Client`, so close it when you're done to release connections. Use it as a context manager:

```python
with FlexInference(api_key="flex_live_...") as client:
    res = client.responses.create({"model": "gpt-5.5", "input": "Hi.", "start_within": "default"})
    print(output_text(res))
# connections are released on exit
```

Or close it yourself:

```python
client = FlexInference(api_key="flex_live_...")
try:
    ...
finally:
    client.close()
```

## Timeouts

Streaming and non-streaming requests are timed differently, so a long generation is never cut off yet a hung request never hangs forever:

- **Non-streaming** requests bound only the body read with `timeout`, default **600s** - the read after the first response has arrived, not a total wall-clock budget (the worst case is roughly `connect_timeout` + `first_byte_timeout` + `timeout`).
- **Streaming** requests have **no total cap** (a healthy stream runs as long as tokens keep arriving). A stalled stream is caught by an idle watchdog: if no chunk arrives within `idle_timeout` (default **60s**), the request is aborted with a typed `FlexInferenceError` (code `idle_timeout`).
- Both wait up to `first_byte_timeout` (default **60s**) for the first response, and `connect_timeout` (default **10s**) to reach the router. The first-byte wait is **auto-raised for a flex `start_within`** (the router withholds headers until the race resolves), so a long deadline just works.

```python
client = FlexInference(
    api_key="flex_live_...",
    timeout=120.0,  # non-streaming total budget
    first_byte_timeout=90.0,  # wait for the first response
    idle_timeout=30.0,  # max silence between streamed chunks
)
```

Any of these can be overridden per request:

```python
res = client.responses.create(
    {"model": "gpt-5.5", "input": "Hi.", "start_within": "priority"},
    timeout=30.0,
)

for event in client.responses.create(
    {"model": "gpt-5.5", "input": "Count to ten.", "start_within": "priority"},
    stream=True,
    idle_timeout=15.0,
):
    ...
```

A custom `httpx.Client` is still accepted for transport, auth, or SSL config; the timeouts above are applied per request and override the client's own timeout.

## Request validation

Before a request leaves your machine, the SDK validates the parts it owns. `start_within` is **required** and must be `"default"`, `"priority"`, `"auto"`, or a duration `"HHh-MMm-SSs"` between 5s and 10m; `model` and `input`/`messages` must be present. A missing or bad value raises a `ValueError` locally instead of making a round trip to a provider 400:

```python
client.responses.create({"model": "gpt-5.5", "input": "hi"})
# ValueError: Invalid request body:
#   Missing required parameter: `start_within`. Set it to "default", "priority", "auto", or a duration "HHh-MMm-SSs".
```

Validation is request-only. Unknown fields pass straight through to the provider (so new OpenAI parameters keep working), and responses are never validated or reshaped.

## Retry

Pass an optional `retry` and the router re-hits the upstream on a transient transport failure,
so a request that would otherwise error because the tier was briefly down instead quietly
succeeds:

```python
res = client.responses.create({
    "model": "gpt-5.5",
    "input": "Write a haiku about cheap GPUs.",
    "start_within": "00h-00m-30s",
    "retry": {"count": 3},  # up to 3 re-hits; backoff "exponential", jitter True
})
```

`retry` is `{"count", "backoff"?, "jitter"?}`. `count` is required and sets how many times the
router re-hits the settled tier (1 to 5). `backoff` is `"exponential"` (the default) or
`"linear"`, and `jitter` (default `True`) spreads each delay so clients don't retry in lockstep.
The object is validated locally before any network call -- a missing or out-of-range `count`, a
bad `backoff`, a non-boolean `jitter`, or an unknown sub-key raises a `ValueError`.

Retry fires only on a genuine `429` or `5xx`, and only before the response commits (its status
line, or the first streamed byte); it honors the provider's `Retry-After` first (clamped to
60s). A client error, an auth failure, an upstream timeout, or a stream that has already started
is never retried, and only the tier the router settled on is re-hit (never the flex leg during a
flex race). The response carries an `x-flexinference-retries` header with the count the router
spent, so if your own client also retries you can turn it down to avoid multiplying attempts.

## Errors

Non-2xx responses raise `FlexInferenceError`, carrying `status`, `type`, `code`, and `param`. The router shapes error bodies to match the endpoint you called (OpenAI on `responses`/`chat`, Anthropic on `messages`, Google on `interactions`) so the SDK you would use for that surface parses them, and `FlexInferenceError` reads all three. `message` and `status` are always set; `code`, `param`, and `doc_url` are populated on the OpenAI surface.

```python
from flexinference import FlexInferenceError

try:
    client.responses.create({"model": "gpt-5.5", "input": "hi", "start_within": "priority"})
except FlexInferenceError as err:
    if err.code == "no_byok_key":
        print("Add your OpenAI key in the dashboard.")
    else:
        raise
```

Every FlexInference error tells you the same four things. It says what went wrong, why it went wrong, how to fix it, and it shows an example of a request that works. This is built for agents as much as for people. An agent can read the message and correct the call instead of guessing and burning tokens. Provider errors are reshaped into the same surface envelope and normalized into `FlexInferenceError`, so you get one consistent error type no matter which model ran.

## Billing / 402

The first **100,000 requests each month are free** - bring your own provider key, no card
required. A **$10/month subscription** raises the monthly allowance to **10,000,000 requests**.
Past that, buy overage at **$1 per extra 1,000,000 requests**. Pre-buy a block, or turn on
auto-charge to top up automatically up to a cap you set. The allowance counts every request,
not just flex.

When you run out of allowance, the router returns `402 Payment Required` on every request until
the allowance resets on the 1st of the month (UTC) or you buy more. The SDK raises a typed
`PaymentRequiredError` (a subclass of `FlexInferenceError`) for HTTP 402, and its `.code` tells
you which case you hit and what to do:

- **`free_quota_exceeded`** - a free org used its 100,000 requests for the month. Subscribe
  ($10 for 10,000,000/month) or wait for the 1st-of-month UTC reset.
- **`quota_exceeded`** - a subscription used its 10,000,000 requests plus any purchased overage,
  with auto-charge off. Buy more requests or enable auto-charge, or wait for the reset.
- **`payment_required`** - an auto-charge overage payment failed. Update the card on the Billing
  page to clear it.

```python
from flexinference import PaymentRequiredError

try:
    client.responses.create({"model": "gpt-5.5", "input": "hi", "start_within": "00h-00m-30s"})
except PaymentRequiredError as err:
    if err.code == "free_quota_exceeded":
        print("Free monthly allowance used up - subscribe or wait for the 1st-of-month reset.")
    elif err.code == "quota_exceeded":
        print("Monthly allowance used up - buy more requests or enable auto-charge.")
    else:  # payment_required
        print("Auto-charge payment failed - update your card on the Billing page.")
except FlexInferenceError:
    raise
```

Because `PaymentRequiredError` subclasses `FlexInferenceError`, existing
`except FlexInferenceError` handlers keep catching 402s too.

## Configuration

| Argument             | Default                            | Description                                                                                  |
| -------------------- | ---------------------------------- | -------------------------------------------------------------------------------------------- |
| `api_key`            | (required)                         | Your `flex_live_` key.                                                                       |
| `base_url`           | `https://api.flexinference.com/v1` | Override the router endpoint.                                                                |
| `client`             | new `httpx.Client`                 | Provide your own `httpx.Client` (transport/auth/SSL); its timeout is overridden per request. |
| `timeout`            | `600.0`                            | Non-streaming body-read budget in seconds, after headers (streaming has no total).           |
| `first_byte_timeout` | `60.0`                             | Wait for the first response in seconds; auto-raised for a flex `start_within`.               |
| `idle_timeout`       | `60.0`                             | Max silence between streamed chunks (seconds) before a stream is treated as hung.            |
| `connect_timeout`    | `10.0`                             | Wait to reach the router in seconds.                                                         |

`timeout`, `first_byte_timeout`, and `idle_timeout` can also be passed per `create()` call.

## License

MIT
