LLM Client¶
FlowgentraAI provides a unified LLM client that works with multiple providers.
Creating a Client¶
from flowgentra_ai import LLMConfig, LLMClient
# OpenAI
config = LLMConfig("openai", "gpt-4", api_key="sk-...")
# Anthropic
config = LLMConfig("anthropic", "claude-3-opus-20240229", api_key="sk-ant-...")
# With options
config = LLMConfig(
"openai", "gpt-4",
api_key="sk-...",
temperature=0.7,
max_tokens=1000,
top_p=0.9,
)
client = LLMClient.from_config(config)
Chat¶
from flowgentra_ai import Message
response = client.chat([
Message.system("You are a helpful assistant."),
Message.user("What is Rust?"),
])
print(response.content)
print(response.role) # "assistant"
Token Usage & Cost¶
response, usage = client.chat_with_usage([Message.user("Hello!")])
if usage:
print(f"Prompt tokens: {usage.prompt_tokens}")
print(f"Completion tokens: {usage.completion_tokens}")
print(f"Total tokens: {usage.total_tokens}")
print(f"Estimated cost: ${usage.estimated_cost('gpt-4'):.4f}")
Function Calling (Tools)¶
from flowgentra_ai import ToolDefinition
tools = [
ToolDefinition(
"get_weather",
"Get current weather for a city",
{"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]},
)
]
response = client.chat_with_tools(
[Message.user("What's the weather in Paris?")],
tools,
)
if response.has_tool_calls():
for tc in response.tool_calls():
print(f"Tool: {tc.name}, Args: {tc.arguments}")
Retry, Cache, Fallback¶
# Automatic retry with exponential backoff
retrying = client.with_retry(max_retries=3)
# Response caching
cached = client.cached(max_entries=100)
# Fallback to another provider
backup = LLMClient.from_config(LLMConfig("anthropic", "claude-3-haiku-20240307", api_key="..."))
robust = client.with_fallback(backup)
# Combine them
production = client.with_retry(max_retries=3).cached(max_entries=500)
Structured Output¶
Force the LLM to return structured data:
from flowgentra_ai import ResponseFormat
config = LLMConfig("openai", "gpt-4", api_key="sk-...")
# Force JSON output
config.set_response_format(ResponseFormat.json())
# Force JSON matching a schema
config.set_response_format(ResponseFormat.json_schema("person", {
"type": "object",
"properties": {
"name": {"type": "string"},
"age": {"type": "integer"},
},
"required": ["name", "age"],
}))
Supported Providers¶
| Provider | Config name | Notes |
|---|---|---|
| OpenAI | "openai" |
GPT-4, GPT-3.5, etc. |
| Anthropic | "anthropic" |
Claude models |
| Mistral | "mistral" |
Mistral AI models |
| Groq | "groq" |
Fast inference |
| Ollama | "ollama" |
Local models, no API key needed |
| HuggingFace | "huggingface" |
Inference API |
| Azure | "azure" |
Azure OpenAI Service |
Messages¶
# Create messages
msg = Message.system("You are a helpful assistant")
msg = Message.user("Hello")
msg = Message.assistant("Hi there!")
msg = Message.tool("result_data", tool_call_id="call_123")
# Inspect
msg.role # "system", "user", "assistant", "tool"
msg.content # message text
msg.is_system() # True/False
msg.is_user() # True/False
msg.is_assistant() # True/False
msg.is_tool() # True/False
msg.has_tool_calls() # True if assistant message contains tool calls
msg.tool_calls() # list of ToolCall objects