AI Gateway

Chat & streaming

Chat completions, SSE streaming, tool calling and vision.

POST https://cloud.roar-ai.com/v1/chat/completions is drop-in OpenAI-compatible: messages in, completion out. The gateway picks the best backend for the model under your routing policy, fails over automatically if one is unhealthy, and meters every request.

Streaming

Set "stream": true to receive server-sent events. The final data frame before [DONE] carries token usage.

Python (streaming)

stream = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[{"role": "user", "content": "Write a haiku"}],
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="")

Tool calling

Pass tools (and optionally tool_choice) exactly as you would with OpenAI. Tool calls come back as tool_calls with finish_reason: "tool_calls" — including over streaming — regardless of which backend served the request. Return results as role: "tool" messages.

Request excerpt

{
  "model": "claude-sonnet",
  "messages": [{"role": "user", "content": "Weather in Oslo?"}],
  "tools": [{
    "type": "function",
    "function": {
      "name": "get_weather",
      "description": "Current weather by city",
      "parameters": {
        "type": "object",
        "properties": {"city": {"type": "string"}},
        "required": ["city"]
      }
    }
  }]
}

Vision

Models that support image input accept OpenAI-style content parts — a data URL or a public image URL:

Image content part

{"role": "user", "content": [
  {"type": "text", "text": "What's in this picture?"},
  {"type": "image_url", "image_url": {"url": "https://example.com/cat.png"}}
]}

Legacy POST /v1/completions (prompt in, text out) is also supported for older integrations.