Batch

Batch trades speed for cost: you send a file of requests, we return the results within 24 hours, and you pay roughly half the normal rate. It suits anything that doesn't need an answer now — bulk classification, backfilling embeddings, evaluating a prompt across thousands of rows.

The API is OpenAI-compatible, so an existing SDK works unchanged.

Check the model supports it

Batch is available on models where we can run it on the provider's own batch tier. GET https://cloud.roar-ai.com/v1/models tells you which, and what it costs:

{
  "id": "gpt-4o-mini",
  "roar_modalities": ["chat"],
  "roar_batch": true,
  "roar_pricing": {
    "batchInputPer1M": 0.075,
    "batchOutputPer1M": 0.30,
    "cachedInputPer1M": 0.0375
  }
}

Sending a batch for a model with "roar_batch": false returns 400 batch_not_supported. Send those requests to /v1/chat/completions instead.

Submit a job

Two steps: upload the requests, then create the batch against the file.

# One JSON object per line. custom_id is how you match results back.
with open("requests.jsonl", "w") as f:
    for i, text in enumerate(documents):
        f.write(json.dumps({
            "custom_id": f"doc-{i}",
            "method": "POST",
            "url": "/v1/chat/completions",
            "body": {
                "model": "gpt-4o-mini",
                "max_tokens": 200,
                "messages": [{"role": "user", "content": f"Summarise: {text}"}],
            },
        }) + "\n")

upload = client.files.create(file=open("requests.jsonl", "rb"), purpose="batch")
batch = client.batches.create(
    input_file_id=upload.id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
)
print(batch.id, batch.status)

Every line is validated before anything is sent upstream, so a typo in line 40,000 is a 400 in seconds rather than a failed job hours later. Each line needs a unique custom_id, "method": "POST", and a url matching the batch's own endpoint.

Collect the results

Poll the batch, then download the output when it completes.

batch = client.batches.retrieve(batch.id)
if batch.status == "completed":
    body = httpx.get(
        f"https://cloud.roar-ai.com/v1/batches/{batch.id}/content",
        headers={"Authorization": f"Bearer {api_key}"},
    ).text
    for line in body.splitlines():
        row = json.loads(line)
        print(row["custom_id"], row["response"]["body"]["choices"][0]["message"]["content"])

Results are JSONL, one line per request, in the same shape whichever model ran the job:

{
  "custom_id": "doc-0",
  "response": {
    "status_code": 200,
    "body": { "object": "chat.completion", "choices": [...], "usage": {...} }
  },
  "error": null
}

A partially failed batch still returns results for the requests that ran; the rest carry error instead of response. You're billed only for what actually ran.

Status values

Cancel with POST https://cloud.roar-ai.com/v1/batches/{id}/cancel, and list your jobs with GET https://cloud.roar-ai.com/v1/batches.

Pricing and limits

The rate is fixed when you submit. If we republish that model's prices while your job is running, your job keeps the rate it was quoted.

stream: true is rejected — results arrive as a file, so streaming has nothing to mean.

If your account has a hard spend cap, a batch is checked against it at submission using an estimate of what it will cost, and that estimate is held against your budget until the job settles. A batch large enough to exceed the cap is refused with 402 rather than discovered halfway through.