Evaluation
Ask an evaluation model typed questions about your data, with a probability on every answer.
POST https://api.roar-ai.com/v1/evaluate is for judging, not writing. You send some state, such as a conversation, a document or another model's answer, plus one or more typed questions about it. The model answers every question in one round trip, each with a probability. Typical uses are grading other models' output, guardrail checks, ticket triage and routing.
Evaluation models are listed in GET https://api.roar-ai.com/v1/models with "roar_modalities": ["evaluation"]. Today that is jev, which has a 64K-token context and typically answers in well under a second.
This is not an OpenAI-compatible endpoint: there is no messages field and no completion. Call it with plain HTTP. Sending an evaluation model to /v1/chat/completions, or a chat model here, returns 400.
The request
curl https://api.roar-ai.com/v1/evaluate \
-H "Authorization: Bearer $ROAR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "jev",
"state": "Customer: my card was charged twice for order 1182.\nAgent: I have refunded the duplicate charge.",
"questions": {
"resolved": {
"type": "boolean",
"instructions": "Did the agent resolve the customer'\''s problem?"
},
"topic": {
"type": "choice",
"instructions": "What is this conversation about?",
"criteria": {
"billing": "Charges, refunds or invoices",
"delivery": "Shipping or tracking",
"other": null
}
},
"tone": {
"type": "score",
"instructions": "How polite was the agent?",
"criteria": ["rude", "neutral", "polite"]
}
}
}'
| Field | Type | Notes |
|---|---|---|
model | string | An evaluation model, e.g. jev. |
state | string, object or array | What the questions are about. An object or array is serialised for you. |
questions | object | At least one. Each key is your name for the question and comes back as the key of its answer. |
Each question has a type and, depending on the type, instructions (the question in words) and criteria:
type | criteria | Required |
|---|---|---|
boolean | Optional { "true": "...", "false": "..." }. Either side may be given alone. | instructions, criteria, or both |
choice | Object mapping each option's name to its description (null if the name says it all). At least two options. | criteria |
score | Array of rungs, lowest first. At least two. | criteria |
instructions and criteria values may be strings or objects. A malformed request gets a 400 whose message names each field at fault by its path, such as questions.topic.criteria.
The response
{
"answers": {
"resolved": { "type": "boolean", "probability": 0.93 },
"topic": {
"type": "choice",
"choice": "billing",
"probabilities": { "billing": 0.97, "delivery": 0.01, "other": 0.02 }
},
"tone": {
"type": "score",
"score": 1.86,
"probabilities": { "rude": 0.01, "neutral": 0.12, "polite": 0.87 }
}
},
"usage": { "inputTokens": 96, "outputTokens": 0 }
}
boolean:probabilitythat the answer is true.choice: the most likely option, plus a probability for each option.score: a value interpolated across the rungs, plus a probability per rung.
The probabilities are useful but not guaranteed to be consistent. Treat one near 0.5 as "not sure" and send those cases to a person or a larger model.
Billing and limits
Evaluation is billed on input tokens only: the state plus the questions. The rate is on the model's page and in GET /v1/models. Requests per minute, budgets and the error envelope are the same as for every other endpoint; see Limits, budgets & errors. If the state and questions together are longer than the model accepts, the error says so and asks you to send less.
Each call appears in your request log like any other. When your account keeps request content, the stored copy holds the state, the questions and the answers you received.
Repeated questions are not cached unless you ask
Response caching is off by default. Two identical evaluation requests are two billed calls. If you really do send the same state and questions again, for example re-grading a fixed test set, opt in per request with a header that gives the window in seconds (up to 3600):
-H "x-roar-cache-ttl: 600"
A repeat inside the window is answered from the cache and costs nothing. x-roar-cache-ttl: off turns caching off for a request even if your account has a default window set. Accounts pinned to a single country are never cached, whatever the header says.