# Evaluation

> Ask an evaluation model typed questions about your data, with a probability on every answer.

`POST https://api.roar-ai.com/v1/evaluate` is for **judging**, not writing. You send some state, such as a conversation, a document or another model's answer, plus one or more typed questions about it. The model answers every question in one round trip, each with a probability. Typical uses are grading other models' output, guardrail checks, ticket triage and routing.

Evaluation models are listed in `GET https://api.roar-ai.com/v1/models` with `"roar_modalities": ["evaluation"]`. Today that is `jev`, which has a 64K-token context and typically answers in well under a second.

This is **not** an OpenAI-compatible endpoint: there is no `messages` field and no completion. Call it with plain HTTP. Sending an evaluation model to `/v1/chat/completions`, or a chat model here, returns `400`.

## The request

<CodeCard caption="curl">

```bash
curl https://api.roar-ai.com/v1/evaluate \
  -H "Authorization: Bearer $ROAR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "jev",
    "state": "Customer: my card was charged twice for order 1182.\nAgent: I have refunded the duplicate charge.",
    "questions": {
      "resolved": {
        "type": "boolean",
        "instructions": "Did the agent resolve the customer'\''s problem?"
      },
      "topic": {
        "type": "choice",
        "instructions": "What is this conversation about?",
        "criteria": {
          "billing": "Charges, refunds or invoices",
          "delivery": "Shipping or tracking",
          "other": null
        }
      },
      "tone": {
        "type": "score",
        "instructions": "How polite was the agent?",
        "criteria": ["rude", "neutral", "polite"]
      }
    }
  }'
```

</CodeCard>

| Field | Type | Notes |
| --- | --- | --- |
| `model` | string | An evaluation model, e.g. `jev`. |
| `state` | string, object or array | What the questions are about. An object or array is serialised for you. |
| `questions` | object | At least one. Each **key** is your name for the question and comes back as the key of its answer. |

Each question has a `type` and, depending on the type, `instructions` (the question in words) and `criteria`:

| `type` | `criteria` | Required |
| --- | --- | --- |
| `boolean` | Optional `{ "true": "...", "false": "..." }`. Either side may be given alone. | `instructions`, `criteria`, or both |
| `choice` | Object mapping each option's name to its description (`null` if the name says it all). At least **two** options. | `criteria` |
| `score` | Array of rungs, **lowest first**. At least **two**. | `criteria` |

`instructions` and criteria values may be strings or objects. A malformed request gets a `400` whose message names each field at fault by its path, such as `questions.topic.criteria`.

## The response

<CodeCard caption="Response">

```json
{
  "answers": {
    "resolved": { "type": "boolean", "probability": 0.93 },
    "topic": {
      "type": "choice",
      "choice": "billing",
      "probabilities": { "billing": 0.97, "delivery": 0.01, "other": 0.02 }
    },
    "tone": {
      "type": "score",
      "score": 1.86,
      "probabilities": { "rude": 0.01, "neutral": 0.12, "polite": 0.87 }
    }
  },
  "usage": { "inputTokens": 96, "outputTokens": 0 }
}
```

</CodeCard>

- **`boolean`**: `probability` that the answer is true.
- **`choice`**: the most likely option, plus a probability for each option.
- **`score`**: a value interpolated across the rungs, plus a probability per rung.

The probabilities are useful but not guaranteed to be consistent. Treat one near 0.5 as "not sure" and send those cases to a person or a larger model.

## Billing and limits

Evaluation is billed on **input tokens only**: the state plus the questions. The rate is on the model's page and in `GET /v1/models`. Requests per minute, budgets and the error envelope are the same as for every other endpoint; see [Limits, budgets & errors](/docs/limits). If the state and questions together are longer than the model accepts, the error says so and asks you to send less.

Each call appears in your request log like any other. When your account keeps request content, the stored copy holds the state, the questions and the answers you received.

## Repeated questions are not cached unless you ask

Response caching is **off by default**. Two identical evaluation requests are two billed calls. If you really do send the same state and questions again, for example re-grading a fixed test set, opt in per request with a header that gives the window in seconds (up to 3600):

<CodeCard caption="Opt in to caching for one request">

```bash
-H "x-roar-cache-ttl: 600"
```

</CodeCard>

A repeat inside the window is answered from the cache and costs nothing. `x-roar-cache-ttl: off` turns caching off for a request even if your account has a default window set. Accounts pinned to a single country are never cached, whatever the header says.