Qwen 3.8 27B Benchmark: Colombo vs Singapore

Qwen 3.8 27B is an open-weight model, so the same model is available from several providers. We serve it from two places: a major cloud provider's region in Singapore, and our own infrastructure in Colombo, available as Qwen 3.8 27B (Colombo). This post compares the two on accuracy, latency, throughput and reliability for traffic originating in Sri Lanka.

Test setup

Deployments

Both deployments are reached through the same Roar AI gateway in Colombo. A request to Colombo stays within the city. A request to Singapore leaves the gateway over the public internet. We have no visibility into the Singapore deployment's hardware, serving configuration or scheduler, which limits how far any accuracy difference can be attributed to either side.

Benchmark

• Dataset: MMLU-Pro, test split. Ten answer options per question, so chance accuracy is 10%. We took the first 25 questions from each of ten subjects, 250 in total.
• Request: POST /v1/chat/completions with stream: true, reasoning_effort: "low" and max_tokens: 8000, a single user message, no system prompt, and default sampling parameters.
• Scoring: the last answer is (X) in the answer text. A response with no parsable answer, or a failed request, is scored as incorrect.
• Load: eight concurrent requests per deployment, sent from a client in Sri Lanka on 15 October 2026. The two runs were sequential, about ten minutes each.
• Metrics: latency is measured at the client. Time to first token (TTFT) is the time to the first streamed token, including reasoning tokens. Throughput is completion tokens, reasoning included, divided by the time from the first token to the end of the stream. Percentiles are computed over completed requests with linear interpolation.

The prompt:

The following is a multiple choice question about {subject}. Think step by step, then
finish your answer with "The answer is (X)" where X is the letter of the correct option.

Question: {question}

Options:
A. {option A}
B. {option B}
...

Results

Accuracy

The headline gap of 6.4 points includes Singapore's 10 failed requests, which are scored as incorrect. On the 240 questions both deployments completed, Colombo scored 82.5% and Singapore 78.8%, a gap of 3.7 points.

The two deployments gave the same answer on 211 of those 240 questions. Where exactly one was correct, Colombo was correct on 15 and Singapore on 6. An exact McNemar test on that split gives p ≈ 0.08. The direction is consistent, but this sample does not establish a difference in accuracy at conventional significance levels.

Unanswered responses were similar on both sides: 5 in Colombo and 8 in Singapore produced no parsable answer, of which 4 and 6 respectively reached the 8,000-token limit while still reasoning.

Colombo scored equal or higher in all ten subjects. The largest differences were in law (16 against 11 correct) and in health and mathematics (three questions each). With 25 questions per subject, a difference of one or two questions is not meaningful on its own.

Latency

Colombo's TTFT was 0.19 s lower at the median and 0.36 s lower at P90. The median difference is consistent with the additional network round trip to Singapore. The wider gap at P90 points to queueing or load variation on the shared Singapore deployment.

The distribution shows the same pattern across the full range: Colombo's curve sits to the left throughout, and its tail is shorter, with P99 at 1.30 s against 2.71 s.

TTFT here includes reasoning tokens. Because the model reasons before it answers, the first answer token arrived after about 9 s on both deployments, and median total response time differed by less than a second. Where TTFT matters is in interactive applications that stream reasoning or partial output, and in agent loops that make many sequential calls.

Throughput

Median per-request throughput was similar, at 63.7 tok/s in Colombo and 60.2 tok/s in Singapore. The difference is in consistency. Colombo's P10–P90 range was 2.9 tok/s wide; Singapore's was 20.6 tok/s, with the slowest request at 22.2 tok/s. This is consistent with dedicated capacity serving one model compared with shared capacity, and it contributes to Singapore's P90 total response time being 8 s longer.

Reliability

All 250 Colombo requests completed. Of the 10 Singapore requests that failed, 7 ended with an error from the provider mid-stream, 2 with a connection reset and 1 with a timeout. Our gateway returned each one to the client as an error. In an application, these would surface as retries or failed responses.

Configuration and tuning

Both deployments serve the same model. It was not retrained or fine-tuned for this comparison.

We selected the Colombo deployment's default reasoning configuration using an earlier 250-question MMLU-Pro run on the same deployment. In that run, low scored 84.0% using about a third of the tokens of the model's default reasoning mode, which scored 81.6% with 17 responses cut off by the token limit. That run very likely drew on an overlapping set of questions.

Both deployments received the same reasoning_effort: "low" request in this test. Each inference stack maps that parameter to its own behaviour, and we cannot see how the Singapore deployment applies it. Other serving settings may also differ between the two.

The accuracy result should therefore be read as a measurement of our configuration on MMLU-Pro. It is not evidence of a general difference in model quality between the two deployments.

Cost

The table below uses Roar AI's published API prices, the rates a developer pays for each model ID. They are not our infrastructure costs.

Singapore's token counts cover the 240 completed requests, because failed requests did not report usage. Colombo completed 10 more requests and used about 5% more output tokens per request, and the run still cost 40% less because its per-token price is lower. Published prices on Roar AI are fixed per model. They do not change with the provider's own pricing.

Limitations

• Sample size. One run of 250 questions per deployment, on one day. The accuracy difference is not statistically significant, and a repeat run could move each figure by several points.
• Benchmark-specific tuning. Our reasoning configuration was selected on MMLU-Pro, as described above.
• Opaque comparison deployment. Hardware, serving configuration and scheduling in Singapore are not disclosed, and load on a shared deployment varies by time of day.
• One client location. Latency was measured from Sri Lanka only. From elsewhere in the region, the latency comparison would differ.
• Failover. The Colombo deployment has no failover outside Sri Lanka, by design. During maintenance, requests to it fail rather than being routed abroad.

Reproducing this benchmark

The raw per-request results for both deployments are published as qwen3.8-27b-colombo-singapore-2026-10-15.json. Each row includes the MMLU-Pro question ID, subject, expected and parsed answers, TTFT, time to first answer token, total time and token counts. Every chart and figure in this post is computed from that file. To repeat the test against your own key, send the prompt above with the listed request parameters to either model ID.

Conclusion

For developers building AI applications in Sri Lanka, this benchmark points to four practical results. A locally hosted deployment of Qwen 3.8 27B performed at least as well as the same model served from Singapore on MMLU-Pro, bearing in mind that our configuration was tuned for it. It returned the first token sooner, with a shorter tail. Its throughput was consistent from request to request. And it costs less per token at a fixed published price, with requests processed entirely within Sri Lanka.

A single benchmark does not establish general superiority, and the right deployment depends on the workload. For latency-sensitive applications, agents making many sequential calls, and data that must remain in the country, local inference is a credible option rather than a compromise. Both deployments are available through the same API. See Qwen 3.8 27B (Colombo) or learn more about the gateway.