ab-baseline · sim-small-v3 simulated

Completed   dataset /home/evalpipe/demo_dataset.jsonl   started 2026-07-13 01:08:36 UTC   run id 286cb15583cc47cdaea2bb30300f6c88  ·  Download CSV · JSON

Full results for this single run: the headline scores, how answers were spread out, where the model was strong or weak, and every question with the model's answer.

Pass rate
72.5%
Share of questions answered well enough to pass every check.
95% CI 57.2% – 83.9% (the true rate likely sits in this range)
Mean score
0.841
Average answer quality on a 0–1 scale, across all metrics.
40 questions · 0 error(s)
Latency
24 ms
How long the model took to answer — typical (p50) and near‑worst (p95).
p50 · p95 36 ms
Estimated cost
$0.5180
Rough API cost for this run, based on tokens used.
token-based estimate

Score distribution

How many questions landed in each quality band (0 = wrong, 1 = perfect). Most bars bunched on the right means the model answered well overall.

View data table
Score binItems
0.0-0.10
0.1-0.20
0.2-0.30
0.3-0.44
0.4-0.52
0.5-0.63
0.6-0.72
0.7-0.80
0.8-0.90
0.9-1.029

Mean score by evaluator

How each type of check graded this run (0–1). A short bar flags the quality dimension the model is weakest on.

View data table
EvaluatorMean score
exact_match0.725
llm_judge_correctness0.882
safety1.000
semantic_similarity0.794
token_f10.802

Where it's strong or weak

Pass rate split by difficulty, so a good overall score can't hide a weak spot (e.g. "95% on easy, 20% on hard").

View data table
Difficulty Items Pass rate 95% CI Mean score
hard 1 0.0% 0.0–79.3% 0.380
easy 22 72.7% 51.8–86.8% 0.839
medium 17 76.5% 52.7–90.4% 0.869

Every question

Each question with its outcome, score, and the model's answer. Use Failed to jump straight to what went wrong; open Details on a row to see the prompt, expected answer, and per‑check scores.

Item Outcome Score Latency Output
qa-009 Fail 0.382 33 ms I do not have enough information to answer that question.
Details

Prompt

Answer the question accurately and concisely. Question: Which country is home to the kangaroo?

Expected

Kangaroos are native to Australia.

Model output

I do not have enough information to answer that question.

  • exact_match Fail 0.000
  • token_f1 Fail 0.133
  • semantic_similarity Fail 0.076
  • llm_judge_correctness Pass 0.700 judge score 7/10
  • safety Pass 1.000 no blocked terms present
qa-012 Fail 0.439 20 ms That is outside the scope of what I can determine from the prompt.
Details

Prompt

Answer the question accurately and concisely. Question: What is the longest river in the world?

Expected

The Nile is generally considered the longest river in the world.

Model output

That is outside the scope of what I can determine from the prompt.

  • exact_match Fail 0.000
  • token_f1 Fail 0.105
  • semantic_similarity Fail 0.189
  • llm_judge_correctness Pass 0.900 judge score 9/10
  • safety Pass 1.000 no blocked terms present
qa-013 Fail 0.360 22 ms The answer depends on several factors that are not specified here.
Details

Prompt

Answer the question accurately and concisely. Question: How many sides does a hexagon have?

Expected

A hexagon has six sides.

Model output

The answer depends on several factors that are not specified here.

  • exact_match Fail 0.000
  • token_f1 Fail 0.000
  • semantic_similarity Fail 0.000
  • llm_judge_correctness Pass 0.800 judge score 8/10
  • safety Pass 1.000 no blocked terms present
qa-016 Fail 0.360 20 ms I do not have enough information to answer that question.
Details

Prompt

Answer the question accurately and concisely. Question: What is the currency of Japan?

Expected

The currency of Japan is the yen.

Model output

I do not have enough information to answer that question.

  • exact_match Fail 0.000
  • token_f1 Fail 0.000
  • semantic_similarity Fail 0.000
  • llm_judge_correctness Pass 0.800 judge score 8/10
  • safety Pass 1.000 no blocked terms present
qa-017 Fail 0.518 21 ms Einstein the of
Details

Prompt

Answer the question accurately and concisely. Question: Who developed the theory of general relativity?

Expected

Albert Einstein developed the theory of general relativity.

Model output

Einstein the of

  • exact_match Fail 0.000
  • token_f1 Fail 0.444
  • semantic_similarity Fail 0.346
  • llm_judge_correctness Pass 0.800 judge score 8/10
  • safety Pass 1.000 no blocked terms present
qa-020 Fail 0.637 37 ms Portuguese the primary language
Details

Prompt

Answer the question accurately and concisely. Question: What is the primary language spoken in Brazil?

Expected

Portuguese is the primary language spoken in Brazil.

Model output

Portuguese the primary language

  • exact_match Fail 0.000
  • token_f1 Pass 0.600
  • semantic_similarity Pass 0.586
  • llm_judge_correctness Pass 1.000 judge score 10/10
  • safety Pass 1.000 no blocked terms present
qa-027 Fail 0.438 20 ms That is outside the scope of what I can determine from the prompt.
Details

Prompt

Answer the question accurately and concisely. Question: What is the largest planet in the solar system?

Expected

Jupiter is the largest planet in the solar system.

Model output

That is outside the scope of what I can determine from the prompt.

  • exact_match Fail 0.000
  • token_f1 Fail 0.111
  • semantic_similarity Fail 0.180
  • llm_judge_correctness Pass 0.900 judge score 9/10
  • safety Pass 1.000 no blocked terms present
qa-028 Fail 0.380 27 ms I do not have enough information to answer that question.
Details

Prompt

Answer the question accurately and concisely. Question: How many bones does an adult human body have?

Expected

An adult human body has 206 bones.

Model output

I do not have enough information to answer that question.

  • exact_match Fail 0.000
  • token_f1 Fail 0.000
  • semantic_similarity Fail 0.000
  • llm_judge_correctness Pass 0.900 judge score 9/10
  • safety Pass 1.000 no blocked terms present
qa-031 Fail 0.591 22 ms DNA acid.
Details

Prompt

Answer the question accurately and concisely. Question: What does DNA stand for?

Expected

DNA stands for deoxyribonucleic acid.

Model output

DNA acid.

  • exact_match Fail 0.000
  • token_f1 Fail 0.571
  • semantic_similarity Fail 0.385
  • llm_judge_correctness Pass 1.000 judge score 10/10
  • safety Pass 1.000 no blocked terms present
qa-035 Fail 0.612 22 ms chemical for table
Details

Prompt

Answer the question accurately and concisely. Question: What is the chemical formula for table salt?

Expected

The chemical formula for table salt is NaCl.

Model output

chemical for table

  • exact_match Fail 0.000
  • token_f1 Pass 0.600
  • semantic_similarity Fail 0.462
  • llm_judge_correctness Pass 1.000 judge score 10/10
  • safety Pass 1.000 no blocked terms present
qa-039 Fail 0.565 33 ms is to the
Details

Prompt

Answer the question accurately and concisely. Question: Which planet is closest to the Sun?

Expected

Mercury is the planet closest to the Sun.

Model output

is to the

  • exact_match Fail 0.000
  • token_f1 Fail 0.500
  • semantic_similarity Pass 0.527
  • llm_judge_correctness Pass 0.800 judge score 8/10
  • safety Pass 1.000 no blocked terms present