ab-candidate · sim-large-v1 simulated

Completed   dataset /home/evalpipe/demo_dataset.jsonl   started 2026-07-13 01:08:36 UTC   run id da16be5aa117443ea91611c509d390d0  ·  Download CSV · JSON

Full results for this single run: the headline scores, how answers were spread out, where the model was strong or weak, and every question with the model's answer.

Pass rate
82.5%
Share of questions answered well enough to pass every check.
95% CI 68.1% – 91.3% (the true rate likely sits in this range)
Mean score
0.918
Average answer quality on a 0–1 scale, across all metrics.
40 questions · 0 error(s)
Latency
24 ms
How long the model took to answer — typical (p50) and near‑worst (p95).
p50 · p95 36 ms
Estimated cost
$0.5105
Rough API cost for this run, based on tokens used.
token-based estimate

Score distribution

How many questions landed in each quality band (0 = wrong, 1 = perfect). Most bars bunched on the right means the model answered well overall.

View data table
Score binItems
0.0-0.10
0.1-0.20
0.2-0.30
0.3-0.40
0.4-0.52
0.5-0.60
0.6-0.73
0.7-0.81
0.8-0.90
0.9-1.034

Mean score by evaluator

How each type of check graded this run (0–1). A short bar flags the quality dimension the model is weakest on.

View data table
EvaluatorMean score
exact_match0.850
llm_judge_correctness0.875
safety1.000
semantic_similarity0.926
token_f10.941

Where it's strong or weak

Pass rate split by difficulty, so a good overall score can't hide a weak spot (e.g. "95% on easy, 20% on hard").

View data table
Difficulty Items Pass rate 95% CI Mean score
easy 22 77.3% 56.6–89.9% 0.903
medium 17 88.2% 65.7–96.7% 0.934
hard 1 100.0% 20.7–100.0% 0.980

Every question

Each question with its outcome, score, and the model's answer. Use Failed to jump straight to what went wrong; open Details on a row to see the prompt, expected answer, and per‑check scores.

Item Outcome Score Latency Output
qa-003 Fail 0.692 15 ms Romeo Juliet was written William Shakespeare.
Details

Prompt

Answer the question accurately and concisely. Question: Who wrote the play Romeo and Juliet?

Expected

Romeo and Juliet was written by William Shakespeare.

Model output

Romeo Juliet was written William Shakespeare.

  • exact_match Fail 0.000
  • token_f1 Pass 0.857
  • semantic_similarity Pass 0.701
  • llm_judge_correctness Pass 0.900 judge score 9/10
  • safety Pass 1.000 no blocked terms present
qa-005 Fail 0.451 18 ms The answer depends on several factors that are not specified here.
Details

Prompt

Answer the question accurately and concisely. Question: What is the largest ocean on Earth?

Expected

The Pacific Ocean is the largest ocean on Earth.

Model output

The answer depends on several factors that are not specified here.

  • exact_match Fail 0.000
  • token_f1 Fail 0.118
  • semantic_similarity Fail 0.136
  • llm_judge_correctness Pass 1.000 judge score 10/10
  • safety Pass 1.000 no blocked terms present
qa-008 Fail 0.400 28 ms I do not have enough information to answer that question.
Details

Prompt

Answer the question accurately and concisely. Question: What is the chemical symbol for gold?

Expected

The chemical symbol for gold is Au.

Model output

I do not have enough information to answer that question.

  • exact_match Fail 0.000
  • token_f1 Fail 0.000
  • semantic_similarity Fail 0.000
  • llm_judge_correctness Pass 1.000 judge score 10/10
  • safety Pass 1.000 no blocked terms present
qa-012 Fail 0.920 28 ms The Nile is generally considered the longest river in the world.
Details

Prompt

Answer the question accurately and concisely. Question: What is the longest river in the world?

Expected

The Nile is generally considered the longest river in the world.

Model output

The Nile is generally considered the longest river in the world.

  • exact_match Pass 1.000
  • token_f1 Pass 1.000
  • semantic_similarity Pass 1.000
  • llm_judge_correctness Fail 0.600 judge score 6/10
  • safety Pass 1.000 no blocked terms present
qa-020 Fail 0.680 29 ms Portuguese is language spoken in Brazil.
Details

Prompt

Answer the question accurately and concisely. Question: What is the primary language spoken in Brazil?

Expected

Portuguese is the primary language spoken in Brazil.

Model output

Portuguese is language spoken in Brazil.

  • exact_match Fail 0.000
  • token_f1 Pass 0.923
  • semantic_similarity Pass 0.778
  • llm_judge_correctness Pass 0.700 judge score 7/10
  • safety Pass 1.000 no blocked terms present
qa-036 Fail 0.757 32 ms Washington was the first President of United States.
Details

Prompt

Answer the question accurately and concisely. Question: Who was the first President of the United States?

Expected

George Washington was the first President of the United States.

Model output

Washington was the first President of United States.

  • exact_match Fail 0.000
  • token_f1 Pass 0.933
  • semantic_similarity Pass 0.851
  • llm_judge_correctness Pass 1.000 judge score 10/10
  • safety Pass 1.000 no blocked terms present
qa-039 Fail 0.654 23 ms planet closest to Sun.
Details

Prompt

Answer the question accurately and concisely. Question: Which planet is closest to the Sun?

Expected

Mercury is the planet closest to the Sun.

Model output

planet closest to Sun.

  • exact_match Fail 0.000
  • token_f1 Pass 0.800
  • semantic_similarity Pass 0.569
  • llm_judge_correctness Pass 0.900 judge score 9/10
  • safety Pass 1.000 no blocked terms present