nightly-eval · sim-small-v2 simulated

Completed   dataset /home/evalpipe/demo_dataset.jsonl   started 2026-07-13 01:08:35 UTC   run id 5a7d0848b8bc44eaa847d20f7e537e99  ·  Download CSV · JSON

Full results for this single run: the headline scores, how answers were spread out, where the model was strong or weak, and every question with the model's answer.

Pass rate
70.0%
Share of questions answered well enough to pass every check.
95% CI 54.6% – 81.9% (the true rate likely sits in this range)
Mean score
0.856
Average answer quality on a 0–1 scale, across all metrics.
40 questions · 0 error(s)
Latency
26 ms
How long the model took to answer — typical (p50) and near‑worst (p95).
p50 · p95 37 ms
Estimated cost
$0.5180
Rough API cost for this run, based on tokens used.
token-based estimate

Score distribution

How many questions landed in each quality band (0 = wrong, 1 = perfect). Most bars bunched on the right means the model answered well overall.

View data table
Score binItems
0.0-0.10
0.1-0.20
0.2-0.30
0.3-0.43
0.4-0.52
0.5-0.61
0.6-0.74
0.7-0.80
0.8-0.90
0.9-1.030

Mean score by evaluator

How each type of check graded this run (0–1). A short bar flags the quality dimension the model is weakest on.

View data table
EvaluatorMean score
exact_match0.750
llm_judge_correctness0.870
safety1.000
semantic_similarity0.822
token_f10.835

Where it's strong or weak

Pass rate split by difficulty, so a good overall score can't hide a weak spot (e.g. "95% on easy, 20% on hard").

View data table
Difficulty Items Pass rate 95% CI Mean score
medium 17 52.9% 31.0–73.8% 0.768
easy 22 81.8% 61.5–92.7% 0.917
hard 1 100.0% 20.7–100.0% 0.980

Every question

Each question with its outcome, score, and the model's answer. Use Failed to jump straight to what went wrong; open Details on a row to see the prompt, expected answer, and per‑check scores.

Item Outcome Score Latency Output
qa-004 Fail 0.320 35 ms I do not have enough information to answer that question.
Details

Prompt

Answer the question accurately and concisely. Question: What planet is known as the Red Planet?

Expected

Mars is known as the Red Planet.

Model output

I do not have enough information to answer that question.

  • exact_match Fail 0.000
  • token_f1 Fail 0.000
  • semantic_similarity Fail 0.000
  • llm_judge_correctness Fail 0.600 judge score 6/10
  • safety Pass 1.000 no blocked terms present
qa-008 Fail 0.400 36 ms I do not have enough information to answer that question.
Details

Prompt

Answer the question accurately and concisely. Question: What is the chemical symbol for gold?

Expected

The chemical symbol for gold is Au.

Model output

I do not have enough information to answer that question.

  • exact_match Fail 0.000
  • token_f1 Fail 0.000
  • semantic_similarity Fail 0.000
  • llm_judge_correctness Pass 1.000 judge score 10/10
  • safety Pass 1.000 no blocked terms present
qa-010 Fail 0.602 37 ms The smallest 2.
Details

Prompt

Answer the question accurately and concisely. Question: What is the smallest prime number?

Expected

The smallest prime number is 2.

Model output

The smallest 2.

  • exact_match Fail 0.000
  • token_f1 Fail 0.571
  • semantic_similarity Pass 0.539
  • llm_judge_correctness Pass 0.900 judge score 9/10
  • safety Pass 1.000 no blocked terms present
qa-012 Fail 0.920 23 ms The Nile is generally considered the longest river in the world.
Details

Prompt

Answer the question accurately and concisely. Question: What is the longest river in the world?

Expected

The Nile is generally considered the longest river in the world.

Model output

The Nile is generally considered the longest river in the world.

  • exact_match Pass 1.000
  • token_f1 Pass 1.000
  • semantic_similarity Pass 1.000
  • llm_judge_correctness Fail 0.600 judge score 6/10
  • safety Pass 1.000 no blocked terms present
qa-014 Fail 0.340 22 ms That is outside the scope of what I can determine from the prompt.
Details

Prompt

Answer the question accurately and concisely. Question: What is the freezing point of water in Fahrenheit?

Expected

Water freezes at 32 degrees Fahrenheit.

Model output

That is outside the scope of what I can determine from the prompt.

  • exact_match Fail 0.000
  • token_f1 Fail 0.000
  • semantic_similarity Fail 0.000
  • llm_judge_correctness Pass 0.700 judge score 7/10
  • safety Pass 1.000 no blocked terms present
qa-017 Fail 0.592 37 ms Albert Einstein relativity.
Details

Prompt

Answer the question accurately and concisely. Question: Who developed the theory of general relativity?

Expected

Albert Einstein developed the theory of general relativity.

Model output

Albert Einstein relativity.

  • exact_match Fail 0.000
  • token_f1 Pass 0.600
  • semantic_similarity Fail 0.462
  • llm_judge_correctness Pass 0.900 judge score 9/10
  • safety Pass 1.000 no blocked terms present
qa-021 Fail 0.605 31 ms The pumps blood
Details

Prompt

Answer the question accurately and concisely. Question: Which organ pumps blood through the human body?

Expected

The heart pumps blood through the human body.

Model output

The pumps blood

  • exact_match Fail 0.000
  • token_f1 Fail 0.500
  • semantic_similarity Pass 0.527
  • llm_judge_correctness Pass 1.000 judge score 10/10
  • safety Pass 1.000 no blocked terms present
qa-029 Fail 0.664 30 ms The capital Canada Ottawa.
Details

Prompt

Answer the question accurately and concisely. Question: What is the capital of Canada?

Expected

The capital of Canada is Ottawa.

Model output

The capital Canada Ottawa.

  • exact_match Fail 0.000
  • token_f1 Pass 0.750
  • semantic_similarity Pass 0.570
  • llm_judge_correctness Pass 1.000 judge score 10/10
  • safety Pass 1.000 no blocked terms present
qa-033 Fail 0.700 23 ms cheetah is land animal.
Details

Prompt

Answer the question accurately and concisely. Question: What is the fastest land animal?

Expected

The cheetah is the fastest land animal.

Model output

cheetah is land animal.

  • exact_match Fail 0.000
  • token_f1 Pass 0.889
  • semantic_similarity Pass 0.609
  • llm_judge_correctness Pass 1.000 judge score 10/10
  • safety Pass 1.000 no blocked terms present
qa-035 Fail 0.371 29 ms The answer depends on several factors that are not specified here.
Details

Prompt

Answer the question accurately and concisely. Question: What is the chemical formula for table salt?

Expected

The chemical formula for table salt is NaCl.

Model output

The answer depends on several factors that are not specified here.

  • exact_match Fail 0.000
  • token_f1 Fail 0.000
  • semantic_similarity Fail 0.056
  • llm_judge_correctness Pass 0.800 judge score 8/10
  • safety Pass 1.000 no blocked terms present
qa-039 Fail 0.920 22 ms Mercury is the planet closest to the Sun.
Details

Prompt

Answer the question accurately and concisely. Question: Which planet is closest to the Sun?

Expected

Mercury is the planet closest to the Sun.

Model output

Mercury is the planet closest to the Sun.

  • exact_match Pass 1.000
  • token_f1 Pass 1.000
  • semantic_similarity Pass 1.000
  • llm_judge_correctness Fail 0.600 judge score 6/10
  • safety Pass 1.000 no blocked terms present
qa-040 Fail 0.425 24 ms That is outside the scope of what I can determine from the prompt.
Details

Prompt

Answer the question accurately and concisely. Question: What is the sum of the interior angles of a triangle?

Expected

The interior angles of a triangle sum to 180 degrees.

Model output

That is outside the scope of what I can determine from the prompt.

  • exact_match Fail 0.000
  • token_f1 Fail 0.105
  • semantic_similarity Fail 0.121
  • llm_judge_correctness Pass 0.900 judge score 9/10
  • safety Pass 1.000 no blocked terms present