nightly-eval · sim-small-v1 simulated

Completed   dataset /home/evalpipe/demo_dataset.jsonl   started 2026-07-13 01:08:35 UTC   run id 877dfc4addf140f88640d4a106afe254  ·  Download CSV · JSON

Full results for this single run: the headline scores, how answers were spread out, where the model was strong or weak, and every question with the model's answer.

Pass rate
37.5%
Share of questions answered well enough to pass every check.
95% CI 24.2% – 53.0% (the true rate likely sits in this range)
Mean score
0.716
Average answer quality on a 0–1 scale, across all metrics.
40 questions · 0 error(s)
Latency
25 ms
How long the model took to answer — typical (p50) and near‑worst (p95).
p50 · p95 38 ms
Estimated cost
$0.5142
Rough API cost for this run, based on tokens used.
token-based estimate

Score distribution

How many questions landed in each quality band (0 = wrong, 1 = perfect). Most bars bunched on the right means the model answered well overall.

View data table
Score binItems
0.0-0.10
0.1-0.20
0.2-0.30
0.3-0.44
0.4-0.57
0.5-0.60
0.6-0.710
0.7-0.83
0.8-0.90
0.9-1.016

Mean score by evaluator

How each type of check graded this run (0–1). A short bar flags the quality dimension the model is weakest on.

View data table
EvaluatorMean score
exact_match0.400
llm_judge_correctness0.880
safety1.000
semantic_similarity0.637
token_f10.662

Where it's strong or weak

Pass rate split by difficulty, so a good overall score can't hide a weak spot (e.g. "95% on easy, 20% on hard").

View data table
Difficulty Items Pass rate 95% CI Mean score
hard 1 0.0% 0.0–79.3% 0.380
medium 17 35.3% 17.3–58.7% 0.700
easy 22 40.9% 23.3–61.3% 0.743

Every question

Each question with its outcome, score, and the model's answer. Use Failed to jump straight to what went wrong; open Details on a row to see the prompt, expected answer, and per‑check scores.

Item Outcome Score Latency Output
qa-001 Fail 0.667 38 ms The capital of is
Details

Prompt

Answer the question accurately and concisely. Question: What is the capital of France?

Expected

The capital of France is Paris.

Model output

The capital of is

  • exact_match Fail 0.000
  • token_f1 Pass 0.750
  • semantic_similarity Pass 0.684
  • llm_judge_correctness Pass 0.900 judge score 9/10
  • safety Pass 1.000 no blocked terms present
qa-003 Fail 0.360 38 ms I do not have enough information to answer that question.
Details

Prompt

Answer the question accurately and concisely. Question: Who wrote the play Romeo and Juliet?

Expected

Romeo and Juliet was written by William Shakespeare.

Model output

I do not have enough information to answer that question.

  • exact_match Fail 0.000
  • token_f1 Fail 0.000
  • semantic_similarity Fail 0.000
  • llm_judge_correctness Pass 0.800 judge score 8/10
  • safety Pass 1.000 no blocked terms present
qa-004 Fail 0.666 32 ms Mars is known Planet.
Details

Prompt

Answer the question accurately and concisely. Question: What planet is known as the Red Planet?

Expected

Mars is known as the Red Planet.

Model output

Mars is known Planet.

  • exact_match Fail 0.000
  • token_f1 Pass 0.800
  • semantic_similarity Pass 0.629
  • llm_judge_correctness Pass 0.900 judge score 9/10
  • safety Pass 1.000 no blocked terms present
qa-005 Fail 0.451 33 ms The answer depends on several factors that are not specified here.
Details

Prompt

Answer the question accurately and concisely. Question: What is the largest ocean on Earth?

Expected

The Pacific Ocean is the largest ocean on Earth.

Model output

The answer depends on several factors that are not specified here.

  • exact_match Fail 0.000
  • token_f1 Fail 0.118
  • semantic_similarity Fail 0.136
  • llm_judge_correctness Pass 1.000 judge score 10/10
  • safety Pass 1.000 no blocked terms present
qa-006 Fail 0.641 30 ms There are on
Details

Prompt

Answer the question accurately and concisely. Question: How many continents are there on Earth?

Expected

There are seven continents on Earth.

Model output

There are on

  • exact_match Fail 0.000
  • token_f1 Pass 0.667
  • semantic_similarity Pass 0.539
  • llm_judge_correctness Pass 1.000 judge score 10/10
  • safety Pass 1.000 no blocked terms present
qa-007 Fail 0.400 28 ms The answer depends on several factors that are not specified here.
Details

Prompt

Answer the question accurately and concisely. Question: What gas do plants absorb during photosynthesis?

Expected

Plants absorb carbon dioxide during photosynthesis.

Model output

The answer depends on several factors that are not specified here.

  • exact_match Fail 0.000
  • token_f1 Fail 0.000
  • semantic_similarity Fail 0.000
  • llm_judge_correctness Pass 1.000 judge score 10/10
  • safety Pass 1.000 no blocked terms present
qa-009 Fail 0.400 20 ms That is outside the scope of what I can determine from the prompt.
Details

Prompt

Answer the question accurately and concisely. Question: Which country is home to the kangaroo?

Expected

Kangaroos are native to Australia.

Model output

That is outside the scope of what I can determine from the prompt.

  • exact_match Fail 0.000
  • token_f1 Fail 0.000
  • semantic_similarity Fail 0.000
  • llm_judge_correctness Pass 1.000 judge score 10/10
  • safety Pass 1.000 no blocked terms present
qa-012 Fail 0.625 26 ms Nile generally the river in the world.
Details

Prompt

Answer the question accurately and concisely. Question: What is the longest river in the world?

Expected

The Nile is generally considered the longest river in the world.

Model output

Nile generally the river in the world.

  • exact_match Fail 0.000
  • token_f1 Pass 0.769
  • semantic_similarity Pass 0.655
  • llm_judge_correctness Pass 0.700 judge score 7/10
  • safety Pass 1.000 no blocked terms present
qa-014 Fail 0.621 33 ms Water degrees Fahrenheit.
Details

Prompt

Answer the question accurately and concisely. Question: What is the freezing point of water in Fahrenheit?

Expected

Water freezes at 32 degrees Fahrenheit.

Model output

Water degrees Fahrenheit.

  • exact_match Fail 0.000
  • token_f1 Pass 0.667
  • semantic_similarity Pass 0.539
  • llm_judge_correctness Pass 0.900 judge score 9/10
  • safety Pass 1.000 no blocked terms present
qa-015 Fail 0.622 25 ms the number 1.
Details

Prompt

Answer the question accurately and concisely. Question: Which element has the atomic number 1?

Expected

Hydrogen has the atomic number 1.

Model output

the number 1.

  • exact_match Fail 0.000
  • token_f1 Fail 0.571
  • semantic_similarity Pass 0.539
  • llm_judge_correctness Pass 1.000 judge score 10/10
  • safety Pass 1.000 no blocked terms present
qa-018 Fail 0.387 21 ms That is outside the scope of what I can determine from the prompt.
Details

Prompt

Answer the question accurately and concisely. Question: What is the tallest mountain above sea level?

Expected

Mount Everest is the tallest mountain above sea level.

Model output

That is outside the scope of what I can determine from the prompt.

  • exact_match Fail 0.000
  • token_f1 Fail 0.105
  • semantic_similarity Fail 0.128
  • llm_judge_correctness Pass 0.700 judge score 7/10
  • safety Pass 1.000 no blocked terms present
qa-019 Fail 0.624 23 ms There are 120
Details

Prompt

Answer the question accurately and concisely. Question: How many minutes are there in two hours?

Expected

There are 120 minutes in two hours.

Model output

There are 120

  • exact_match Fail 0.000
  • token_f1 Pass 0.600
  • semantic_similarity Pass 0.620
  • llm_judge_correctness Pass 0.900 judge score 9/10
  • safety Pass 1.000 no blocked terms present
qa-021 Fail 0.605 33 ms The heart blood the
Details

Prompt

Answer the question accurately and concisely. Question: Which organ pumps blood through the human body?

Expected

The heart pumps blood through the human body.

Model output

The heart blood the

  • exact_match Fail 0.000
  • token_f1 Fail 0.500
  • semantic_similarity Pass 0.525
  • llm_judge_correctness Pass 1.000 judge score 10/10
  • safety Pass 1.000 no blocked terms present
qa-023 Fail 0.647 25 ms Apollo 11 humans on in
Details

Prompt

Answer the question accurately and concisely. Question: In which year did the Apollo 11 mission land humans on the Moon?

Expected

Apollo 11 landed humans on the Moon in 1969.

Model output

Apollo 11 humans on in

  • exact_match Fail 0.000
  • token_f1 Pass 0.769
  • semantic_similarity Pass 0.566
  • llm_judge_correctness Pass 0.900 judge score 9/10
  • safety Pass 1.000 no blocked terms present
qa-024 Fail 0.454 24 ms That is outside the scope of what I can determine from the prompt.
Details

Prompt

Answer the question accurately and concisely. Question: What is the capital of Japan?

Expected

The capital of Japan is Tokyo.

Model output

That is outside the scope of what I can determine from the prompt.

  • exact_match Fail 0.000
  • token_f1 Fail 0.250
  • semantic_similarity Fail 0.219
  • llm_judge_correctness Pass 0.800 judge score 8/10
  • safety Pass 1.000 no blocked terms present
qa-025 Fail 0.712 21 ms Nitrogen makes up most of atmosphere.
Details

Prompt

Answer the question accurately and concisely. Question: Which gas makes up most of Earth's atmosphere?

Expected

Nitrogen makes up most of Earth's atmosphere.

Model output

Nitrogen makes up most of atmosphere.

  • exact_match Fail 0.000
  • token_f1 Pass 0.923
  • semantic_similarity Pass 0.836
  • llm_judge_correctness Pass 0.800 judge score 8/10
  • safety Pass 1.000 no blocked terms present
qa-026 Fail 0.500 16 ms Prejudice Austen.
Details

Prompt

Answer the question accurately and concisely. Question: Who wrote the novel Pride and Prejudice?

Expected

Pride and Prejudice was written by Jane Austen.

Model output

Prejudice Austen.

  • exact_match Fail 0.000
  • token_f1 Fail 0.400
  • semantic_similarity Fail 0.298
  • llm_judge_correctness Pass 0.800 judge score 8/10
  • safety Pass 1.000 no blocked terms present
qa-027 Fail 0.761 22 ms Jupiter is the planet in the solar system.
Details

Prompt

Answer the question accurately and concisely. Question: What is the largest planet in the solar system?

Expected

Jupiter is the largest planet in the solar system.

Model output

Jupiter is the planet in the solar system.

  • exact_match Fail 0.000
  • token_f1 Pass 0.923
  • semantic_similarity Pass 0.883
  • llm_judge_correctness Pass 1.000 judge score 10/10
  • safety Pass 1.000 no blocked terms present
qa-028 Fail 0.380 34 ms I do not have enough information to answer that question.
Details

Prompt

Answer the question accurately and concisely. Question: How many bones does an adult human body have?

Expected

An adult human body has 206 bones.

Model output

I do not have enough information to answer that question.

  • exact_match Fail 0.000
  • token_f1 Fail 0.000
  • semantic_similarity Fail 0.000
  • llm_judge_correctness Pass 0.900 judge score 9/10
  • safety Pass 1.000 no blocked terms present
qa-031 Fail 0.714 23 ms DNA stands for deoxyribonucleic
Details

Prompt

Answer the question accurately and concisely. Question: What does DNA stand for?

Expected

DNA stands for deoxyribonucleic acid.

Model output

DNA stands for deoxyribonucleic

  • exact_match Fail 0.000
  • token_f1 Pass 0.889
  • semantic_similarity Pass 0.882
  • llm_judge_correctness Pass 0.800 judge score 8/10
  • safety Pass 1.000 no blocked terms present
qa-035 Fail 0.400 37 ms I do not have enough information to answer that question.
Details

Prompt

Answer the question accurately and concisely. Question: What is the chemical formula for table salt?

Expected

The chemical formula for table salt is NaCl.

Model output

I do not have enough information to answer that question.

  • exact_match Fail 0.000
  • token_f1 Fail 0.000
  • semantic_similarity Fail 0.000
  • llm_judge_correctness Pass 1.000 judge score 10/10
  • safety Pass 1.000 no blocked terms present
qa-037 Fail 0.633 30 ms A has seven
Details

Prompt

Answer the question accurately and concisely. Question: How many colors are there in a rainbow?

Expected

A rainbow has seven colors.

Model output

A has seven

  • exact_match Fail 0.000
  • token_f1 Pass 0.667
  • semantic_similarity Pass 0.596
  • llm_judge_correctness Pass 0.900 judge score 9/10
  • safety Pass 1.000 no blocked terms present
qa-038 Fail 0.430 15 ms That is outside the scope of what I can determine from the prompt.
Details

Prompt

Answer the question accurately and concisely. Question: What is the hardest natural substance on Earth?

Expected

Diamond is the hardest natural substance on Earth.

Model output

That is outside the scope of what I can determine from the prompt.

  • exact_match Fail 0.000
  • token_f1 Fail 0.111
  • semantic_similarity Fail 0.137
  • llm_judge_correctness Pass 0.900 judge score 9/10
  • safety Pass 1.000 no blocked terms present
qa-039 Fail 0.920 24 ms Mercury is the planet closest to the Sun.
Details

Prompt

Answer the question accurately and concisely. Question: Which planet is closest to the Sun?

Expected

Mercury is the planet closest to the Sun.

Model output

Mercury is the planet closest to the Sun.

  • exact_match Pass 1.000
  • token_f1 Pass 1.000
  • semantic_similarity Pass 1.000
  • llm_judge_correctness Fail 0.600 judge score 6/10
  • safety Pass 1.000 no blocked terms present
qa-040 Fail 0.350 34 ms The answer depends on several factors that are not specified here.
Details

Prompt

Answer the question accurately and concisely. Question: What is the sum of the interior angles of a triangle?

Expected

The interior angles of a triangle sum to 180 degrees.

Model output

The answer depends on several factors that are not specified here.

  • exact_match Fail 0.000
  • token_f1 Fail 0.000
  • semantic_similarity Fail 0.050
  • llm_judge_correctness Pass 0.700 judge score 7/10
  • safety Pass 1.000 no blocked terms present