nightly-eval · sim-small-v1 simulated

Completed   dataset /home/evalpipe/demo_dataset.jsonl   started 2026-07-13 01:08:35 UTC   run id 877dfc4addf140f88640d4a106afe254  ·  Download CSV · JSON

Full results for this single run: the headline scores, how answers were spread out, where the model was strong or weak, and every question with the model's answer.

Pass rate
37.5%
Share of questions answered well enough to pass every check.
95% CI 24.2% – 53.0% (the true rate likely sits in this range)
Mean score
0.716
Average answer quality on a 0–1 scale, across all metrics.
40 questions · 0 error(s)
Latency
25 ms
How long the model took to answer — typical (p50) and near‑worst (p95).
p50 · p95 38 ms
Estimated cost
$0.5142
Rough API cost for this run, based on tokens used.
token-based estimate

Score distribution

How many questions landed in each quality band (0 = wrong, 1 = perfect). Most bars bunched on the right means the model answered well overall.

View data table
Score binItems
0.0-0.10
0.1-0.20
0.2-0.30
0.3-0.44
0.4-0.57
0.5-0.60
0.6-0.710
0.7-0.83
0.8-0.90
0.9-1.016

Mean score by evaluator

How each type of check graded this run (0–1). A short bar flags the quality dimension the model is weakest on.

View data table
EvaluatorMean score
exact_match0.400
llm_judge_correctness0.880
safety1.000
semantic_similarity0.637
token_f10.662

Where it's strong or weak

Pass rate split by difficulty, so a good overall score can't hide a weak spot (e.g. "95% on easy, 20% on hard").

View data table
Difficulty Items Pass rate 95% CI Mean score
hard 1 0.0% 0.0–79.3% 0.380
medium 17 35.3% 17.3–58.7% 0.700
easy 22 40.9% 23.3–61.3% 0.743

Every question

Each question with its outcome, score, and the model's answer. Use Failed to jump straight to what went wrong; open Details on a row to see the prompt, expected answer, and per‑check scores.

Item Outcome Score Latency Output
qa-001 Fail 0.667 38 ms The capital of is
Details

Prompt

Answer the question accurately and concisely. Question: What is the capital of France?

Expected

The capital of France is Paris.

Model output

The capital of is

  • exact_match Fail 0.000
  • token_f1 Pass 0.750
  • semantic_similarity Pass 0.684
  • llm_judge_correctness Pass 0.900 judge score 9/10
  • safety Pass 1.000 no blocked terms present
qa-002 Pass 1.000 16 ms Water boils at 100 degrees Celsius at sea level.
Details

Prompt

Answer the question accurately and concisely. Question: What is the boiling point of water at sea level in Celsius?

Expected

Water boils at 100 degrees Celsius at sea level.

Model output

Water boils at 100 degrees Celsius at sea level.

  • exact_match Pass 1.000
  • token_f1 Pass 1.000
  • semantic_similarity Pass 1.000
  • llm_judge_correctness Pass 1.000 judge score 10/10
  • safety Pass 1.000 no blocked terms present
qa-003 Fail 0.360 38 ms I do not have enough information to answer that question.
Details

Prompt

Answer the question accurately and concisely. Question: Who wrote the play Romeo and Juliet?

Expected

Romeo and Juliet was written by William Shakespeare.

Model output

I do not have enough information to answer that question.

  • exact_match Fail 0.000
  • token_f1 Fail 0.000
  • semantic_similarity Fail 0.000
  • llm_judge_correctness Pass 0.800 judge score 8/10
  • safety Pass 1.000 no blocked terms present
qa-004 Fail 0.666 32 ms Mars is known Planet.
Details

Prompt

Answer the question accurately and concisely. Question: What planet is known as the Red Planet?

Expected

Mars is known as the Red Planet.

Model output

Mars is known Planet.

  • exact_match Fail 0.000
  • token_f1 Pass 0.800
  • semantic_similarity Pass 0.629
  • llm_judge_correctness Pass 0.900 judge score 9/10
  • safety Pass 1.000 no blocked terms present
qa-005 Fail 0.451 33 ms The answer depends on several factors that are not specified here.
Details

Prompt

Answer the question accurately and concisely. Question: What is the largest ocean on Earth?

Expected

The Pacific Ocean is the largest ocean on Earth.

Model output

The answer depends on several factors that are not specified here.

  • exact_match Fail 0.000
  • token_f1 Fail 0.118
  • semantic_similarity Fail 0.136
  • llm_judge_correctness Pass 1.000 judge score 10/10
  • safety Pass 1.000 no blocked terms present
qa-006 Fail 0.641 30 ms There are on
Details

Prompt

Answer the question accurately and concisely. Question: How many continents are there on Earth?

Expected

There are seven continents on Earth.

Model output

There are on

  • exact_match Fail 0.000
  • token_f1 Pass 0.667
  • semantic_similarity Pass 0.539
  • llm_judge_correctness Pass 1.000 judge score 10/10
  • safety Pass 1.000 no blocked terms present
qa-007 Fail 0.400 28 ms The answer depends on several factors that are not specified here.
Details

Prompt

Answer the question accurately and concisely. Question: What gas do plants absorb during photosynthesis?

Expected

Plants absorb carbon dioxide during photosynthesis.

Model output

The answer depends on several factors that are not specified here.

  • exact_match Fail 0.000
  • token_f1 Fail 0.000
  • semantic_similarity Fail 0.000
  • llm_judge_correctness Pass 1.000 judge score 10/10
  • safety Pass 1.000 no blocked terms present
qa-008 Pass 1.000 23 ms The chemical symbol for gold is Au.
Details

Prompt

Answer the question accurately and concisely. Question: What is the chemical symbol for gold?

Expected

The chemical symbol for gold is Au.

Model output

The chemical symbol for gold is Au.

  • exact_match Pass 1.000
  • token_f1 Pass 1.000
  • semantic_similarity Pass 1.000
  • llm_judge_correctness Pass 1.000 judge score 10/10
  • safety Pass 1.000 no blocked terms present
qa-009 Fail 0.400 20 ms That is outside the scope of what I can determine from the prompt.
Details

Prompt

Answer the question accurately and concisely. Question: Which country is home to the kangaroo?

Expected

Kangaroos are native to Australia.

Model output

That is outside the scope of what I can determine from the prompt.

  • exact_match Fail 0.000
  • token_f1 Fail 0.000
  • semantic_similarity Fail 0.000
  • llm_judge_correctness Pass 1.000 judge score 10/10
  • safety Pass 1.000 no blocked terms present
qa-010 Pass 0.940 20 ms The smallest prime number is 2.
Details

Prompt

Answer the question accurately and concisely. Question: What is the smallest prime number?

Expected

The smallest prime number is 2.

Model output

The smallest prime number is 2.

  • exact_match Pass 1.000
  • token_f1 Pass 1.000
  • semantic_similarity Pass 1.000
  • llm_judge_correctness Pass 0.700 judge score 7/10
  • safety Pass 1.000 no blocked terms present
qa-011 Pass 0.980 36 ms The Mona Lisa was painted by Leonardo da Vinci.
Details

Prompt

Answer the question accurately and concisely. Question: Who painted the Mona Lisa?

Expected

The Mona Lisa was painted by Leonardo da Vinci.

Model output

The Mona Lisa was painted by Leonardo da Vinci.

  • exact_match Pass 1.000
  • token_f1 Pass 1.000
  • semantic_similarity Pass 1.000
  • llm_judge_correctness Pass 0.900 judge score 9/10
  • safety Pass 1.000 no blocked terms present
qa-012 Fail 0.625 26 ms Nile generally the river in the world.
Details

Prompt

Answer the question accurately and concisely. Question: What is the longest river in the world?

Expected

The Nile is generally considered the longest river in the world.

Model output

Nile generally the river in the world.

  • exact_match Fail 0.000
  • token_f1 Pass 0.769
  • semantic_similarity Pass 0.655
  • llm_judge_correctness Pass 0.700 judge score 7/10
  • safety Pass 1.000 no blocked terms present
qa-013 Pass 0.960 38 ms A hexagon has six sides.
Details

Prompt

Answer the question accurately and concisely. Question: How many sides does a hexagon have?

Expected

A hexagon has six sides.

Model output

A hexagon has six sides.

  • exact_match Pass 1.000
  • token_f1 Pass 1.000
  • semantic_similarity Pass 1.000
  • llm_judge_correctness Pass 0.800 judge score 8/10
  • safety Pass 1.000 no blocked terms present
qa-014 Fail 0.621 33 ms Water degrees Fahrenheit.
Details

Prompt

Answer the question accurately and concisely. Question: What is the freezing point of water in Fahrenheit?

Expected

Water freezes at 32 degrees Fahrenheit.

Model output

Water degrees Fahrenheit.

  • exact_match Fail 0.000
  • token_f1 Pass 0.667
  • semantic_similarity Pass 0.539
  • llm_judge_correctness Pass 0.900 judge score 9/10
  • safety Pass 1.000 no blocked terms present
qa-015 Fail 0.622 25 ms the number 1.
Details

Prompt

Answer the question accurately and concisely. Question: Which element has the atomic number 1?

Expected

Hydrogen has the atomic number 1.

Model output

the number 1.

  • exact_match Fail 0.000
  • token_f1 Fail 0.571
  • semantic_similarity Pass 0.539
  • llm_judge_correctness Pass 1.000 judge score 10/10
  • safety Pass 1.000 no blocked terms present
qa-016 Pass 0.960 34 ms The currency of Japan is the yen.
Details

Prompt

Answer the question accurately and concisely. Question: What is the currency of Japan?

Expected

The currency of Japan is the yen.

Model output

The currency of Japan is the yen.

  • exact_match Pass 1.000
  • token_f1 Pass 1.000
  • semantic_similarity Pass 1.000
  • llm_judge_correctness Pass 0.800 judge score 8/10
  • safety Pass 1.000 no blocked terms present
qa-017 Pass 1.000 22 ms Albert Einstein developed the theory of general relativity.
Details

Prompt

Answer the question accurately and concisely. Question: Who developed the theory of general relativity?

Expected

Albert Einstein developed the theory of general relativity.

Model output

Albert Einstein developed the theory of general relativity.

  • exact_match Pass 1.000
  • token_f1 Pass 1.000
  • semantic_similarity Pass 1.000
  • llm_judge_correctness Pass 1.000 judge score 10/10
  • safety Pass 1.000 no blocked terms present
qa-018 Fail 0.387 21 ms That is outside the scope of what I can determine from the prompt.
Details

Prompt

Answer the question accurately and concisely. Question: What is the tallest mountain above sea level?

Expected

Mount Everest is the tallest mountain above sea level.

Model output

That is outside the scope of what I can determine from the prompt.

  • exact_match Fail 0.000
  • token_f1 Fail 0.105
  • semantic_similarity Fail 0.128
  • llm_judge_correctness Pass 0.700 judge score 7/10
  • safety Pass 1.000 no blocked terms present
qa-019 Fail 0.624 23 ms There are 120
Details

Prompt

Answer the question accurately and concisely. Question: How many minutes are there in two hours?

Expected

There are 120 minutes in two hours.

Model output

There are 120

  • exact_match Fail 0.000
  • token_f1 Pass 0.600
  • semantic_similarity Pass 0.620
  • llm_judge_correctness Pass 0.900 judge score 9/10
  • safety Pass 1.000 no blocked terms present
qa-020 Pass 1.000 19 ms Portuguese is the primary language spoken in Brazil.
Details

Prompt

Answer the question accurately and concisely. Question: What is the primary language spoken in Brazil?

Expected

Portuguese is the primary language spoken in Brazil.

Model output

Portuguese is the primary language spoken in Brazil.

  • exact_match Pass 1.000
  • token_f1 Pass 1.000
  • semantic_similarity Pass 1.000
  • llm_judge_correctness Pass 1.000 judge score 10/10
  • safety Pass 1.000 no blocked terms present
qa-021 Fail 0.605 33 ms The heart blood the
Details

Prompt

Answer the question accurately and concisely. Question: Which organ pumps blood through the human body?

Expected

The heart pumps blood through the human body.

Model output

The heart blood the

  • exact_match Fail 0.000
  • token_f1 Fail 0.500
  • semantic_similarity Pass 0.525
  • llm_judge_correctness Pass 1.000 judge score 10/10
  • safety Pass 1.000 no blocked terms present
qa-022 Pass 0.960 23 ms The square root of 144 is 12.
Details

Prompt

Answer the question accurately and concisely. Question: What is the square root of 144?

Expected

The square root of 144 is 12.

Model output

The square root of 144 is 12.

  • exact_match Pass 1.000
  • token_f1 Pass 1.000
  • semantic_similarity Pass 1.000
  • llm_judge_correctness Pass 0.800 judge score 8/10
  • safety Pass 1.000 no blocked terms present
qa-023 Fail 0.647 25 ms Apollo 11 humans on in
Details

Prompt

Answer the question accurately and concisely. Question: In which year did the Apollo 11 mission land humans on the Moon?

Expected

Apollo 11 landed humans on the Moon in 1969.

Model output

Apollo 11 humans on in

  • exact_match Fail 0.000
  • token_f1 Pass 0.769
  • semantic_similarity Pass 0.566
  • llm_judge_correctness Pass 0.900 judge score 9/10
  • safety Pass 1.000 no blocked terms present
qa-024 Fail 0.454 24 ms That is outside the scope of what I can determine from the prompt.
Details

Prompt

Answer the question accurately and concisely. Question: What is the capital of Japan?

Expected

The capital of Japan is Tokyo.

Model output

That is outside the scope of what I can determine from the prompt.

  • exact_match Fail 0.000
  • token_f1 Fail 0.250
  • semantic_similarity Fail 0.219
  • llm_judge_correctness Pass 0.800 judge score 8/10
  • safety Pass 1.000 no blocked terms present
qa-025 Fail 0.712 21 ms Nitrogen makes up most of atmosphere.
Details

Prompt

Answer the question accurately and concisely. Question: Which gas makes up most of Earth's atmosphere?

Expected

Nitrogen makes up most of Earth's atmosphere.

Model output

Nitrogen makes up most of atmosphere.

  • exact_match Fail 0.000
  • token_f1 Pass 0.923
  • semantic_similarity Pass 0.836
  • llm_judge_correctness Pass 0.800 judge score 8/10
  • safety Pass 1.000 no blocked terms present
qa-026 Fail 0.500 16 ms Prejudice Austen.
Details

Prompt

Answer the question accurately and concisely. Question: Who wrote the novel Pride and Prejudice?

Expected

Pride and Prejudice was written by Jane Austen.

Model output

Prejudice Austen.

  • exact_match Fail 0.000
  • token_f1 Fail 0.400
  • semantic_similarity Fail 0.298
  • llm_judge_correctness Pass 0.800 judge score 8/10
  • safety Pass 1.000 no blocked terms present
qa-027 Fail 0.761 22 ms Jupiter is the planet in the solar system.
Details

Prompt

Answer the question accurately and concisely. Question: What is the largest planet in the solar system?

Expected

Jupiter is the largest planet in the solar system.

Model output

Jupiter is the planet in the solar system.

  • exact_match Fail 0.000
  • token_f1 Pass 0.923
  • semantic_similarity Pass 0.883
  • llm_judge_correctness Pass 1.000 judge score 10/10
  • safety Pass 1.000 no blocked terms present
qa-028 Fail 0.380 34 ms I do not have enough information to answer that question.
Details

Prompt

Answer the question accurately and concisely. Question: How many bones does an adult human body have?

Expected

An adult human body has 206 bones.

Model output

I do not have enough information to answer that question.

  • exact_match Fail 0.000
  • token_f1 Fail 0.000
  • semantic_similarity Fail 0.000
  • llm_judge_correctness Pass 0.900 judge score 9/10
  • safety Pass 1.000 no blocked terms present
qa-029 Pass 0.960 23 ms The capital of Canada is Ottawa.
Details

Prompt

Answer the question accurately and concisely. Question: What is the capital of Canada?

Expected

The capital of Canada is Ottawa.

Model output

The capital of Canada is Ottawa.

  • exact_match Pass 1.000
  • token_f1 Pass 1.000
  • semantic_similarity Pass 1.000
  • llm_judge_correctness Pass 0.800 judge score 8/10
  • safety Pass 1.000 no blocked terms present
qa-030 Pass 1.000 24 ms Mercury is liquid at room temperature.
Details

Prompt

Answer the question accurately and concisely. Question: Which metal is liquid at room temperature?

Expected

Mercury is liquid at room temperature.

Model output

Mercury is liquid at room temperature.

  • exact_match Pass 1.000
  • token_f1 Pass 1.000
  • semantic_similarity Pass 1.000
  • llm_judge_correctness Pass 1.000 judge score 10/10
  • safety Pass 1.000 no blocked terms present
qa-031 Fail 0.714 23 ms DNA stands for deoxyribonucleic
Details

Prompt

Answer the question accurately and concisely. Question: What does DNA stand for?

Expected

DNA stands for deoxyribonucleic acid.

Model output

DNA stands for deoxyribonucleic

  • exact_match Fail 0.000
  • token_f1 Pass 0.889
  • semantic_similarity Pass 0.882
  • llm_judge_correctness Pass 0.800 judge score 8/10
  • safety Pass 1.000 no blocked terms present
qa-032 Pass 0.980 35 ms A soccer team has 11 players on the field.
Details

Prompt

Answer the question accurately and concisely. Question: How many players are on a soccer team on the field?

Expected

A soccer team has 11 players on the field.

Model output

A soccer team has 11 players on the field.

  • exact_match Pass 1.000
  • token_f1 Pass 1.000
  • semantic_similarity Pass 1.000
  • llm_judge_correctness Pass 0.900 judge score 9/10
  • safety Pass 1.000 no blocked terms present
qa-033 Pass 0.980 23 ms The cheetah is the fastest land animal.
Details

Prompt

Answer the question accurately and concisely. Question: What is the fastest land animal?

Expected

The cheetah is the fastest land animal.

Model output

The cheetah is the fastest land animal.

  • exact_match Pass 1.000
  • token_f1 Pass 1.000
  • semantic_similarity Pass 1.000
  • llm_judge_correctness Pass 0.900 judge score 9/10
  • safety Pass 1.000 no blocked terms present
qa-034 Pass 0.980 30 ms The Indian Ocean lies between Africa and Australia.
Details

Prompt

Answer the question accurately and concisely. Question: Which ocean lies between Africa and Australia?

Expected

The Indian Ocean lies between Africa and Australia.

Model output

The Indian Ocean lies between Africa and Australia.

  • exact_match Pass 1.000
  • token_f1 Pass 1.000
  • semantic_similarity Pass 1.000
  • llm_judge_correctness Pass 0.900 judge score 9/10
  • safety Pass 1.000 no blocked terms present
qa-035 Fail 0.400 37 ms I do not have enough information to answer that question.
Details

Prompt

Answer the question accurately and concisely. Question: What is the chemical formula for table salt?

Expected

The chemical formula for table salt is NaCl.

Model output

I do not have enough information to answer that question.

  • exact_match Fail 0.000
  • token_f1 Fail 0.000
  • semantic_similarity Fail 0.000
  • llm_judge_correctness Pass 1.000 judge score 10/10
  • safety Pass 1.000 no blocked terms present
qa-036 Pass 0.960 34 ms George Washington was the first President of the United States.
Details

Prompt

Answer the question accurately and concisely. Question: Who was the first President of the United States?

Expected

George Washington was the first President of the United States.

Model output

George Washington was the first President of the United States.

  • exact_match Pass 1.000
  • token_f1 Pass 1.000
  • semantic_similarity Pass 1.000
  • llm_judge_correctness Pass 0.800 judge score 8/10
  • safety Pass 1.000 no blocked terms present
qa-037 Fail 0.633 30 ms A has seven
Details

Prompt

Answer the question accurately and concisely. Question: How many colors are there in a rainbow?

Expected

A rainbow has seven colors.

Model output

A has seven

  • exact_match Fail 0.000
  • token_f1 Pass 0.667
  • semantic_similarity Pass 0.596
  • llm_judge_correctness Pass 0.900 judge score 9/10
  • safety Pass 1.000 no blocked terms present
qa-038 Fail 0.430 15 ms That is outside the scope of what I can determine from the prompt.
Details

Prompt

Answer the question accurately and concisely. Question: What is the hardest natural substance on Earth?

Expected

Diamond is the hardest natural substance on Earth.

Model output

That is outside the scope of what I can determine from the prompt.

  • exact_match Fail 0.000
  • token_f1 Fail 0.111
  • semantic_similarity Fail 0.137
  • llm_judge_correctness Pass 0.900 judge score 9/10
  • safety Pass 1.000 no blocked terms present
qa-039 Fail 0.920 24 ms Mercury is the planet closest to the Sun.
Details

Prompt

Answer the question accurately and concisely. Question: Which planet is closest to the Sun?

Expected

Mercury is the planet closest to the Sun.

Model output

Mercury is the planet closest to the Sun.

  • exact_match Pass 1.000
  • token_f1 Pass 1.000
  • semantic_similarity Pass 1.000
  • llm_judge_correctness Fail 0.600 judge score 6/10
  • safety Pass 1.000 no blocked terms present
qa-040 Fail 0.350 34 ms The answer depends on several factors that are not specified here.
Details

Prompt

Answer the question accurately and concisely. Question: What is the sum of the interior angles of a triangle?

Expected

The interior angles of a triangle sum to 180 degrees.

Model output

The answer depends on several factors that are not specified here.

  • exact_match Fail 0.000
  • token_f1 Fail 0.000
  • semantic_similarity Fail 0.050
  • llm_judge_correctness Pass 0.700 judge score 7/10
  • safety Pass 1.000 no blocked terms present