nightly-eval · sim-small-v1 simulated

Completed   dataset /home/evalpipe/demo_dataset.jsonl   started 2026-07-13 01:08:35 UTC   run id be4ff9c1852c4399b3b96325c94b48d7  ·  Download CSV · JSON

Full results for this single run: the headline scores, how answers were spread out, where the model was strong or weak, and every question with the model's answer.

Pass rate
52.5%
Share of questions answered well enough to pass every check.
95% CI 37.5% – 67.1% (the true rate likely sits in this range)
Mean score
0.759
Average answer quality on a 0–1 scale, across all metrics.
40 questions · 0 error(s)
Latency
28 ms
How long the model took to answer — typical (p50) and near‑worst (p95).
p50 · p95 36 ms
Estimated cost
$0.5455
Rough API cost for this run, based on tokens used.
token-based estimate

Score distribution

How many questions landed in each quality band (0 = wrong, 1 = perfect). Most bars bunched on the right means the model answered well overall.

View data table
Score binItems
0.0-0.10
0.1-0.20
0.2-0.30
0.3-0.44
0.4-0.56
0.5-0.65
0.6-0.71
0.7-0.82
0.8-0.90
0.9-1.022

Mean score by evaluator

How each type of check graded this run (0–1). A short bar flags the quality dimension the model is weakest on.

View data table
EvaluatorMean score
exact_match0.550
llm_judge_correctness0.873
safety1.000
semantic_similarity0.681
token_f10.692

Where it's strong or weak

Pass rate split by difficulty, so a good overall score can't hide a weak spot (e.g. "95% on easy, 20% on hard").

View data table
Difficulty Items Pass rate 95% CI Mean score
easy 22 50.0% 30.7–69.3% 0.765
medium 17 52.9% 31.0–73.8% 0.739
hard 1 100.0% 20.7–100.0% 0.980

Every question

Each question with its outcome, score, and the model's answer. Use Failed to jump straight to what went wrong; open Details on a row to see the prompt, expected answer, and per‑check scores.

Item Outcome Score Latency Output
qa-003 Pass 1.000 28 ms Romeo and Juliet was written by William Shakespeare.
Details

Prompt

Answer the question accurately and concisely. Question: Who wrote the play Romeo and Juliet?

Expected

Romeo and Juliet was written by William Shakespeare.

Model output

Romeo and Juliet was written by William Shakespeare.

  • exact_match Pass 1.000
  • token_f1 Pass 1.000
  • semantic_similarity Pass 1.000
  • llm_judge_correctness Pass 1.000 judge score 10/10
  • safety Pass 1.000 no blocked terms present
qa-004 Pass 0.980 27 ms Mars is known as the Red Planet.
Details

Prompt

Answer the question accurately and concisely. Question: What planet is known as the Red Planet?

Expected

Mars is known as the Red Planet.

Model output

Mars is known as the Red Planet.

  • exact_match Pass 1.000
  • token_f1 Pass 1.000
  • semantic_similarity Pass 1.000
  • llm_judge_correctness Pass 0.900 judge score 9/10
  • safety Pass 1.000 no blocked terms present
qa-005 Pass 1.000 29 ms The Pacific Ocean is the largest ocean on Earth.
Details

Prompt

Answer the question accurately and concisely. Question: What is the largest ocean on Earth?

Expected

The Pacific Ocean is the largest ocean on Earth.

Model output

The Pacific Ocean is the largest ocean on Earth.

  • exact_match Pass 1.000
  • token_f1 Pass 1.000
  • semantic_similarity Pass 1.000
  • llm_judge_correctness Pass 1.000 judge score 10/10
  • safety Pass 1.000 no blocked terms present
qa-006 Pass 0.980 33 ms There are seven continents on Earth.
Details

Prompt

Answer the question accurately and concisely. Question: How many continents are there on Earth?

Expected

There are seven continents on Earth.

Model output

There are seven continents on Earth.

  • exact_match Pass 1.000
  • token_f1 Pass 1.000
  • semantic_similarity Pass 1.000
  • llm_judge_correctness Pass 0.900 judge score 9/10
  • safety Pass 1.000 no blocked terms present
qa-010 Pass 0.940 21 ms The smallest prime number is 2.
Details

Prompt

Answer the question accurately and concisely. Question: What is the smallest prime number?

Expected

The smallest prime number is 2.

Model output

The smallest prime number is 2.

  • exact_match Pass 1.000
  • token_f1 Pass 1.000
  • semantic_similarity Pass 1.000
  • llm_judge_correctness Pass 0.700 judge score 7/10
  • safety Pass 1.000 no blocked terms present
qa-011 Pass 0.980 17 ms The Mona Lisa was painted by Leonardo da Vinci.
Details

Prompt

Answer the question accurately and concisely. Question: Who painted the Mona Lisa?

Expected

The Mona Lisa was painted by Leonardo da Vinci.

Model output

The Mona Lisa was painted by Leonardo da Vinci.

  • exact_match Pass 1.000
  • token_f1 Pass 1.000
  • semantic_similarity Pass 1.000
  • llm_judge_correctness Pass 0.900 judge score 9/10
  • safety Pass 1.000 no blocked terms present
qa-013 Pass 0.960 20 ms A hexagon has six sides.
Details

Prompt

Answer the question accurately and concisely. Question: How many sides does a hexagon have?

Expected

A hexagon has six sides.

Model output

A hexagon has six sides.

  • exact_match Pass 1.000
  • token_f1 Pass 1.000
  • semantic_similarity Pass 1.000
  • llm_judge_correctness Pass 0.800 judge score 8/10
  • safety Pass 1.000 no blocked terms present
qa-014 Pass 0.980 17 ms Water freezes at 32 degrees Fahrenheit.
Details

Prompt

Answer the question accurately and concisely. Question: What is the freezing point of water in Fahrenheit?

Expected

Water freezes at 32 degrees Fahrenheit.

Model output

Water freezes at 32 degrees Fahrenheit.

  • exact_match Pass 1.000
  • token_f1 Pass 1.000
  • semantic_similarity Pass 1.000
  • llm_judge_correctness Pass 0.900 judge score 9/10
  • safety Pass 1.000 no blocked terms present
qa-016 Pass 0.960 28 ms The currency of Japan is the yen.
Details

Prompt

Answer the question accurately and concisely. Question: What is the currency of Japan?

Expected

The currency of Japan is the yen.

Model output

The currency of Japan is the yen.

  • exact_match Pass 1.000
  • token_f1 Pass 1.000
  • semantic_similarity Pass 1.000
  • llm_judge_correctness Pass 0.800 judge score 8/10
  • safety Pass 1.000 no blocked terms present
qa-017 Pass 1.000 30 ms Albert Einstein developed the theory of general relativity.
Details

Prompt

Answer the question accurately and concisely. Question: Who developed the theory of general relativity?

Expected

Albert Einstein developed the theory of general relativity.

Model output

Albert Einstein developed the theory of general relativity.

  • exact_match Pass 1.000
  • token_f1 Pass 1.000
  • semantic_similarity Pass 1.000
  • llm_judge_correctness Pass 1.000 judge score 10/10
  • safety Pass 1.000 no blocked terms present
qa-019 Pass 0.960 22 ms There are 120 minutes in two hours.
Details

Prompt

Answer the question accurately and concisely. Question: How many minutes are there in two hours?

Expected

There are 120 minutes in two hours.

Model output

There are 120 minutes in two hours.

  • exact_match Pass 1.000
  • token_f1 Pass 1.000
  • semantic_similarity Pass 1.000
  • llm_judge_correctness Pass 0.800 judge score 8/10
  • safety Pass 1.000 no blocked terms present
qa-020 Pass 1.000 35 ms Portuguese is the primary language spoken in Brazil.
Details

Prompt

Answer the question accurately and concisely. Question: What is the primary language spoken in Brazil?

Expected

Portuguese is the primary language spoken in Brazil.

Model output

Portuguese is the primary language spoken in Brazil.

  • exact_match Pass 1.000
  • token_f1 Pass 1.000
  • semantic_similarity Pass 1.000
  • llm_judge_correctness Pass 1.000 judge score 10/10
  • safety Pass 1.000 no blocked terms present
qa-023 Pass 1.000 23 ms Apollo 11 landed humans on the Moon in 1969.
Details

Prompt

Answer the question accurately and concisely. Question: In which year did the Apollo 11 mission land humans on the Moon?

Expected

Apollo 11 landed humans on the Moon in 1969.

Model output

Apollo 11 landed humans on the Moon in 1969.

  • exact_match Pass 1.000
  • token_f1 Pass 1.000
  • semantic_similarity Pass 1.000
  • llm_judge_correctness Pass 1.000 judge score 10/10
  • safety Pass 1.000 no blocked terms present
qa-025 Pass 0.960 25 ms Nitrogen makes up most of Earth's atmosphere.
Details

Prompt

Answer the question accurately and concisely. Question: Which gas makes up most of Earth's atmosphere?

Expected

Nitrogen makes up most of Earth's atmosphere.

Model output

Nitrogen makes up most of Earth's atmosphere.

  • exact_match Pass 1.000
  • token_f1 Pass 1.000
  • semantic_similarity Pass 1.000
  • llm_judge_correctness Pass 0.800 judge score 8/10
  • safety Pass 1.000 no blocked terms present
qa-026 Pass 0.940 36 ms Pride and Prejudice was written by Jane Austen.
Details

Prompt

Answer the question accurately and concisely. Question: Who wrote the novel Pride and Prejudice?

Expected

Pride and Prejudice was written by Jane Austen.

Model output

Pride and Prejudice was written by Jane Austen.

  • exact_match Pass 1.000
  • token_f1 Pass 1.000
  • semantic_similarity Pass 1.000
  • llm_judge_correctness Pass 0.700 judge score 7/10
  • safety Pass 1.000 no blocked terms present
qa-027 Pass 0.960 32 ms Jupiter is the largest planet in the solar system.
Details

Prompt

Answer the question accurately and concisely. Question: What is the largest planet in the solar system?

Expected

Jupiter is the largest planet in the solar system.

Model output

Jupiter is the largest planet in the solar system.

  • exact_match Pass 1.000
  • token_f1 Pass 1.000
  • semantic_similarity Pass 1.000
  • llm_judge_correctness Pass 0.800 judge score 8/10
  • safety Pass 1.000 no blocked terms present
qa-028 Pass 0.980 26 ms An adult human body has 206 bones.
Details

Prompt

Answer the question accurately and concisely. Question: How many bones does an adult human body have?

Expected

An adult human body has 206 bones.

Model output

An adult human body has 206 bones.

  • exact_match Pass 1.000
  • token_f1 Pass 1.000
  • semantic_similarity Pass 1.000
  • llm_judge_correctness Pass 0.900 judge score 9/10
  • safety Pass 1.000 no blocked terms present
qa-032 Pass 0.980 35 ms A soccer team has 11 players on the field.
Details

Prompt

Answer the question accurately and concisely. Question: How many players are on a soccer team on the field?

Expected

A soccer team has 11 players on the field.

Model output

A soccer team has 11 players on the field.

  • exact_match Pass 1.000
  • token_f1 Pass 1.000
  • semantic_similarity Pass 1.000
  • llm_judge_correctness Pass 0.900 judge score 9/10
  • safety Pass 1.000 no blocked terms present
qa-034 Pass 0.980 29 ms The Indian Ocean lies between Africa and Australia.
Details

Prompt

Answer the question accurately and concisely. Question: Which ocean lies between Africa and Australia?

Expected

The Indian Ocean lies between Africa and Australia.

Model output

The Indian Ocean lies between Africa and Australia.

  • exact_match Pass 1.000
  • token_f1 Pass 1.000
  • semantic_similarity Pass 1.000
  • llm_judge_correctness Pass 0.900 judge score 9/10
  • safety Pass 1.000 no blocked terms present
qa-038 Pass 0.940 35 ms Diamond is the hardest natural substance on Earth.
Details

Prompt

Answer the question accurately and concisely. Question: What is the hardest natural substance on Earth?

Expected

Diamond is the hardest natural substance on Earth.

Model output

Diamond is the hardest natural substance on Earth.

  • exact_match Pass 1.000
  • token_f1 Pass 1.000
  • semantic_similarity Pass 1.000
  • llm_judge_correctness Pass 0.700 judge score 7/10
  • safety Pass 1.000 no blocked terms present
qa-040 Pass 0.980 17 ms The interior angles of a triangle sum to 180 degrees.
Details

Prompt

Answer the question accurately and concisely. Question: What is the sum of the interior angles of a triangle?

Expected

The interior angles of a triangle sum to 180 degrees.

Model output

The interior angles of a triangle sum to 180 degrees.

  • exact_match Pass 1.000
  • token_f1 Pass 1.000
  • semantic_similarity Pass 1.000
  • llm_judge_correctness Pass 0.900 judge score 9/10
  • safety Pass 1.000 no blocked terms present