nightly-eval · sim-small-v1 simulated

Completed   dataset /home/evalpipe/demo_dataset.jsonl   started 2026-07-13 01:08:35 UTC   run id be4ff9c1852c4399b3b96325c94b48d7  ·  Download CSV · JSON

Full results for this single run: the headline scores, how answers were spread out, where the model was strong or weak, and every question with the model's answer.

Pass rate
52.5%
Share of questions answered well enough to pass every check.
95% CI 37.5% – 67.1% (the true rate likely sits in this range)
Mean score
0.759
Average answer quality on a 0–1 scale, across all metrics.
40 questions · 0 error(s)
Latency
28 ms
How long the model took to answer — typical (p50) and near‑worst (p95).
p50 · p95 36 ms
Estimated cost
$0.5455
Rough API cost for this run, based on tokens used.
token-based estimate

Score distribution

How many questions landed in each quality band (0 = wrong, 1 = perfect). Most bars bunched on the right means the model answered well overall.

View data table
Score binItems
0.0-0.10
0.1-0.20
0.2-0.30
0.3-0.44
0.4-0.56
0.5-0.65
0.6-0.71
0.7-0.82
0.8-0.90
0.9-1.022

Mean score by evaluator

How each type of check graded this run (0–1). A short bar flags the quality dimension the model is weakest on.

View data table
EvaluatorMean score
exact_match0.550
llm_judge_correctness0.873
safety1.000
semantic_similarity0.681
token_f10.692

Where it's strong or weak

Pass rate split by difficulty, so a good overall score can't hide a weak spot (e.g. "95% on easy, 20% on hard").

View data table
Difficulty Items Pass rate 95% CI Mean score
easy 22 50.0% 30.7–69.3% 0.765
medium 17 52.9% 31.0–73.8% 0.739
hard 1 100.0% 20.7–100.0% 0.980

Every question

Each question with its outcome, score, and the model's answer. Use Failed to jump straight to what went wrong; open Details on a row to see the prompt, expected answer, and per‑check scores.

Item Outcome Score Latency Output
qa-001 Fail 0.474 31 ms That is outside the scope of what I can determine from the prompt.
Details

Prompt

Answer the question accurately and concisely. Question: What is the capital of France?

Expected

The capital of France is Paris.

Model output

That is outside the scope of what I can determine from the prompt.

  • exact_match Fail 0.000
  • token_f1 Fail 0.250
  • semantic_similarity Fail 0.219
  • llm_judge_correctness Pass 0.900 judge score 9/10
  • safety Pass 1.000 no blocked terms present
qa-002 Fail 0.400 27 ms That is outside the scope of what I can determine from the prompt.
Details

Prompt

Answer the question accurately and concisely. Question: What is the boiling point of water at sea level in Celsius?

Expected

Water boils at 100 degrees Celsius at sea level.

Model output

That is outside the scope of what I can determine from the prompt.

  • exact_match Fail 0.000
  • token_f1 Fail 0.000
  • semantic_similarity Fail 0.000
  • llm_judge_correctness Pass 1.000 judge score 10/10
  • safety Pass 1.000 no blocked terms present
qa-007 Fail 0.700 31 ms Plants absorb carbon dioxide
Details

Prompt

Answer the question accurately and concisely. Question: What gas do plants absorb during photosynthesis?

Expected

Plants absorb carbon dioxide during photosynthesis.

Model output

Plants absorb carbon dioxide

  • exact_match Fail 0.000
  • token_f1 Pass 0.800
  • semantic_similarity Pass 0.798
  • llm_judge_correctness Pass 0.900 judge score 9/10
  • safety Pass 1.000 no blocked terms present
qa-008 Fail 0.453 23 ms That is outside the scope of what I can determine from the prompt.
Details

Prompt

Answer the question accurately and concisely. Question: What is the chemical symbol for gold?

Expected

The chemical symbol for gold is Au.

Model output

That is outside the scope of what I can determine from the prompt.

  • exact_match Fail 0.000
  • token_f1 Fail 0.118
  • semantic_similarity Fail 0.147
  • llm_judge_correctness Pass 1.000 judge score 10/10
  • safety Pass 1.000 no blocked terms present
qa-009 Fail 0.533 27 ms are
Details

Prompt

Answer the question accurately and concisely. Question: Which country is home to the kangaroo?

Expected

Kangaroos are native to Australia.

Model output

are

  • exact_match Fail 0.000
  • token_f1 Fail 0.333
  • semantic_similarity Fail 0.333
  • llm_judge_correctness Pass 1.000 judge score 10/10
  • safety Pass 1.000 no blocked terms present
qa-012 Fail 0.733 35 ms The is considered the longest river in the world.
Details

Prompt

Answer the question accurately and concisely. Question: What is the longest river in the world?

Expected

The Nile is generally considered the longest river in the world.

Model output

The is considered the longest river in the world.

  • exact_match Fail 0.000
  • token_f1 Pass 0.857
  • semantic_similarity Pass 0.808
  • llm_judge_correctness Pass 1.000 judge score 10/10
  • safety Pass 1.000 no blocked terms present
qa-015 Fail 0.413 34 ms The answer depends on several factors that are not specified here.
Details

Prompt

Answer the question accurately and concisely. Question: Which element has the atomic number 1?

Expected

Hydrogen has the atomic number 1.

Model output

The answer depends on several factors that are not specified here.

  • exact_match Fail 0.000
  • token_f1 Fail 0.000
  • semantic_similarity Fail 0.066
  • llm_judge_correctness Pass 1.000 judge score 10/10
  • safety Pass 1.000 no blocked terms present
qa-018 Fail 0.516 36 ms Mount mountain above
Details

Prompt

Answer the question accurately and concisely. Question: What is the tallest mountain above sea level?

Expected

Mount Everest is the tallest mountain above sea level.

Model output

Mount mountain above

  • exact_match Fail 0.000
  • token_f1 Fail 0.545
  • semantic_similarity Fail 0.434
  • llm_judge_correctness Fail 0.600 judge score 6/10
  • safety Pass 1.000 no blocked terms present
qa-021 Fail 0.563 35 ms heart pumps through
Details

Prompt

Answer the question accurately and concisely. Question: Which organ pumps blood through the human body?

Expected

The heart pumps blood through the human body.

Model output

heart pumps through

  • exact_match Fail 0.000
  • token_f1 Pass 0.667
  • semantic_similarity Fail 0.449
  • llm_judge_correctness Pass 0.700 judge score 7/10
  • safety Pass 1.000 no blocked terms present
qa-022 Fail 0.372 34 ms The answer depends on several factors that are not specified here.
Details

Prompt

Answer the question accurately and concisely. Question: What is the square root of 144?

Expected

The square root of 144 is 12.

Model output

The answer depends on several factors that are not specified here.

  • exact_match Fail 0.000
  • token_f1 Fail 0.000
  • semantic_similarity Fail 0.061
  • llm_judge_correctness Pass 0.800 judge score 8/10
  • safety Pass 1.000 no blocked terms present
qa-024 Fail 0.739 23 ms The capital of Japan is
Details

Prompt

Answer the question accurately and concisely. Question: What is the capital of Japan?

Expected

The capital of Japan is Tokyo.

Model output

The capital of Japan is

  • exact_match Fail 0.000
  • token_f1 Pass 0.889
  • semantic_similarity Pass 0.905
  • llm_judge_correctness Pass 0.900 judge score 9/10
  • safety Pass 1.000 no blocked terms present
qa-029 Fail 0.400 31 ms I do not have enough information to answer that question.
Details

Prompt

Answer the question accurately and concisely. Question: What is the capital of Canada?

Expected

The capital of Canada is Ottawa.

Model output

I do not have enough information to answer that question.

  • exact_match Fail 0.000
  • token_f1 Fail 0.000
  • semantic_similarity Fail 0.000
  • llm_judge_correctness Pass 1.000 judge score 10/10
  • safety Pass 1.000 no blocked terms present
qa-030 Fail 0.400 23 ms The answer depends on several factors that are not specified here.
Details

Prompt

Answer the question accurately and concisely. Question: Which metal is liquid at room temperature?

Expected

Mercury is liquid at room temperature.

Model output

The answer depends on several factors that are not specified here.

  • exact_match Fail 0.000
  • token_f1 Fail 0.000
  • semantic_similarity Fail 0.000
  • llm_judge_correctness Pass 1.000 judge score 10/10
  • safety Pass 1.000 no blocked terms present
qa-031 Fail 0.380 29 ms That is outside the scope of what I can determine from the prompt.
Details

Prompt

Answer the question accurately and concisely. Question: What does DNA stand for?

Expected

DNA stands for deoxyribonucleic acid.

Model output

That is outside the scope of what I can determine from the prompt.

  • exact_match Fail 0.000
  • token_f1 Fail 0.000
  • semantic_similarity Fail 0.000
  • llm_judge_correctness Pass 0.900 judge score 9/10
  • safety Pass 1.000 no blocked terms present
qa-033 Fail 0.587 18 ms The is the land
Details

Prompt

Answer the question accurately and concisely. Question: What is the fastest land animal?

Expected

The cheetah is the fastest land animal.

Model output

The is the land

  • exact_match Fail 0.000
  • token_f1 Fail 0.571
  • semantic_similarity Pass 0.562
  • llm_judge_correctness Pass 0.800 judge score 8/10
  • safety Pass 1.000 no blocked terms present
qa-035 Fail 0.371 24 ms The answer depends on several factors that are not specified here.
Details

Prompt

Answer the question accurately and concisely. Question: What is the chemical formula for table salt?

Expected

The chemical formula for table salt is NaCl.

Model output

The answer depends on several factors that are not specified here.

  • exact_match Fail 0.000
  • token_f1 Fail 0.000
  • semantic_similarity Fail 0.056
  • llm_judge_correctness Pass 0.800 judge score 8/10
  • safety Pass 1.000 no blocked terms present
qa-036 Fail 0.578 19 ms Washington was first of
Details

Prompt

Answer the question accurately and concisely. Question: Who was the first President of the United States?

Expected

George Washington was the first President of the United States.

Model output

Washington was first of

  • exact_match Fail 0.000
  • token_f1 Pass 0.667
  • semantic_similarity Fail 0.424
  • llm_judge_correctness Pass 0.800 judge score 8/10
  • safety Pass 1.000 no blocked terms present
qa-037 Fail 0.380 37 ms The answer depends on several factors that are not specified here.
Details

Prompt

Answer the question accurately and concisely. Question: How many colors are there in a rainbow?

Expected

A rainbow has seven colors.

Model output

The answer depends on several factors that are not specified here.

  • exact_match Fail 0.000
  • token_f1 Fail 0.000
  • semantic_similarity Fail 0.000
  • llm_judge_correctness Pass 0.900 judge score 9/10
  • safety Pass 1.000 no blocked terms present
qa-039 Fail 0.920 29 ms Mercury is the planet closest to the Sun.
Details

Prompt

Answer the question accurately and concisely. Question: Which planet is closest to the Sun?

Expected

Mercury is the planet closest to the Sun.

Model output

Mercury is the planet closest to the Sun.

  • exact_match Pass 1.000
  • token_f1 Pass 1.000
  • semantic_similarity Pass 1.000
  • llm_judge_correctness Fail 0.600 judge score 6/10
  • safety Pass 1.000 no blocked terms present