Evaluation dashboard

See how your AI models perform over time. Each run sends a fixed set of test questions to a model and grades every answer — the cards below summarise your most recent run and your totals.

Latest pass rate

82.5%

▲ +10.0 pts vs the run before

Share of the newest run's questions the model answered well enough to pass every check. Higher is better.

Newest run: ab-candidate · model sim-large-v1 simulated

Latest mean score

0.918

Average answer quality on a 0–1 scale (1 = perfect), averaged across every metric.

Across 40 test questions

Evaluations run

How many times you've tested a model against a question set so far.

280 questions graded in total

Estimated spend

$3.72

Rough model/API cost of all runs, based on the number of tokens used.

Across all completed runs

Pass rate over runs

Each dot is one evaluation, oldest to newest. A rising line means later models passed more of the test questions. Hover a dot for details.

View data table

Run	Pass rate
nightly-eval · sim-small-v1	37.5%
nightly-eval · sim-small-v1	52.5%
nightly-eval · sim-small-v2	70.0%
nightly-eval · sim-small-v2	67.5%
nightly-eval · sim-small-v3	62.5%
ab-baseline · sim-small-v3	72.5%
ab-candidate · sim-large-v1	82.5%

Score distribution — latest run

Per-item mean score for ab-candidate · sim-large-v1, 10 bins. Open run →

View data table

Score bin	Items
0.0-0.1	0
0.1-0.2	0
0.2-0.3	0
0.3-0.4	0
0.4-0.5	2
0.5-0.6	0
0.6-0.7	3
0.7-0.8	1
0.8-0.9	0
0.9-1.0	34

Mean score by evaluator — latest run

How each metric graded ab-candidate, on a 0–1 scale.

View data table

Evaluator	Mean score
exact_match	0.850
llm_judge_correctness	0.875
safety	1.000
semantic_similarity	0.926
token_f1	0.941

Recent runs

Open a run for per-item results and score distributions.

Run	Model	Status	Items	Pass rate	Mean score	p95 latency	Est. cost
ab-candidate	sim-large-v1 simulated	Completed	40	82.5%	0.918	36 ms	$0.5105
ab-baseline	sim-small-v3 simulated	Completed	40	72.5%	0.841	36 ms	$0.5180
nightly-eval	sim-small-v3 simulated	Completed	40	62.5%	0.819	36 ms	$0.5430
nightly-eval	sim-small-v2 simulated	Completed	40	67.5%	0.821	35 ms	$0.5667
nightly-eval	sim-small-v2 simulated	Completed	40	70.0%	0.856	37 ms	$0.5180
nightly-eval	sim-small-v1 simulated	Completed	40	52.5%	0.759	36 ms	$0.5455
nightly-eval	sim-small-v1 simulated	Completed	40	37.5%	0.716	38 ms	$0.5142

Compare models in real time

Open full playground →

Send one prompt to OpenAI, Anthropic, Gemini or a local model and score every output live with the metric suite — no run required.

OpenAI · ChatGPT OPENAI_API_KEY not set Anthropic ANTHROPIC_API_KEY not set Google Gemini · free tier GEMINI_API_KEY not set Groq · free tier GROQ_API_KEY not set OpenRouter · free models OPENROUTER_API_KEY not set Ollama · local, free OpenAI-compatible · proxy Simulation · offline

Prompt

Reference answer (optional — enables reference-based metrics)

Metrics (scored on every model's output)

Exact match Token F1 Semantic similarity Contains reference

Model A

Provider

Model id

Simulated quality (0–1)

Model B

Provider

Model id

Simulated quality (0–1)

Model C optional

Provider

Model id

Simulated quality (0–1)

Paste a key to test a hosted model right away — it's sent to your server for this one request only, never stored, logged, or returned. Leave it blank to use the server-side environment variable instead.