Public tests. Same questions. Every answer graded.
We test Amber on public benchmarks against other AIs — and publish the results, good or bad.
Same 100 questions. Same grader. Every AI.
SimpleQA is a public factuality benchmark — hard, specific questions with known answers. It measures what most tests ignore: not just how often an AI is right, but how often it makes something up instead of admitting it doesn't know. Same 100 questions, same independent grader (GPT-4o), for every model.
| Model | Got it right | Made something up | Said "I don't know" | Calibration (F) |
|---|---|---|---|---|
| Amber | 37% | 21% | 42% | 0.468 |
| ChatGPT (GPT-4o) | 44% | 55% | 1% | 0.442 |
| The same model, without Amber | 32% | 57% | 11% | 0.339 |
The headline. On the same questions, Amber makes up ~2.6× fewer answers than either baseline — and on calibration (the balance of knowing what it knows), Amber scores highest, ahead of GPT-4o.
Honest tradeoff. ChatGPT answers more questions correctly overall (44% vs 37%) because it answers almost everything. Amber answers fewer — but when it does answer, it's right 64% of its attempts vs ChatGPT's 44%, and rarely makes things up. We'd rather Amber say "I don't know" than guess wrong.
Checked three ways. Amber's made-up rate held at 21–22% across three independent graders, including one from a different company — so the result isn't a quirk of who graded it.
Human proof. No stats needed.
Two confident, different wrong answers. One honest "I don't know."
So you can check.
- Public dataset, unmodified. 100 questions, fixed random seed.
- Same questions, same grader, for every AI.
- Validated across three independent graders.
- Every answer and every grade is logged.