Transcript
Old tests can become too easy. In 2026, AI evaluation is shifting towards real tasks, errors and judgement. Better testing asks what happens outside the exam.
Three key points
- Old tests can become too easy to tell systems apart.
- Evaluation is shifting towards real tasks, errors and judgement.
- Better testing asks what happens outside the exam.
Measuring what matters
A test is only useful while it is hard enough to separate good work from adequate work. Once every system scores near the top, the number stops telling anyone much. Nicole Junkermann sees the more thoughtful evaluation work moving towards tasks that look like real jobs: longer problems with messy inputs, where the interesting question is not only whether the answer is right but how the system behaves when it is wrong. Does it notice its own mistake, ask for help or carry on confidently? For anyone weighing up a new technology, those are the details that matter in practice, and they sit much closer to how people judge each other's work than any single exam score.
More videos
- Spotting Gaps in a Plan
- Finding Common Themes in Feedback
- Why AI Interoperability Matters
- AI on Your Own Device
- Checking an AI Match Report
Find all six clips, the tech notes and the reading list on the Nicole Junkermann homepage.