Back to the live feed
Research
How to Test AI Agents When No Benchmark Exists — Nathanael Weill, Air Canada｜AI Engineer
AI-generated summaryA model wrote this from the publisher’s headline and short excerpt. It can leave out context or contain errors, so read the original report before relying on it. How summaries are made
Summary
Air Canada AI engineer Nathanael Weill says teams can evaluate AI agents without a public benchmark by creating test questions from known answers and checking results with code. For the airline’s supplier-contract agent, his team built 275 rule-based tests and used generated contract examples and checks that changes such as a new supplier name do not alter the flags. Evaluation grew from one test a week to dozens a day in an earlier project, but Weill says generated tests support development rather than decide whether a system is ready to launch. He warns that AI judges can be too unreliable to grade agents.