Evals in the wild: a field guide
A short field guide to evaluating systems that have already left the lab — where the clean benchmark ends and the messy real distribution begins.
Notes
Benchmarks measure the lab. Production measures the world. This field guide is about the gap between them — and why the interesting failures only show up once real users arrive with distributions no benchmark anticipated.