[Theory] [Talks at TTIC] 4/7 TTIC Colloquium: Sanmi Koyejo, Stanford

[Brandie Jones via Theory]

*When:*        Monday, April 7th at *11:30am CT*

*Where:       *Talk will be given *live, in-person* at

                       TTIC, 6045 S. Kenwood Avenue

                       5th Floor, Room 530


*Virtually:*  via Panopto (livestream
<https://uchicago.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=b0137928-cd56-44de-911e-b28e01151565>
)


*Who: *         Sanmi Koyejo, Stanford

*Title:*          Beyond Benchmarks: Building a Science of AI Measurement
*Abstract:  *The widespread deployment of AI systems in critical domains
demands more rigorous approaches to evaluating their capabilities and
safety. While current evaluation practices rely on static benchmarks, these
methods face fundamental efficiency, reliability, and real-world relevance
challenges. This talk presents a path toward a measurement framework that
bridges established psychometric principles with modern AI evaluation
needs. We demonstrate how techniques from Item Response Theory, amortized
computation, and predictability analysis can substantially improve the
rigor and efficiency of AI evaluation. Through case studies, we show how
this approach can enable more reliable, scalable, and meaningful evaluation
of AI systems. This work points toward a broader vision: evolving AI
evaluation from a collection of benchmarks into a rigorous measurement
science that can effectively guide research, deployment, and policy
decisions.

*Short Bio*: Sanmi Koyejo is an assistant professor in the Department of
Computer Science at Stanford University and a co-founder of Virtue AI. At
Stanford, Koyejo leads the Stanford Trustworthy Artificial Intelligence
Research (STAIR) lab, which works to develop the principles and practice of
safe and secure AI. Koyejo has received several awards, including a Skip
Ellis Early Career Award, a Sloan Fellowship, and a PECASE. Koyejo serves
on the Neural Information Processing Systems Foundation Board and as
president of Black in AI.


*Host: Matt Walter <mwalter at ttic.edu> & Jingyan Wang <jingyanw at ttic.edu>*


-- 
*Brandie Jones *
*Executive **Administrative Assistant*
Toyota Technological Institute
6045 S. Kenwood Avenue
Chicago, IL  60637
www.ttic.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cs.uchicago.edu/pipermail/theory/attachments/20250331/ee3b4312/attachment.html>