<div dir="ltr"><div><div class="gmail_default" style=""><font face="georgia, serif" style="" color="#000000"><font style="vertical-align:inherit"><font style="vertical-align:inherit"><b>When:</b>    </font></font><font style="vertical-align:inherit"><font style="vertical-align:inherit">    Wednesday, October 22nd at <b style="background-color:rgb(255,255,0)">11am CT</b><b> </b></font></font></font></div><div class="gmail_default"><font face="georgia, serif" color="#000000"><font style="vertical-align:inherit"><font style="vertical-align:inherit"><b><br></b></font></font></font></div><div class="gmail_default"><font face="georgia, serif" color="#000000"></font></div><div class="gmail_default"><font face="georgia, serif" color="#000000"><b>Where:       </b><span class="gmail-il">Talk</span> will be given <font style="font-weight:bold"><u>live, in-person</u></font><font style="font-weight:bold"> </font>at</font></div><p class="MsoNormal" style="margin:0in;line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="georgia, serif" color="#000000">                       <span class="gmail-il">TTIC</span>, 6045 S. Kenwood Avenue</font></p><p class="MsoNormal" style="margin:0in;line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font color="#000000" face="georgia, serif">                       5th Floor, Room 530<b> </b></font></p><p class="MsoNormal" style="margin:0in;line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><b><font face="georgia, serif" color="#000000"><br></font></b></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="georgia, serif" color="#000000"><b style="letter-spacing:0.2px">Virtually:</b><span style="letter-spacing:0.2px">  via Panopto </span>(<a href="https://uchicago.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=fa7cab9f-5be7-40c1-b739-b34d0127a3af" target="_blank">livestream</a><span style="letter-spacing:0.2px">)</span><br></font></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="georgia, serif" color="#000000"><br></font></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="georgia, serif" color="#000000"><font style="vertical-align:inherit"><font style="vertical-align:inherit"><b>Who: </b>         </font></font><span class="gmail_default"></span><span class="gmail_default"></span><span class="gmail_default" style="">Keyon Vafa, Harvard University </span></font></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"></p><div><p style="letter-spacing:0.2px"><font face="georgia, serif" color="#000000"><span style="letter-spacing:normal"><b>Title:</b>          </span><span style="letter-spacing:normal"><span class="gmail_default"></span></span><span style="letter-spacing:normal"><span class="gmail_default"></span></span><span style="letter-spacing:normal"><span class="gmail_default" style=""></span></span><span style="letter-spacing:normal">Evaluating the Implicit World Models of Generative Models</span></font></p><div><font face="georgia, serif" color="#000000"><b>Abstract:  </b>The challenge of evaluation is making conclusions about a model's capabilities from a small amount of data. While there are many benchmarks that allow us to quantify a model's performance on different types of tasks, it is unclear how to turn these results into robust conclusions about a model's understanding or its capabilities. This talk will propose theoretically-grounded definitions and metrics that test for a model's implicit understanding, or its world model. We will focus on two settings: one where models are designed to perform a single task, and another where a foundation model is intended to perform many tasks. These exercises demonstrate that models can make highly accurate predictions with incoherent world models, revealing their fragility.</font></div></div><div><div><font face="georgia, serif" color="#000000"><br></font></div><div><font face="georgia, serif" color="#000000"><b>Bio</b>: Keyon Vafa is a postdoctoral fellow at Harvard University. His research focuses on developing new evaluation methodology in order to evaluate and improve generative models in AI. Keyon completed his PhD in computer science from Columbia University, where he was an NSF GRFP Fellow and the recipient of the Morton B. Friedman Memorial Prize for excellence in engineering. He organized the NeurIPS 2024 Workshop on Behavioral Machine Learning and the ICML 2025 Workshop on Assessing World Models, and he is a member of the early career board of the Harvard Data Science Review.</font></div><div><font face="georgia, serif" color="#000000"><br></font></div></div><div><font face="georgia, serif" color="#000000"><b>Host:<span class="gmail_default"> <a href="mailto:shiry@ttic.edu" target="_blank">Shiry Ginosar</a></span></b></font></div><font color="#888888"><br clear="all"></font><br clear="all"></div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><b style="background-color:rgb(255,255,255)"><font color="#3d85c6">Brandie Jones </font></b><div><div><div><font color="#3d85c6"><b><i>Executive </i></b></font><b style="color:rgb(61,133,198)"><i>Administrative Assistant</i></b></div></div><div><b style="color:rgb(61,133,198)"><i>Outreach Administrator </i></b></div><div><span style="background-color:rgb(255,255,255)"><font color="#3d85c6">Toyota Technological Institute</font></span></div><div><span style="background-color:rgb(255,255,255)"><font color="#3d85c6">6045 S. Kenwood Avenue</font></span></div><div><span style="background-color:rgb(255,255,255)"><font color="#3d85c6">Chicago, IL  60637</font></span></div></div><div><span style="background-color:rgb(255,255,255)"><font color="#3d85c6"><a href="http://www.ttic.edu" target="_blank">www.ttic.edu</a> </font></span></div><div><span style="background-color:rgb(255,255,255)"><font color="#3d85c6"><br></font></span></div></div></div></div></div>