<div dir="ltr"><div dir="ltr"><div class="gmail_default" style=""><div class="gmail_default" style=""><div class="gmail_default" style=""><div class="gmail_default" style="font-size:small;color:rgb(80,0,80)"><font style="font-family:arial,sans-serif;color:rgb(0,0,0);vertical-align:inherit"><font style="vertical-align:inherit"><b>When:</b> </font></font><font style="font-family:arial,sans-serif;vertical-align:inherit"><font style="vertical-align:inherit"><font style="color:rgb(0,0,0)"> Fri</font><span class="gmail_default" style="color:rgb(0,0,0)">day, April 12<span class="gmail_default">, </span>2024</span><font style="color:rgb(0,0,0)"> at</font><b style="color:rgb(0,0,0)"> <span style="background-color:rgb(255,255,0)"><u>11:00</u></span></b><b><u><font color="#000000" style="background-color:rgb(255,255,0)"> am</font></u></b><b><u><font color="#000000" style="background-color:rgb(255,255,0)"> CT</font></u><font color="#000000"><u> </u> </font></b></font></font><br></div><div class="gmail_default" style=""><p style="font-size:small;color:rgb(80,0,80);font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;margin:0px"><b style="font-family:arial,sans-serif"><font color="#500050"><br></font></b></p><p style="font-size:small;color:rgb(80,0,80);font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;margin:0px"><b style="font-family:arial,sans-serif"><font color="#500050">Where: </font></b><font color="#000000" style="font-family:arial,sans-serif">Talk will be given </font><font color="#000000" style="font-family:arial,sans-serif;font-weight:bold"><u>live, in-person</u></font><font style="font-family:arial,sans-serif;font-weight:bold"> </font><span style="font-family:arial,sans-serif">at</span><br></p><p class="MsoNormal" style="font-size:small;margin:0in;color:rgb(80,0,80);line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="arial, sans-serif"><font color="#500050"> </font><font color="#000000"> TTIC, 6045 S. Kenwood Avenue</font></font></p><p class="MsoNormal" style="font-size:small;margin:0in;color:rgb(80,0,80);line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="arial, sans-serif" color="#000000"> 5th Floor, <b>Room 529 </b></font></p><p class="MsoNormal" style="margin:0in;color:rgb(80,0,80);line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><span style="background-color:rgb(255,255,255)"><b style="font-size:14px;color:rgb(60,64,67);font-family:Roboto,Helvetica,Arial,sans-serif">Virtually:</b><span style="font-size:14px;color:rgb(60,64,67);font-family:Roboto,Helvetica,Arial,sans-serif"> </span><a href="https://www.google.com/url?q=https://uchicago.zoom.us/j/98297764499?pwd%3DajNQSTZnMHRmMENkd1hjdjlNeW1xdz09&sa=D&source=calendar&ust=1713290314655199&usg=AOvVaw2G35njOZgShzvSRt7jxB2X" target="_blank" style="color:rgb(26,115,232)"><b style=""><font face="arial, sans-serif" style="">zoom</font></b></a></span></p><p class="MsoNormal" style="font-size:small;margin:0in;color:rgb(80,0,80);line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><b style="font-family:georgia,serif;color:rgb(60,64,67);letter-spacing:0.2px"><font size="1" style="background-color:rgb(255,255,255)"> </font></b><br></p><p class="MsoNormal" style="font-size:small;margin:0in 0in 0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font style="color:rgb(80,0,80);vertical-align:inherit"><font style="vertical-align:inherit"><b style="font-family:arial,sans-serif">Who: </b><font face="arial, sans-serif"> </font><font color="#500050" style="font-family:arial,sans-serif"> </font><font color="#000000"><font color="#500050"><font face="arial, sans-serif"> </font></font></font></font></font><font face="arial, sans-serif">Hao Peng, UIUC</font></p></div></div><div class="gmail_default" style="font-size:small"><div dir="ltr"><div><div class="MsoNormal" align="center" style="margin:0in 0in 8pt;text-align:center;line-height:15.6933px;font-size:11pt;font-family:Calibri,sans-serif"><hr size="3" width="100%" noshade align="center" style="color:rgb(46,116,181)"></div></div><div><div dir="ltr"><font face="arial, sans-serif"><b style="color:rgb(60,64,67);letter-spacing:0.2px">Title:</b><span style="color:rgb(60,64,67);letter-spacing:0.2px"> Pushing the Boundaries of Length Generalization and Reasoning Capabilities of Open LLMs</span><span style="letter-spacing:0.2px"><br style="color:rgb(60,64,67);letter-spacing:0.2px"><br style="color:rgb(60,64,67);letter-spacing:0.2px"></span><b style="color:rgb(60,64,67);letter-spacing:0.2px">Abstract: </b><span style="color:rgb(60,64,67);letter-spacing:0.2px">Recent advancements in open-source pretrained large language models (LLMs) have created new opportunities for exploring exciting post-pre training innovations. This talk shares some of our recent works. The first part of my talk focuses on context length generalization in LLMs. I will begin with a theoretical analysis that identifies major factors contributing to the failures of several commonly-used techniques, which leads to the development of a simple yet effective algorithm that enables pretrained LLMs to generalize to extreme context lengths without any parameter update. I will then shift focus to continual pretraining for length generalization, and share our recent findings highlighting the importance of training data mixture—a crucial yet previously overlooked factor. The second part of my talk will be about Eurus, our recently-released suite of open LLMs. On diverse benchmarks covering challenging math, coding, and reasoning problems, Eurus achieves state-of-the-art performance among all open-source models and outperforms GPT-3.5 Turbo. On two established reward modeling benchmarks, our 7B reward model achieves better correlation with human judgment than all existing models including GPT-4. I will especially highlight UltraInteract, our newly-curated alignment dataset that enables Eurus’s strong performance.</span><span style="letter-spacing:0.2px"><br style="color:rgb(60,64,67);letter-spacing:0.2px"><br style="color:rgb(60,64,67);letter-spacing:0.2px"></span><b style="color:rgb(60,64,67);letter-spacing:0.2px">Bio: </b><span style="color:rgb(60,64,67);letter-spacing:0.2px">Hao Peng is an Assistant Professor in the Department of Computer Science of the University of Illinois at Urbana-Champaign (UIUC). He received his PhD from the University of Washington and his bachelor’s degree from Peking University. Before joining UIUC, he spent one year at the Allen Institute for Artificial Intelligence as a Young Investigator, and time at Microsoft Research, Google, and DeepMind as an intern. His research interest broadly spans natural language processing and machine learning.</span></font><br></div><div dir="ltr"><br></div><b> Host: </b><a href="mailto:jzhou@ttic.edu" target="_blank">Jiawei Zhou</a></div></div></div></div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small"><br></div></div><div><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><span style="font-family:arial,helvetica,sans-serif;font-size:x-small">Mary C. Marre</span><br></div><div><div><font face="arial, helvetica, sans-serif" size="1">Faculty Administrative Support</font></div><div><i><font face="arial, helvetica, sans-serif" color="#3d85c6" size="1"><b>Toyota Technological Institute</b></font></i></div><div><i><font face="arial, helvetica, sans-serif" color="#3d85c6" size="1">6045 S. Kenwood Avenue, Rm 517</font></i></div><div><font size="1"><i><font face="arial, helvetica, sans-serif" color="#3d85c6">Chicago, IL 60637</font></i><br></font></div><div><font size="1"><i><font face="arial, helvetica, sans-serif" color="#3d85c6">773-834-1757</font></i></font></div><div><b><i><a href="mailto:mmarre@ttic.edu" target="_blank"><font face="arial, helvetica, sans-serif" size="1">mmarre@ttic.edu</font></a></i></b></div></div></div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Apr 9, 2024 at 5:26 PM Mary Marre <<a href="mailto:mmarre@ttic.edu">mmarre@ttic.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div><div style="font-size:small;color:rgb(80,0,80)"><font style="font-family:arial,sans-serif;color:rgb(0,0,0);vertical-align:inherit"><font style="vertical-align:inherit"><b>When:</b> </font></font><font style="font-family:arial,sans-serif;vertical-align:inherit"><font style="vertical-align:inherit"><font style="color:rgb(0,0,0)"> Fri</font><span class="gmail_default" style="color:rgb(0,0,0)">day, April 12<span class="gmail_default">, </span>2024</span><font style="color:rgb(0,0,0)"> at</font><b style="color:rgb(0,0,0)"> <span style="background-color:rgb(255,255,0)"><u>11:00</u></span></b><b><u><font color="#000000" style="background-color:rgb(255,255,0)"> am</font></u></b><b><u><font color="#000000" style="background-color:rgb(255,255,0)"> CT</font></u><font color="#000000"><u> </u> </font></b></font></font><br></div><div><p style="font-size:small;color:rgb(80,0,80);font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;margin:0px"><b style="font-family:arial,sans-serif"><font color="#500050"><br></font></b></p><p style="font-size:small;color:rgb(80,0,80);font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;margin:0px"><b style="font-family:arial,sans-serif"><font color="#500050">Where: </font></b><font color="#000000" style="font-family:arial,sans-serif">Talk will be given </font><font color="#000000" style="font-family:arial,sans-serif;font-weight:bold"><u>live, in-person</u></font><font style="font-family:arial,sans-serif;font-weight:bold"> </font><span style="font-family:arial,sans-serif">at</span><br></p><p class="MsoNormal" style="font-size:small;margin:0in;color:rgb(80,0,80);line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="arial, sans-serif"><font color="#500050"> </font><font color="#000000"> TTIC, 6045 S. Kenwood Avenue</font></font></p><p class="MsoNormal" style="font-size:small;margin:0in;color:rgb(80,0,80);line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="arial, sans-serif" color="#000000"> 5th Floor, <b>Room 529 </b></font></p><p class="MsoNormal" style="font-size:small;margin:0in;color:rgb(80,0,80);line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><b style="font-family:georgia,serif;color:rgb(60,64,67);letter-spacing:0.2px"><font size="1"> </font></b><br></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font style="color:rgb(80,0,80);vertical-align:inherit"><font style="vertical-align:inherit"><b style="font-size:small;font-family:arial,sans-serif">Who: </b><font face="arial, sans-serif" style="font-size:small"> </font><font color="#500050" style="font-size:small;font-family:arial,sans-serif"> </font><font color="#000000"><font color="#500050"><font face="arial, sans-serif"> </font></font></font></font></font><font face="arial, sans-serif">Hao Peng, UIUC</font></p></div></div><div><div dir="ltr"><div style="font-size:small"><div class="MsoNormal" align="center" style="margin:0in 0in 8pt;text-align:center;line-height:15.6933px;font-size:11pt;font-family:Calibri,sans-serif"><hr size="3" width="100%" noshade align="center" style="color:rgb(46,116,181)"></div></div><div><div dir="ltr"><font face="arial, sans-serif"><b style="color:rgb(60,64,67);letter-spacing:0.2px">Title:</b><span style="color:rgb(60,64,67);letter-spacing:0.2px"> Pushing the Boundaries of Length Generalization and Reasoning Capabilities of Open LLMs</span><span style="letter-spacing:0.2px"><br style="color:rgb(60,64,67);letter-spacing:0.2px"><br style="color:rgb(60,64,67);letter-spacing:0.2px"></span><b style="color:rgb(60,64,67);letter-spacing:0.2px">Abstract: </b><span style="color:rgb(60,64,67);letter-spacing:0.2px">Recent advancements in open-source pretrained large language models (LLMs) have created new opportunities for exploring exciting post-pre training innovations. This talk shares some of our recent works. The first part of my talk focuses on context length generalization in LLMs. I will begin with a theoretical analysis that identifies major factors contributing to the failures of several commonly-used techniques, which leads to the development of a simple yet effective algorithm that enables pretrained LLMs to generalize to extreme context lengths without any parameter update. I will then shift focus to continual pretraining for length generalization, and share our recent findings highlighting the importance of training data mixture—a crucial yet previously overlooked factor. The second part of my talk will be about Eurus, our recently-released suite of open LLMs. On diverse benchmarks covering challenging math, coding, and reasoning problems, Eurus achieves state-of-the-art performance among all open-source models and outperforms GPT-3.5 Turbo. On two established reward modeling benchmarks, our 7B reward model achieves better correlation with human judgment than all existing models including GPT-4. I will especially highlight UltraInteract, our newly-curated alignment dataset that enables Eurus’s strong performance.</span><span style="letter-spacing:0.2px"><br style="color:rgb(60,64,67);letter-spacing:0.2px"><br style="color:rgb(60,64,67);letter-spacing:0.2px"></span><b style="color:rgb(60,64,67);letter-spacing:0.2px">Bio: </b><span style="color:rgb(60,64,67);letter-spacing:0.2px">Hao Peng is an Assistant Professor in the Department of Computer Science of the University of Illinois at Urbana-Champaign (UIUC). He received his PhD from the University of Washington and his bachelor’s degree from Peking University. Before joining UIUC, he spent one year at the Allen Institute for Artificial Intelligence as a Young Investigator, and time at Microsoft Research, Google, and DeepMind as an intern. His research interest broadly spans natural language processing and machine learning.</span></font><br></div><div dir="ltr" style="font-size:small"><br></div><b style="font-size:small"> Host: </b><a href="mailto:jzhou@ttic.edu" style="font-size:small" target="_blank">Jiawei Zhou</a></div></div></div></div><div style="font-size:small"><br></div><div style="font-size:small"><br></div><div style="font-size:small"><br></div></div><div><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><span style="font-family:arial,helvetica,sans-serif;font-size:x-small">Mary C. Marre</span><br></div><div><div><font face="arial, helvetica, sans-serif" size="1">Faculty Administrative Support</font></div><div><i><font face="arial, helvetica, sans-serif" color="#3d85c6" size="1"><b>Toyota Technological Institute</b></font></i></div><div><i><font face="arial, helvetica, sans-serif" color="#3d85c6" size="1">6045 S. Kenwood Avenue, Rm 517</font></i></div><div><font size="1"><i><font face="arial, helvetica, sans-serif" color="#3d85c6">Chicago, IL 60637</font></i><br></font></div><div><font size="1"><i><font face="arial, helvetica, sans-serif" color="#3d85c6">773-834-1757</font></i></font></div><div><b><i><a href="mailto:mmarre@ttic.edu" target="_blank"><font face="arial, helvetica, sans-serif" size="1">mmarre@ttic.edu</font></a></i></b></div></div></div></div></div></div>
</blockquote></div></div>