<div dir="ltr"><div dir="ltr"><div class="gmail_default" style="font-size:small"><div class="gmail_default"><p style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;color:rgb(80,0,80);margin:0px"><font face="arial, sans-serif" color="#000000"><font style="vertical-align:inherit"><font style="vertical-align:inherit"><b>When:</b> </font></font><font style="vertical-align:inherit"><font style="vertical-align:inherit"> Tuesday, February 22nd at<b> <span style="background-color:rgb(255,255,0)">11:00 am CT</span></b></font></font><br></font></p><p style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;color:rgb(80,0,80);margin:0px"><font face="arial, sans-serif" color="#000000"><font style="vertical-align:inherit"><font style="vertical-align:inherit"><b><span style="background-color:rgb(255,255,0)"><br></span></b></font></font></font></p><div class="gmail_default"><font face="arial, sans-serif"><b>Where: </b><font color="#500050">Talk will be given </font><font color="#0000ff" style="font-weight:bold"><u>live, in-person</u></font><font color="#0000ff" style="font-weight:bold"> </font><font color="#000000">at</font></font></div><p class="MsoNormal" style="margin:0in;color:rgb(80,0,80);line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="arial, sans-serif"> TTIC, 6045 S. Kenwood Avenue</font></p><p class="MsoNormal" style="margin:0in;color:rgb(80,0,80);line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="arial, sans-serif"> 5th Floor, Room 530<b><span style="color:black"> </span></b></font></p><p class="MsoNormal" style="margin:0in;color:rgb(80,0,80);line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="arial, sans-serif"><b><span style="color:black"><br></span></b></font></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="arial, sans-serif"><font style="color:rgb(0,0,0);vertical-align:inherit"><font style="vertical-align:inherit"><b>Where:</b> </font></font><font style="color:rgb(80,0,80)">Zoom Virtual Talk (</font><b><a href="https://uchicagogroup.zoom.us/webinar/register/WN_kfIOOoD0RLCENJB7OSyIeg" target="_blank"><font color="#0000ff">register in advance here</font></a></b><font style="color:rgb(80,0,80)">)</font></font></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;color:rgb(80,0,80);line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="arial, sans-serif"><br></font></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="arial, sans-serif"><font style="vertical-align:inherit"><font style="vertical-align:inherit"><font color="#000000"><b>Who: </b> </font><font color="#500050"> </font><font color="#000000"> </font></font></font></font>Zhiyuan Li, Princeton University</p><br></div><div><div dir="ltr"><div dir="ltr"><br><b>Title:</b> <span class="gmail_default"> </span>Toward Mathematical Understanding of Real-life Deep Learning<br><br><b>Abstract: </b>There is great interest in developing a mathematical understanding of the tremendous success of deep learning. Most of this understanding has been done in simplified settings (depth 2 or 3; NTK regime). This talk presents my recent works providing a mathematical understanding of real-life nets and losses, incorporating the effect of normalization, architectural features, stochasticity, and finite learning rate(LR). It leverages insights from continuous mathematics (including Stochastic Differential Equation(SDE)) which I will use to show interesting new mechanisms for implicit regularization during training. I will finish by presenting a new practical advance from our theoretical insights: a robust variant of BERT (a language model at the heart of the ongoing revolution in Natural Language Processing) called SIBERT that uses a new scale-invariance architecture and is trainable with vanilla SGD.<br><br><b>Bio: </b>Zhiyuan Li is a PhD candidate in the Department of Computer Science at Princeton University, advised by Sanjeev Arora. Previously, he obtained his bachelor’s degree in Computer Science from Tsinghua University. He has also spent time as a research intern at Google Research. His current research goal is to develop a mathematical theory towards a better understanding of modern deep learning, as well as to design more efficient and principled machine learning methods using theoretical insights. He is a recipient of Microsoft Research PhD Fellowship in 2020.<br><br><b>Host: </b><a href="mailto:mcallester@ttic.edu" target="_blank"><b>David McAllester</b></a></div><div dir="ltr"><br></div><div dir="ltr"><br></div><div dir="ltr"><br></div></div></div></div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><span style="font-family:arial,helvetica,sans-serif;font-size:x-small">Mary C. Marre</span><br></div><div><div><font face="arial, helvetica, sans-serif" size="1">Faculty Administrative Support</font></div><div><i><font face="arial, helvetica, sans-serif" color="#3d85c6" size="1"><b>Toyota Technological Institute</b></font></i></div><div><i><font face="arial, helvetica, sans-serif" color="#3d85c6" size="1">6045 S. Kenwood Avenue</font></i></div><div><font size="1"><i><font face="arial, helvetica, sans-serif" color="#3d85c6">Chicago, IL 60637</font></i><br></font></div><div><b><i><a href="mailto:mmarre@ttic.edu" target="_blank"><font face="arial, helvetica, sans-serif" size="1">mmarre@ttic.edu</font></a></i></b></div></div></div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Feb 21, 2022 at 3:41 PM Mary Marre <<a href="mailto:mmarre@ttic.edu">mmarre@ttic.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div style="font-size:small"><div><p style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;color:rgb(80,0,80);margin:0px"><font face="arial, sans-serif" color="#000000"><font style="vertical-align:inherit"><font style="vertical-align:inherit"><b>When:</b> </font></font><font style="vertical-align:inherit"><font style="vertical-align:inherit"> Tuesday, February 22nd at<b> <span style="background-color:rgb(255,255,0)">11:00 am CT</span></b></font></font><br></font></p><p style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;color:rgb(80,0,80);margin:0px"><font face="arial, sans-serif" color="#000000"><font style="vertical-align:inherit"><font style="vertical-align:inherit"><b><span style="background-color:rgb(255,255,0)"><br></span></b></font></font></font></p><div><font face="arial, sans-serif"><b>Where: </b><font color="#500050">Talk will be given </font><font color="#0000ff" style="font-weight:bold"><u>live, in-person</u></font><font color="#0000ff" style="font-weight:bold"> </font><font color="#000000">at</font></font></div><p class="MsoNormal" style="margin:0in;color:rgb(80,0,80);line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="arial, sans-serif"> TTIC, 6045 S. Kenwood Avenue</font></p><p class="MsoNormal" style="margin:0in;color:rgb(80,0,80);line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="arial, sans-serif"> 5th Floor, Room 530<b><span style="color:black"> </span></b></font></p><p class="MsoNormal" style="margin:0in;color:rgb(80,0,80);line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="arial, sans-serif"><b><span style="color:black"><br></span></b></font></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="arial, sans-serif"><font style="color:rgb(0,0,0);vertical-align:inherit"><font style="vertical-align:inherit"><b>Where:</b> </font></font><font style="color:rgb(80,0,80)">Zoom Virtual Talk (</font><b><a href="https://uchicagogroup.zoom.us/webinar/register/WN_kfIOOoD0RLCENJB7OSyIeg" target="_blank"><font color="#0000ff">register in advance here</font></a></b><font style="color:rgb(80,0,80)">)</font></font></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;color:rgb(80,0,80);line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="arial, sans-serif"><br></font></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="arial, sans-serif"><font style="vertical-align:inherit"><font style="vertical-align:inherit"><font color="#000000"><b>Who: </b> </font><font color="#500050"> </font><font color="#000000"> </font></font></font></font><span>Zhiyuan</span> Li, Princeton University</p><br></div><div><div dir="ltr"><div dir="ltr"><br><b>Title:</b> <span class="gmail_default"> </span>Toward Mathematical Understanding of Real-life Deep Learning<br><br><b>Abstract: </b>There is great interest in developing a mathematical understanding of the tremendous success of deep learning. Most of this understanding has been done in simplified settings (depth 2 or 3; NTK regime). This talk presents my recent works providing a mathematical understanding of real-life nets and losses, incorporating the effect of normalization, architectural features, stochasticity, and finite learning rate(LR). It leverages insights from continuous mathematics (including Stochastic Differential Equation(SDE)) which I will use to show interesting new mechanisms for implicit regularization during training. I will finish by presenting a new practical advance from our theoretical insights: a robust variant of BERT (a language model at the heart of the ongoing revolution in Natural Language Processing) called SIBERT that uses a new scale-invariance architecture and is trainable with vanilla SGD.<br><br><b>Bio: </b><span>Zhiyuan</span> Li is a PhD candidate in the Department of Computer Science at Princeton University, advised by Sanjeev Arora. Previously, he obtained his bachelor’s degree in Computer Science from Tsinghua University. He has also spent time as a research intern at Google Research. His current research goal is to develop a mathematical theory towards a better understanding of modern deep learning, as well as to design more efficient and principled machine learning methods using theoretical insights. He is a recipient of Microsoft Research PhD Fellowship in 2020.<br><br><b>Host: </b><a href="mailto:mcallester@ttic.edu" target="_blank"><b>David McAllester</b></a><div><span style="font-family:arial,helvetica,sans-serif;font-size:x-small"><br></span></div><br></div><div dir="ltr"><br></div></div></div></div><div><div dir="ltr"><div dir="ltr"><div><span style="font-family:arial,helvetica,sans-serif;font-size:x-small">Mary C. Marre</span><br></div><div><div><font face="arial, helvetica, sans-serif" size="1">Faculty Administrative Support</font></div><div><i><font face="arial, helvetica, sans-serif" color="#3d85c6" size="1"><b>Toyota Technological Institute</b></font></i></div><div><i><font face="arial, helvetica, sans-serif" color="#3d85c6" size="1">6045 S. Kenwood Avenue</font></i></div><div><font size="1"><i><font face="arial, helvetica, sans-serif" color="#3d85c6">Chicago, IL 60637</font></i><br></font></div><div><b><i><a href="mailto:mmarre@ttic.edu" target="_blank"><font face="arial, helvetica, sans-serif" size="1">mmarre@ttic.edu</font></a></i></b></div></div></div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Feb 16, 2022 at 3:10 PM Mary Marre <<a href="mailto:mmarre@ttic.edu" target="_blank">mmarre@ttic.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="font-size:small"><p style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;color:rgb(80,0,80);margin:0px"><font face="arial, sans-serif" color="#000000"><font style="vertical-align:inherit"><font style="vertical-align:inherit"><b>When:</b> </font></font><font style="vertical-align:inherit"><font style="vertical-align:inherit"> Tuesday, February 22nd at<b> <span style="background-color:rgb(255,255,0)">11:00 am CT</span></b></font></font><br></font></p><p style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;color:rgb(80,0,80);margin:0px"><font face="arial, sans-serif" color="#000000"><font style="vertical-align:inherit"><font style="vertical-align:inherit"><b><span style="background-color:rgb(255,255,0)"><br></span></b></font></font></font></p><div><font face="arial, sans-serif"><b>Where: </b><font color="#500050">Talk will be given </font><font color="#0000ff" style="font-weight:bold"><u>live, in-person</u></font><font color="#0000ff" style="font-weight:bold"> </font><font color="#000000">at</font></font></div><p class="MsoNormal" style="margin:0in;color:rgb(80,0,80);line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="arial, sans-serif"> TTIC, 6045 S. Kenwood Avenue</font></p><p class="MsoNormal" style="margin:0in;color:rgb(80,0,80);line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="arial, sans-serif"> 5th Floor, Room 530<b><span style="color:black"> </span></b></font></p><p class="MsoNormal" style="margin:0in;color:rgb(80,0,80);line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="arial, sans-serif"><b><span style="color:black"><br></span></b></font></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="arial, sans-serif"><font style="color:rgb(0,0,0);vertical-align:inherit"><font style="vertical-align:inherit"><b>Where:</b> </font></font><font style="color:rgb(80,0,80)">Zoom Virtual Talk (</font><b><a href="https://uchicagogroup.zoom.us/webinar/register/WN_kfIOOoD0RLCENJB7OSyIeg" target="_blank"><font color="#0000ff">register in advance here</font></a></b><font style="color:rgb(80,0,80)">)</font></font></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;color:rgb(80,0,80);line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="arial, sans-serif"><br></font></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="arial, sans-serif"><font style="vertical-align:inherit"><font style="vertical-align:inherit"><font color="#000000"><b>Who: </b> </font><font color="#500050"> </font><font color="#000000"> </font></font></font></font>Zhiyuan Li, Princeton University</p><br></div><div><div dir="ltr"><div dir="ltr"><br><b>Title:</b> <span class="gmail_default" style="font-size:small"> </span>Toward Mathematical Understanding of Real-life Deep Learning<br><br><b>Abstract: </b>There is great interest in developing a mathematical understanding of the tremendous success of deep learning. Most of this understanding has been done in simplified settings (depth 2 or 3; NTK regime). This talk presents my recent works providing a mathematical understanding of real-life nets and losses, incorporating the effect of normalization, architectural features, stochasticity, and finite learning rate(LR). It leverages insights from continuous mathematics (including Stochastic Differential Equation(SDE)) which I will use to show interesting new mechanisms for implicit regularization during training. I will finish by presenting a new practical advance from our theoretical insights: a robust variant of BERT (a language model at the heart of the ongoing revolution in Natural Language Processing) called SIBERT that uses a new scale-invariance architecture and is trainable with vanilla SGD.<br><br><b>Bio: </b>Zhiyuan Li is a PhD candidate in the Department of Computer Science at Princeton University, advised by Sanjeev Arora. Previously, he obtained his bachelor’s degree in Computer Science from Tsinghua University. He has also spent time as a research intern at Google Research. His current research goal is to develop a mathematical theory towards a better understanding of modern deep learning, as well as to design more efficient and principled machine learning methods using theoretical insights. He is a recipient of Microsoft Research PhD Fellowship in 2020.<br><br><b>Host: </b><a href="mailto:mcallester@ttic.edu" target="_blank"><b>David McAllester</b></a><div><span style="font-family:arial,helvetica,sans-serif;font-size:x-small"><br></span></div><div><span style="font-family:arial,helvetica,sans-serif;font-size:x-small"><br></span></div><div><span style="font-family:arial,helvetica,sans-serif;font-size:x-small"><br></span></div><div><span style="font-family:arial,helvetica,sans-serif;font-size:x-small"><br></span></div><div><span style="font-family:arial,helvetica,sans-serif;font-size:x-small"><br></span></div><div><span style="font-family:arial,helvetica,sans-serif;font-size:x-small">Mary C. Marre</span><br></div><div><div><font face="arial, helvetica, sans-serif" size="1">Faculty Administrative Support</font></div><div><i><font face="arial, helvetica, sans-serif" color="#3d85c6" size="1"><b>Toyota Technological Institute</b></font></i></div><div><i><font face="arial, helvetica, sans-serif" color="#3d85c6" size="1">6045 S. Kenwood Avenue</font></i></div><div><font size="1"><i><font face="arial, helvetica, sans-serif" color="#3d85c6">Chicago, IL 60637</font></i><br></font></div><div><b><i><a href="mailto:mmarre@ttic.edu" target="_blank"><font face="arial, helvetica, sans-serif" size="1">mmarre@ttic.edu</font></a></i></b></div></div></div></div></div></div>
</blockquote></div></div>
</blockquote></div></div>