<div dir="ltr"><div dir="ltr"><div class="gmail_default" style="font-size:small"><div class="gmail_default"><div class="gmail_default"><div class="gmail_default"><p style="color:rgb(80,0,80);font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;margin:0px"><font face="arial, sans-serif" color="#000000"><font style="vertical-align:inherit"><font style="vertical-align:inherit"><b>When:</b> </font></font><font style="vertical-align:inherit"><font style="vertical-align:inherit"> Monday, March 7th at<b> <span style="background-color:rgb(255,255,0)">11:30 am CT</span></b></font></font><br></font></p><p style="color:rgb(80,0,80);font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;margin:0px"><br></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;color:rgb(80,0,80);line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="arial, sans-serif"><font style="color:rgb(0,0,0);vertical-align:inherit"><font style="vertical-align:inherit"><b>Where:</b> </font></font>Zoom Virtual Talk (<b><a href="https://www.google.com/url?q=https://uchicagogroup.zoom.us/webinar/register/WN_CzxpKP4XTwySfk-iwHSeiw&sa=D&source=calendar&ust=1646520803994961&usg=AOvVaw0S8Uw1D6ATrcuea9EUnXl1" target="_blank"><font color="#0000ff">register in advance here</font></a></b>)</font></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;color:rgb(80,0,80);line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="arial, sans-serif"><br></font></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;color:rgb(80,0,80);line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="arial, sans-serif"><font style="vertical-align:inherit"><font style="vertical-align:inherit"><font color="#000000"><b>Who: </b> </font><font color="#500050"> </font><font color="#000000"> </font></font></font></font><span style="color:rgb(34,34,34)">Guodong Zhang, University of Toronto</span></p></div><div><div dir="ltr"><div dir="ltr"><br></div><div dir="ltr"><br></div></div></div></div><div class="gmail_default"><div dir="auto"><span style="font-family:arial,sans-serif"><b>Title:</b> Scalable and Multiagent Deep Learning</span></div><div dir="auto"><span style="font-family:arial,sans-serif"><br></span></div><div dir="auto"><font face="arial, sans-serif"><b>Abstract: </b></font><span style="color:rgb(33,37,41);font-family:Arial;white-space:pre-wrap">Deep learning has achieved huge successes over the last few years, largely due to three important ideas: deep models with residual connections, parallelism, and gradient-based learning. However, it was shown that (1) deep ResNets behave like ensembles of shallow networks; (2) naively increasing the scale of data parallelism leads to diminishing return; (<span class="gmail-il">3</span>) gradient-based learning could converge to spurious fixed points in the multi-agent setting. </span><font face="arial, sans-serif"><br></font><span id="gmail-m_4202545553815987989gmail-m_3500277655617628900gmail-m_3229759637601255223m_2612011372659729876gmail-docs-internal-guid-12f16b14-7fff-b936-7a3b-b7843873a8ef"><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;color:rgb(33,37,41);font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">In this talk, I will present some of my works on understanding and addressing these issues. First, I will give a general recipe for training very deep neural networks without shortcuts. Second, I will present a noisy quadratic model for neural network optimization, which qualitatively predicts scaling properties of a variety of optimizers and in particular suggests that second-order algorithms would benefit more from data parallelism. Third, I will describe a novel algorithm that finds desired equilibria and saves us from converging to spurious fixed points in multi-agent games. In the end, I will conclude with future directions towards building intelligent machines that can learn from experience efficiently, reason about their own decisions, and act in our interests.</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><br></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;color:rgb(33,37,41);font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"><b style="color:rgb(34,34,34);font-family:Arial,Helvetica,sans-serif;white-space:normal">Bio:</b><span style="color:rgb(34,34,34);font-family:Arial,Helvetica,sans-serif;white-space:normal"> Guodong Zhang is a final-year PhD candidate at the University of Toronto, advised by Roger Grosse.</span><span style="color:rgb(34,34,34);white-space:normal;font-family:arial,sans-serif"> His research lies at the intersection between machine learning, optimization, and Bayesian statistics. In particular, his research focuses on understanding and developing algorithms for optimization, Bayesian inference, and multi-agent games in the context of deep learning. He has been recognized through the Apple PhD fellowship, Borealis AI fellowship, and many other scholarships. </span><span id="gmail-m_4202545553815987989gmail-m_3500277655617628900gmail-m_3229759637601255223m_2612011372659729876gmail-docs-internal-guid-c49b3518-7fff-1ec3-8777-a72e2c5fe8f1" style="color:rgb(34,34,34);font-family:Arial,Helvetica,sans-serif;white-space:normal"><span style="font-family:Arial;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">In the past, he has also interned at DeepMind, Google Brain, and Microsoft Research.</span></span>
</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;color:rgb(33,37,41);font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"><span style="color:rgb(34,34,34);font-family:Arial,Helvetica,sans-serif;white-space:normal"><span style="font-family:Arial;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"><br></span></span></span></p></span></div></div><div class="gmail_default"><b>Host:</b> <b><font color="#0000ff"><a href="mailto:mcallester@ttic.edu" target="_blank">David McAllester</a></font></b></div></div><div><div dir="ltr"><div dir="ltr"><div><br></div><div><br></div><div><br></div><div><br></div></div></div></div></div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><span style="font-family:arial,helvetica,sans-serif;font-size:x-small">Mary C. Marre</span><br></div><div><div><font face="arial, helvetica, sans-serif" size="1">Faculty Administrative Support</font></div><div><i><font face="arial, helvetica, sans-serif" color="#3d85c6" size="1"><b>Toyota Technological Institute</b></font></i></div><div><i><font face="arial, helvetica, sans-serif" color="#3d85c6" size="1">6045 S. Kenwood Avenue</font></i></div><div><font size="1"><i><font face="arial, helvetica, sans-serif" color="#3d85c6">Chicago, IL 60637</font></i><br></font></div><div><b><i><a href="mailto:mmarre@ttic.edu" target="_blank"><font face="arial, helvetica, sans-serif" size="1">mmarre@ttic.edu</font></a></i></b></div></div></div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Mar 6, 2022 at 2:00 PM Mary Marre <<a href="mailto:mmarre@ttic.edu">mmarre@ttic.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div style="font-size:small"><div><div><div><p style="color:rgb(80,0,80);font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;margin:0px"><font face="arial, sans-serif" color="#000000"><font style="vertical-align:inherit"><font style="vertical-align:inherit"><b>When:</b> </font></font><font style="vertical-align:inherit"><font style="vertical-align:inherit"> Monday, March 7th at<b> <span style="background-color:rgb(255,255,0)">11:30 am CT</span></b></font></font><br></font></p><p style="color:rgb(80,0,80);font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;margin:0px"><br></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;color:rgb(80,0,80);line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="arial, sans-serif"><font style="color:rgb(0,0,0);vertical-align:inherit"><font style="vertical-align:inherit"><b>Where:</b> </font></font>Zoom Virtual Talk (<b><a href="https://www.google.com/url?q=https://uchicagogroup.zoom.us/webinar/register/WN_CzxpKP4XTwySfk-iwHSeiw&sa=D&source=calendar&ust=1646520803994961&usg=AOvVaw0S8Uw1D6ATrcuea9EUnXl1" target="_blank"><font color="#0000ff">register in advance here</font></a></b>)</font></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;color:rgb(80,0,80);line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="arial, sans-serif"><br></font></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;color:rgb(80,0,80);line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="arial, sans-serif"><font style="vertical-align:inherit"><font style="vertical-align:inherit"><font color="#000000"><b>Who: </b> </font><font color="#500050"> </font><font color="#000000"> </font></font></font></font><span style="color:rgb(34,34,34)">Guodong Zhang, University of Toronto</span></p></div><div><div dir="ltr"><div dir="ltr"><br></div><div dir="ltr"><br></div></div></div></div><div><div dir="auto"><span style="font-family:arial,sans-serif"><b>Title:</b> Scalable and Multiagent Deep Learning</span></div><div dir="auto"><span style="font-family:arial,sans-serif"><br></span></div><div dir="auto"><font face="arial, sans-serif"><b>Abstract: </b></font><span style="color:rgb(33,37,41);font-family:Arial;white-space:pre-wrap">Deep learning has achieved huge successes over the last few years, largely due to three important ideas: deep models with residual connections, parallelism, and gradient-based learning. However, it was shown that (1) deep ResNets behave like ensembles of shallow networks; (2) naively increasing the scale of data parallelism leads to diminishing return; (<span>3</span>) gradient-based learning could converge to spurious fixed points in the multi-agent setting. </span><font face="arial, sans-serif"><br></font><span id="gmail-m_4202545553815987989gmail-m_3500277655617628900gmail-m_3229759637601255223m_2612011372659729876gmail-docs-internal-guid-12f16b14-7fff-b936-7a3b-b7843873a8ef"><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;color:rgb(33,37,41);font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">In this talk, I will present some of my works on understanding and addressing these issues. First, I will give a general recipe for training very deep neural networks without shortcuts. Second, I will present a noisy quadratic model for neural network optimization, which qualitatively predicts scaling properties of a variety of optimizers and in particular suggests that second-order algorithms would benefit more from data parallelism. Third, I will describe a novel algorithm that finds desired equilibria and saves us from converging to spurious fixed points in multi-agent games. In the end, I will conclude with future directions towards building intelligent machines that can learn from experience efficiently, reason about their own decisions, and act in our interests.</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><br></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;color:rgb(33,37,41);font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"><b style="color:rgb(34,34,34);font-family:Arial,Helvetica,sans-serif;white-space:normal">Bio:</b><span style="color:rgb(34,34,34);font-family:Arial,Helvetica,sans-serif;white-space:normal"> Guodong Zhang is a final-year PhD candidate at the University of Toronto, advised by Roger Grosse.</span><span style="color:rgb(34,34,34);white-space:normal;font-family:arial,sans-serif"> His research lies at the intersection between machine learning, optimization, and Bayesian statistics. In particular, his research focuses on understanding and developing algorithms for optimization, Bayesian inference, and multi-agent games in the context of deep learning. He has been recognized through the Apple PhD fellowship, Borealis AI fellowship, and many other scholarships. </span><span id="gmail-m_4202545553815987989gmail-m_3500277655617628900gmail-m_3229759637601255223m_2612011372659729876gmail-docs-internal-guid-c49b3518-7fff-1ec3-8777-a72e2c5fe8f1" style="color:rgb(34,34,34);font-family:Arial,Helvetica,sans-serif;white-space:normal"><span style="font-family:Arial;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">In the past, he has also interned at DeepMind, Google Brain, and Microsoft Research.</span></span>
</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;color:rgb(33,37,41);font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"><span style="color:rgb(34,34,34);font-family:Arial,Helvetica,sans-serif;white-space:normal"><span style="font-family:Arial;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"><br></span></span></span></p></span></div></div><div><b>Host:</b> <b><font color="#0000ff"><a href="mailto:mcallester@ttic.edu" target="_blank">David McAllester</a></font></b></div></div><div><div dir="ltr"><div dir="ltr"><div><br></div><div><br></div><div><br></div><div><br></div></div></div></div></div><div><div dir="ltr"><div dir="ltr"><div><span style="font-family:arial,helvetica,sans-serif;font-size:x-small">Mary C. Marre</span><br></div><div><div><font face="arial, helvetica, sans-serif" size="1">Faculty Administrative Support</font></div><div><i><font face="arial, helvetica, sans-serif" color="#3d85c6" size="1"><b>Toyota Technological Institute</b></font></i></div><div><i><font face="arial, helvetica, sans-serif" color="#3d85c6" size="1">6045 S. Kenwood Avenue</font></i></div><div><font size="1"><i><font face="arial, helvetica, sans-serif" color="#3d85c6">Chicago, IL 60637</font></i><br></font></div><div><b><i><a href="mailto:mmarre@ttic.edu" target="_blank"><font face="arial, helvetica, sans-serif" size="1">mmarre@ttic.edu</font></a></i></b></div></div></div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Feb 28, 2022 at 5:01 PM Mary Marre <<a href="mailto:mmarre@ttic.edu" target="_blank">mmarre@ttic.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="font-size:small"><div><div><p style="color:rgb(80,0,80);font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;margin:0px"><font face="arial, sans-serif" color="#000000"><font style="vertical-align:inherit"><font style="vertical-align:inherit"><b>When:</b> </font></font><font style="vertical-align:inherit"><font style="vertical-align:inherit"> Monday, March 7th at<b> <span style="background-color:rgb(255,255,0)">11:30 am CT</span></b></font></font><br></font></p><p style="color:rgb(80,0,80);font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;margin:0px"><br></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;color:rgb(80,0,80);line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="arial, sans-serif"><font style="color:rgb(0,0,0);vertical-align:inherit"><font style="vertical-align:inherit"><b>Where:</b> </font></font>Zoom Virtual Talk (<b><a href="https://www.google.com/url?q=https://uchicagogroup.zoom.us/webinar/register/WN_CzxpKP4XTwySfk-iwHSeiw&sa=D&source=calendar&ust=1646520803994961&usg=AOvVaw0S8Uw1D6ATrcuea9EUnXl1" target="_blank"><font color="#0000ff">register in advance here</font></a></b>)</font></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;color:rgb(80,0,80);line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="arial, sans-serif"><br></font></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;color:rgb(80,0,80);line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="arial, sans-serif"><font style="vertical-align:inherit"><font style="vertical-align:inherit"><font color="#000000"><b>Who: </b> </font><font color="#500050"> </font><font color="#000000"> </font></font></font></font><span style="color:rgb(34,34,34)">Guodong Zhang, University of Toronto</span></p></div><div><div dir="ltr"><div dir="ltr"><br></div><div dir="ltr"><br></div></div></div></div><div><div dir="auto"><span style="font-family:arial,sans-serif"><b>Title:</b> Scalable and Multiagent Deep Learning</span></div><div dir="auto"><span style="font-family:arial,sans-serif"><br></span></div><div dir="auto"><font face="arial, sans-serif"><b>Abstract: </b></font><span style="color:rgb(33,37,41);font-family:Arial;white-space:pre-wrap">Deep learning has achieved huge successes over the last few years, largely due to three important ideas: deep models with residual connections, parallelism, and gradient-based learning. However, it was shown that (1) deep ResNets behave like ensembles of shallow networks; (2) naively increasing the scale of data parallelism leads to diminishing return; (3) gradient-based learning could converge to spurious fixed points in the multi-agent setting. </span><font face="arial, sans-serif"><br></font><span id="gmail-m_4202545553815987989gmail-m_3500277655617628900gmail-m_3229759637601255223m_2612011372659729876gmail-docs-internal-guid-12f16b14-7fff-b936-7a3b-b7843873a8ef"><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;color:rgb(33,37,41);font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">In this talk, I will present some of my works on understanding and addressing these issues. First, I will give a general recipe for training very deep neural networks without shortcuts. Second, I will present a noisy quadratic model for neural network optimization, which qualitatively predicts scaling properties of a variety of optimizers and in particular suggests that second-order algorithms would benefit more from data parallelism. Third, I will describe a novel algorithm that finds desired equilibria and saves us from converging to spurious fixed points in multi-agent games. In the end, I will conclude with future directions towards building intelligent machines that can learn from experience efficiently, reason about their own decisions, and act in our interests.</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><br></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;color:rgb(33,37,41);font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"><b style="color:rgb(34,34,34);font-family:Arial,Helvetica,sans-serif;white-space:normal">Bio:</b><span style="color:rgb(34,34,34);font-family:Arial,Helvetica,sans-serif;white-space:normal"> Guodong Zhang is a final-year PhD candidate at the University of Toronto, advised by Roger Grosse.</span><span style="color:rgb(34,34,34);white-space:normal;font-family:arial,sans-serif"> His research lies at the intersection between machine learning, optimization, and Bayesian statistics. In particular, his research focuses on understanding and developing algorithms for optimization, Bayesian inference, and multi-agent games in the context of deep learning. He has been recognized through the Apple PhD fellowship, Borealis AI fellowship, and many other scholarships. </span><span id="gmail-m_4202545553815987989gmail-m_3500277655617628900gmail-m_3229759637601255223m_2612011372659729876gmail-docs-internal-guid-c49b3518-7fff-1ec3-8777-a72e2c5fe8f1" style="color:rgb(34,34,34);font-family:Arial,Helvetica,sans-serif;white-space:normal"><span style="font-family:Arial;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">In the past, he has also interned at DeepMind, Google Brain, and Microsoft Research.</span></span>
</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;color:rgb(33,37,41);font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"><span style="color:rgb(34,34,34);font-family:Arial,Helvetica,sans-serif;white-space:normal"><span style="font-family:Arial;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"><br></span></span></span></p></span></div></div><div><b>Host:</b> <b><font color="#0000ff"><a href="mailto:mcallester@ttic.edu" target="_blank">David McAllester</a></font></b></div></div><div><div dir="ltr"><div dir="ltr"><div><br></div><div><span style="font-family:arial,helvetica,sans-serif;font-size:x-small"><br></span></div><div><span style="font-family:arial,helvetica,sans-serif;font-size:x-small">Mary C. Marre</span><br></div><div><div><font face="arial, helvetica, sans-serif" size="1">Faculty Administrative Support</font></div><div><i><font face="arial, helvetica, sans-serif" color="#3d85c6" size="1"><b>Toyota Technological Institute</b></font></i></div><div><i><font face="arial, helvetica, sans-serif" color="#3d85c6" size="1">6045 S. Kenwood Avenue</font></i></div><div><font size="1"><i><font face="arial, helvetica, sans-serif" color="#3d85c6">Chicago, IL 60637</font></i><br></font></div><div><b><i><a href="mailto:mmarre@ttic.edu" target="_blank"><font face="arial, helvetica, sans-serif" size="1">mmarre@ttic.edu</font></a></i></b></div></div></div></div></div></div>
</blockquote></div></div>
</blockquote></div></div>