<div dir="ltr"><div dir="ltr"><div class="gmail_default" style="font-size:small"><div class="gmail_default"><p class="gmail-m_2910299410442543454gmail-m_6186297207631658852gmail-m_-2549525831759316499gmail-m_4645642444737212984m_313362707539895620m_5707201597699274603m_-5457714984863114200gmail-m_-4054378802175222545gmail-m_3947158569005651454gmail-m_-6060531980912436720gmail-m_-5698796270993385318m_6700458045820563854gmail-m_427904505759088481gmail-m_496884293731504483gmail-m_4528131826496012784gmail-m_5482748634321315606gmail-m_417349372124497387gmail-m_4614845926281015477gmail-m_6041720643954058195gmail-m_-7670275615116031415gmail-m_-5817829815640557222gmail-m_5987474974647831651gmail-m_-4783362384882292594m_6961031835771836416gmail-m_3149180880964055314gmail-m_5803000941478265060gmail-m_3739772758111120207gmail-m_-4374496420704574181gmail-m_-8232014986225864746gmail-m_2118555233517397122gmail-m_-6347337869693432729gmail-m_9103776001042077600gmail-p1" style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;margin:0px"><font face="arial, sans-serif"><font style="vertical-align:inherit"><font style="vertical-align:inherit"><b>When:</b> </font></font><font style="vertical-align:inherit"><font style="vertical-align:inherit"> Monday, September 23th at 11:00 am</font></font><br></font></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="arial, sans-serif"> </font></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="arial, sans-serif"><font style="vertical-align:inherit"><font style="vertical-align:inherit"><b>Where:</b> </font></font><font style="vertical-align:inherit"><font style="vertical-align:inherit">TTIC, 6045 S. Kenwood Avenue, 5th Floor, Room 526</font></font></font></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="arial, sans-serif"> </font></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="arial, sans-serif"><font style="vertical-align:inherit"><font style="vertical-align:inherit"><b>Who: </b> </font></font></font><span class="gmail-m_2910299410442543454gmail-il">Michael W.</span> <span class="gmail-m_2910299410442543454gmail-il">Mahoney</span><font face="arial, sans-serif">, </font>University of California at Berkeley</p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><br></p></div><div class="gmail_default"><p class="gmail-m_2910299410442543454gmail-m_6186297207631658852gmail-m_-2549525831759316499gmail-m_4645642444737212984m_313362707539895620m_5707201597699274603m_-5457714984863114200gmail-m_-4054378802175222545gmail-m_3947158569005651454gmail-m_-1597221607886257450m_-5823613242803553233MsoPlainText"><font face="arial, sans-serif"><b>Title: </b> </font>Why Deep Learning Works: Traditional and Heavy-Tailed Implicit Self-Regularization in Deep Neural Networks<font face="arial, sans-serif"> </font></p><p class="gmail-m_2910299410442543454gmail-m_6186297207631658852gmail-m_-2549525831759316499gmail-m_4645642444737212984m_313362707539895620m_5707201597699274603m_-5457714984863114200gmail-m_-4054378802175222545gmail-m_3947158569005651454gmail-m_-1597221607886257450m_-5823613242803553233MsoPlainText"><font face="arial, sans-serif"><b>Abstract:</b> </font>Random Matrix Theory (RMT) is applied to analyze the weight matrices of Deep Neural Networks (DNNs), including both production quality, pre-trained models and smaller models trained from scratch. Empirical and theoretical results clearly indicate that the DNN training process itself implicitly implements a form of self-regularization, implicitly sculpting a more regularized energy or penalty landscape. In particular, the empirical spectral density (ESD) of DNN layer matrices displays signatures of traditionally-regularized statistical models, even in the absence of exogenously specifying traditional forms of explicit regularization. Building on relatively recent results in RMT, most notably its extension to Universality classes of Heavy-Tailed matrices, and applying them to these empirical results, we develop a theory to identify 5+1 Phases of Training, corresponding to increasing amounts of implicit self-regularization. For smaller and/or older DNNs, this implicit self-regularization is like traditional Tikhonov regularization, in that there appears to be a ``size scale'' separating signal from noise. For state-of-the-art DNNs, however, we identify a novel form of heavy-tailed self-regularization, similar to the self-organization seen in the statistical physics of disordered systems. This implicit self-regularization can depend strongly on the many knobs of the training process. In particular, by exploiting the generalization gap phenomena, we demonstrate that we can cause a small model to exhibit all 5+1 phases of training simply by changing the batch size. This demonstrates that---all else being equal---DNN optimization with larger batch sizes leads to less-well implicitly-regularized models, and it provides an explanation for the generalization gap phenomena. Coupled with work on energy landscapes and heavy-tailed spin glasses, it also suggests an explanation of why deep learning works. Joint work with Charles Martin of Calculation Consulting, Inc.</p><p class="gmail-m_2910299410442543454gmail-m_6186297207631658852gmail-m_-2549525831759316499gmail-m_4645642444737212984m_313362707539895620m_5707201597699274603m_-5457714984863114200gmail-m_-4054378802175222545gmail-m_3947158569005651454gmail-m_-1597221607886257450m_-5823613242803553233MsoPlainText"><br></p><p class="gmail-m_2910299410442543454gmail-m_6186297207631658852gmail-m_-2549525831759316499gmail-m_4645642444737212984m_313362707539895620m_5707201597699274603m_-5457714984863114200gmail-m_-4054378802175222545gmail-m_3947158569005651454gmail-m_-1597221607886257450m_-5823613242803553233MsoPlainText"><font face="arial, sans-serif">Host: <a href="mailto:avrim@ttic.edu" target="_blank">Avrim Blum</a></font></p><p class="gmail-m_2910299410442543454gmail-m_6186297207631658852gmail-m_-2549525831759316499gmail-m_4645642444737212984m_313362707539895620m_5707201597699274603m_-5457714984863114200gmail-m_-4054378802175222545gmail-m_3947158569005651454gmail-m_-1597221607886257450m_-5823613242803553233MsoPlainText"><br></p></div></div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><font face="arial, helvetica, sans-serif">Mary C. Marre</font><div><font face="arial, helvetica, sans-serif">Administrative Assistant</font></div><div><i><font face="arial, helvetica, sans-serif" color="#3d85c6"><b>Toyota Technological Institute</b></font></i></div><div><i><font face="arial, helvetica, sans-serif" color="#3d85c6">6045 S. Kenwood Avenue</font></i></div><div><i><font face="arial, helvetica, sans-serif" color="#3d85c6">Room 517</font></i></div><div><i><font face="arial, helvetica, sans-serif" color="#3d85c6">Chicago, IL 60637</font></i></div><div><i><font face="arial, helvetica, sans-serif">p:(773) 834-1757</font></i></div><div><i><font face="arial, helvetica, sans-serif">f: (773) 357-6970</font></i></div><div><b><i><a href="mailto:mmarre@ttic.edu" target="_blank"><font face="arial, helvetica, sans-serif">mmarre@ttic.edu</font></a></i></b></div></div></div></div></div></div></div></div></div></div></div></div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Sep 16, 2019 at 6:03 PM Mary Marre <<a href="mailto:mmarre@ttic.edu">mmarre@ttic.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="font-size:small"><div><p class="gmail-m_2910299410442543454gmail-m_6186297207631658852gmail-m_-2549525831759316499gmail-m_4645642444737212984m_313362707539895620m_5707201597699274603m_-5457714984863114200gmail-m_-4054378802175222545gmail-m_3947158569005651454gmail-m_-6060531980912436720gmail-m_-5698796270993385318m_6700458045820563854gmail-m_427904505759088481gmail-m_496884293731504483gmail-m_4528131826496012784gmail-m_5482748634321315606gmail-m_417349372124497387gmail-m_4614845926281015477gmail-m_6041720643954058195gmail-m_-7670275615116031415gmail-m_-5817829815640557222gmail-m_5987474974647831651gmail-m_-4783362384882292594m_6961031835771836416gmail-m_3149180880964055314gmail-m_5803000941478265060gmail-m_3739772758111120207gmail-m_-4374496420704574181gmail-m_-8232014986225864746gmail-m_2118555233517397122gmail-m_-6347337869693432729gmail-m_9103776001042077600gmail-p1" style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;margin:0px"><font face="arial, sans-serif"><font style="vertical-align:inherit"><font style="vertical-align:inherit"><b>When:</b> </font></font><font style="vertical-align:inherit"><font style="vertical-align:inherit"> Monday, September 23th at 11:00 am</font></font><br></font></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="arial, sans-serif"> </font></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="arial, sans-serif"><font style="vertical-align:inherit"><font style="vertical-align:inherit"><b>Where:</b> </font></font><font style="vertical-align:inherit"><font style="vertical-align:inherit">TTIC, 6045 S. Kenwood Avenue, 5th Floor, Room 526</font></font></font></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="arial, sans-serif"> </font></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="arial, sans-serif"><font style="vertical-align:inherit"><font style="vertical-align:inherit"><b>Who: </b> </font></font></font><span class="gmail-m_2910299410442543454gmail-il">Michael W.</span> <span class="gmail-m_2910299410442543454gmail-il">Mahoney</span><font face="arial, sans-serif">, </font>University of California at Berkeley</p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><br></p></div><div><p class="gmail-m_2910299410442543454gmail-m_6186297207631658852gmail-m_-2549525831759316499gmail-m_4645642444737212984m_313362707539895620m_5707201597699274603m_-5457714984863114200gmail-m_-4054378802175222545gmail-m_3947158569005651454gmail-m_-1597221607886257450m_-5823613242803553233MsoPlainText"><font face="arial, sans-serif"><b>Title: </b> </font>Why Deep Learning Works: Traditional and Heavy-Tailed Implicit Self-Regularization in Deep Neural Networks<font face="arial, sans-serif"> </font></p><p class="gmail-m_2910299410442543454gmail-m_6186297207631658852gmail-m_-2549525831759316499gmail-m_4645642444737212984m_313362707539895620m_5707201597699274603m_-5457714984863114200gmail-m_-4054378802175222545gmail-m_3947158569005651454gmail-m_-1597221607886257450m_-5823613242803553233MsoPlainText"><font face="arial, sans-serif"><b>Abstract:</b> </font>Random Matrix Theory (RMT) is applied to analyze the weight matrices of Deep Neural Networks (DNNs), including both production quality, pre-trained models and smaller models trained from scratch. Empirical and theoretical results clearly indicate that the DNN training process itself implicitly implements a form of self-regularization, implicitly sculpting a more regularized energy or penalty landscape. In particular, the empirical spectral density (ESD) of DNN layer matrices displays signatures of traditionally-regularized statistical models, even in the absence of exogenously specifying traditional forms of explicit regularization. Building on relatively recent results in RMT, most notably its extension to Universality classes of Heavy-Tailed matrices, and applying them to these empirical results, we develop a theory to identify 5+1 Phases of Training, corresponding to increasing amounts of implicit self-regularization. For smaller and/or older DNNs, this implicit self-regularization is like traditional Tikhonov regularization, in that there appears to be a ``size scale'' separating signal from noise. For state-of-the-art DNNs, however, we identify a novel form of heavy-tailed self-regularization, similar to the self-organization seen in the statistical physics of disordered systems. This implicit self-regularization can depend strongly on the many knobs of the training process. In particular, by exploiting the generalization gap phenomena, we demonstrate that we can cause a small model to exhibit all 5+1 phases of training simply by changing the batch size. This demonstrates that---all else being equal---DNN optimization with larger batch sizes leads to less-well implicitly-regularized models, and it provides an explanation for the generalization gap phenomena. Coupled with work on energy landscapes and heavy-tailed spin glasses, it also suggests an explanation of why deep learning works. Joint work with Charles Martin of Calculation Consulting, Inc.</p><p class="gmail-m_2910299410442543454gmail-m_6186297207631658852gmail-m_-2549525831759316499gmail-m_4645642444737212984m_313362707539895620m_5707201597699274603m_-5457714984863114200gmail-m_-4054378802175222545gmail-m_3947158569005651454gmail-m_-1597221607886257450m_-5823613242803553233MsoPlainText"><br></p><p class="gmail-m_2910299410442543454gmail-m_6186297207631658852gmail-m_-2549525831759316499gmail-m_4645642444737212984m_313362707539895620m_5707201597699274603m_-5457714984863114200gmail-m_-4054378802175222545gmail-m_3947158569005651454gmail-m_-1597221607886257450m_-5823613242803553233MsoPlainText"><font face="arial, sans-serif">Host: <a href="mailto:avrim@ttic.edu" target="_blank">Avrim Blum</a></font></p><p class="gmail-m_2910299410442543454gmail-m_6186297207631658852gmail-m_-2549525831759316499gmail-m_4645642444737212984m_313362707539895620m_5707201597699274603m_-5457714984863114200gmail-m_-4054378802175222545gmail-m_3947158569005651454gmail-m_-1597221607886257450m_-5823613242803553233MsoPlainText"><br></p><p class="gmail-m_2910299410442543454gmail-m_6186297207631658852gmail-m_-2549525831759316499gmail-m_4645642444737212984m_313362707539895620m_5707201597699274603m_-5457714984863114200gmail-m_-4054378802175222545gmail-m_3947158569005651454gmail-m_-1597221607886257450m_-5823613242803553233MsoPlainText"><br></p></div></div><div><div dir="ltr" class="gmail-m_2910299410442543454gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><font face="arial, helvetica, sans-serif">Mary C. Marre</font><div><font face="arial, helvetica, sans-serif">Administrative Assistant</font></div><div><i><font face="arial, helvetica, sans-serif" color="#3d85c6"><b>Toyota Technological Institute</b></font></i></div><div><i><font face="arial, helvetica, sans-serif" color="#3d85c6">6045 S. Kenwood Avenue</font></i></div><div><i><font face="arial, helvetica, sans-serif" color="#3d85c6">Room 517</font></i></div><div><i><font face="arial, helvetica, sans-serif" color="#3d85c6">Chicago, IL 60637</font></i></div><div><i><font face="arial, helvetica, sans-serif">p:(773) 834-1757</font></i></div><div><i><font face="arial, helvetica, sans-serif">f: (773) 357-6970</font></i></div><div><b><i><a href="mailto:mmarre@ttic.edu" target="_blank"><font face="arial, helvetica, sans-serif">mmarre@ttic.edu</font></a></i></b></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>
</blockquote></div></div>