<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body>
<div>
<div>
<div dir="ltr" style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Please remove me from the list </div>
</div>
<div id="ms-outlook-mobile-signature">
<div><br>
</div>
Get <a href="https://aka.ms/o0ukef">Outlook for iOS</a></div>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> cs <cs-bounces+sunalber=cs.uchicago.edu@mailman.cs.uchicago.edu> on behalf of Megan Woodward <meganwoodward@uchicago.edu><br>
<b>Sent:</b> Tuesday, March 22, 2022 8:12:19 AM<br>
<b>To:</b> cs@cs.uchicago.edu <cs@cs.uchicago.edu>; colloquium@cs.uchicago.edu <colloquium@cs.uchicago.edu><br>
<b>Subject:</b> [CS] Greg Pauloski MS Presentation/Mar 23, 2022</font>
<div> </div>
</div>
<div>
<div class="" style="word-wrap:break-word; line-break:after-white-space"><span class="" style="font-size:14.666666984558105px">This is an announcement of Greg Pauloski's MS Presentation</span><br class="" style="font-size:14.666666984558105px">
<span class="" style="font-size:14.666666984558105px">===============================================</span><br class="" style="font-size:14.666666984558105px">
<span class="" style="font-size:14.666666984558105px">Candidate: Greg Pauloski</span><br class="" style="font-size:14.666666984558105px">
<br class="" style="font-size:14.666666984558105px">
<span class="" style="font-size:14.666666984558105px">Date: Wednesday, March 23, 2022</span><br class="" style="font-size:14.666666984558105px">
<br class="" style="font-size:14.666666984558105px">
<span class="" style="font-size:14.666666984558105px">Time: 3 pm CST</span><br class="" style="font-size:14.666666984558105px">
<br class="" style="font-size:14.666666984558105px">
<span class="" style="font-size:14.666666984558105px">Remote Location: </span><a href="https://uchicago.zoom.us/j/96526880582?pwd=V2pjQ3pvdFk3cWFNWWJyNW9SUnVDZz09" class="" style="font-size:14.666666984558105px">https://uchicago.zoom.us/j/96526880582?pwd=V2pjQ3pvdFk3cWFNWWJyNW9SUnVDZz09</a><br class="" style="font-size:14.666666984558105px">
<br class="" style="font-size:14.666666984558105px">
<span class="" style="font-size:14.666666984558105px">M.S. Paper Title: Scalable Deep Neural Network Training with Distributed K-FAC</span><br class="" style="font-size:14.666666984558105px">
<br class="" style="font-size:14.666666984558105px">
<span class="" style="font-size:14.666666984558105px">Abstract: Scaling deep neural network training to more processors and larger batch sizes is key to reducing end-to-end training time; yet, maintaining comparable convergence and hardware utilization at larger
scales is challenging. Increases in training scales have enabled natural gradient optimization methods as a reasonable alternative to stochastic gradient descent (SGD) and variants thereof. Kronecker-factored Approximate Curvature (K-FAC), a natural gradient
method, has recently been shown to converge with fewer iterations in deep neural network (DNN) training than SGD; however, K-FAC’s larger memory footprint and increased communication necessitates careful distribution of work for efficient usage. This thesis
investigates scalable K-FAC algorithms to understand K-FAC’s applicability in large-scale deep neural network training and presents KAISA, a K-FAC-enabled, Adaptable, Improved, and Scalable second-order optimizer framework. Specifically, layer-wise distribution
strategies, inverse-free second-order gradient evaluation, dynamic K-FAC update decoupling, and more are explored with the goal of preserving convergence while minimizing training time. KAISA can adapt the memory footprint, communication, and computation given
specific models and hardware to improve performance and increase scalability, and this adaptable distribution scheme generalizes existing strategies while providing a framework for scaling second-order methods beyond K-FAC. Compared to the original optimizers,
KAISA converges 18.1–36.3% faster across applications with the same global batch size. Under a fixed memory budget, KAISA converges 32.5% and 41.6% faster in ResNet-50 and BERT-Large, respectively. KAISA can balance memory and communication to achieve scaling
efficiency equal to or better than the baseline optimizers.</span><br class="" style="font-size:14.666666984558105px">
<br class="" style="font-size:14.666666984558105px">
<span class="" style="font-size:14.666666984558105px">Advisors: Kyle Chard and Ian Foster</span><br class="" style="font-size:14.666666984558105px">
<br class="" style="font-size:14.666666984558105px">
<span class="" style="font-size:14.666666984558105px">Committee Members: Kyle Chard, Ian Foster, and Zhao Zhang</span>
<div class=""><span class="" style="font-size:14.666666984558105px"><br class="">
</span></div>
<div class=""><span class="" style="font-size:14.666666984558105px"><br class="">
</span></div>
<div class=""></div>
</div>
<div class="" style="word-wrap:break-word; line-break:after-white-space">
<div class=""><span class="" style="font-size:14.666666984558105px"><br class="">
</span><br class="">
</div>
</div>
</div>
</body>
</html>