<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div dir="auto" class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div class=""><div dir="auto" class="" style="word-wrap: break-word; line-break: after-white-space;"><font face="Arial" class=""><font color="#831100" class=""><span class="" style="font-size: 14px;"><b class="">University of Chicago and Toyota Technological Institute at Chicago<br class="">Machine Learning Seminar Series</b></span></font><br class=""><br class=""><br class=""><b class="" style="font-size: 14px;">Sebastian Stitch</b><br class=""><span class="" style="font-size: 14px;">EPFL, Switzerland</span><br class=""><br class=""><b class="" style="font-size: 14px;">Wednesday, March 13, 2019 at 1:00 pm<br class="">Harper Center (Booth) Room 219</b><br class=""><br class=""><br class=""><b class="" style="font-size: 14px;">Title:</b></font></div><div dir="auto" class="" style="word-wrap: break-word; line-break: after-white-space;"><font face="Arial" class="" style="font-size: 14px;"><span class="">Error Feedback for Communication Efficient SGD</span> </font><font face="Arial" class="" style="font-size: 14px;"> <br class=""><b class=""><font face="Arial" class="" style="font-weight: normal;"><b class=""><br class=""></b></font></b></font></div><div dir="auto" class="" style="word-wrap: break-word; line-break: after-white-space;"><font face="Arial" class=""><span class=""><font face="Arial" class="" style="font-size: 14px;"><b class="">Abstract:</b><br class=""></font><div class=""><span class="" style="font-size: 14px;">Huge scale machine learning problems are nowadays tackled by<font class=""> distributed optimization algorithms, i.e. algorithms that leverage the compute power of many devices for training. The communication overhead is a key bottleneck that hinders perfect scalability. Various recent works proposed to use Sequantization or sparsification techniques to reduce the amount of data that needs to be communicated. We analyze Stochastic Gradient Descent (SGD) with k-sparsification (for instance top-k or random-k) and compression (for instance quantization) and show that these schemes converge at the same rate as vanilla SGD when equipped with error compensation (i.e. keeping track of accumulated errors in memory). That is, communication can be reduced by a factor of the dimension of the problem (sometimes even more) whilst still converging at the same rate.</font></span></div><font face="Arial" class="" style="font-size: 14px;"><br class=""><b class="">Bio:</b><br class=""></font><div class=""><span class="" style="font-size: 14px;">Dr. Sebastian Stich is a postdoctoral researcher at EPFL in Switzerland, working at the Machine Learning and Optimization Laboratory of Prof. Martin Jaggi. He received a MSc in Mathematics with distinction from ETH Zurich in 2010 and a PhD in Theoretical Computer Science from ETH Zurich in 2014. Before joining EPFL, he held for two years a postdoctoral position at UCLouvain to work with Prof. Yurii Nesterov and Prof. François Glineur on coordinate descent methods for large scale optimization problems. Dr. Stich is broadly interested in the complexity analysis of the optimization algorithms that are used in nowadays machine learning applications, with recent focus on distributed algorithms that allow to tackle high dimensional problems. Website: <a href="http://www.sstich.ch" class="">www.sstich.ch</a></span></div><font face="Arial" class="" style="font-size: 14px;"><br class=""><br class=""><b class=""><i class="">Host: </i></b></font></span></font><font face="Arial" class=""><span class="" style="font-size: 14px;"><b class=""><i class="">Nati Srebro </i></b></span></font></div></div><div dir="auto" class="" style="word-wrap: break-word; line-break: after-white-space;"><font face="Arial" class=""><span class="" style="font-size: 14px;"><b class=""><i class=""><br class=""></i></b></span></font></div><div dir="auto" class="" style="word-wrap: break-word; line-break: after-white-space;"><font face="Arial" class=""><span class="" style="font-size: 14px;"><b class=""><i class=""><br class=""></i></b></span></font></div><div dir="auto" class="" style="word-wrap: break-word; line-break: after-white-space;"><font face="Arial" class=""><span class="" style="font-size: 14px;"><b class=""><i class=""><br class=""></i></b></span></font></div><div dir="auto" class="" style="word-wrap: break-word; line-break: after-white-space;"></div></div></body></html>