[Theory] 2/22 Talks at TTIC: Zhiyuan Li, Princeton University

[Mary Marre]

*When:*        Tuesday, February 22nd at* 11:00 am CT*


*Where:       *Talk will be given *live, in-person* at

                   TTIC, 6045 S. Kenwood Avenue

                   5th Floor, Room 530


*Where:*       Zoom Virtual Talk (*register in advance here
<https://uchicagogroup.zoom.us/webinar/register/WN_kfIOOoD0RLCENJB7OSyIeg>*)


*Who: *         Zhiyuan Li, Princeton University


*Title:*          Toward Mathematical Understanding of Real-life Deep
Learning

*Abstract: *There is great interest in developing a mathematical
understanding of the tremendous success of deep learning. Most of this
understanding has been done in simplified settings (depth 2 or 3; NTK
regime). This talk presents my recent works providing a mathematical
understanding of real-life nets and losses, incorporating the effect of
normalization, architectural features, stochasticity, and finite learning
rate(LR). It leverages insights from continuous mathematics (including
Stochastic Differential Equation(SDE)) which I will use to show interesting
new mechanisms for implicit regularization during training. I will finish
by presenting a new practical advance from our theoretical insights: a
robust variant of BERT (a language model at the heart of the ongoing
revolution in Natural Language Processing) called SIBERT that uses a new
scale-invariance architecture and is trainable with vanilla SGD.

*Bio: *Zhiyuan Li is a PhD candidate in the Department of Computer Science
at Princeton University, advised by Sanjeev Arora. Previously, he obtained
his bachelor’s degree in Computer Science from Tsinghua University. He has
also spent time as a research intern at Google Research. His current
research goal is to develop a mathematical theory towards a better
understanding of modern deep learning, as well as to design more efficient
and principled machine learning methods using theoretical insights. He is a
recipient of Microsoft Research PhD Fellowship in 2020.

*Host: **David McAllester* <mcallester at ttic.edu>





Mary C. Marre
Faculty Administrative Support
*Toyota Technological Institute*
*6045 S. Kenwood Avenue*
*Chicago, IL  60637*
*mmarre at ttic.edu <mmarre at ttic.edu>*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cs.uchicago.edu/pipermail/theory/attachments/20220216/6858514c/attachment-0001.html>