[Theory] NOW: 2/22 Talks at TTIC: Zhiyuan Li, Princeton University

[Mary Marre]

*When:*        Tuesday, February 22nd at* 11:00 am CT*


*Where:       *Talk will be given *live, in-person* at

                   TTIC, 6045 S. Kenwood Avenue

                   5th Floor, Room 530


*Where:*       Zoom Virtual Talk (*register in advance here
<https://uchicagogroup.zoom.us/webinar/register/WN_kfIOOoD0RLCENJB7OSyIeg>*)


*Who: *         Zhiyuan Li, Princeton University


*Title:*          Toward Mathematical Understanding of Real-life Deep
Learning

*Abstract: *There is great interest in developing a mathematical
understanding of the tremendous success of deep learning. Most of this
understanding has been done in simplified settings (depth 2 or 3; NTK
regime). This talk presents my recent works providing a mathematical
understanding of real-life nets and losses, incorporating the effect of
normalization, architectural features, stochasticity, and finite learning
rate(LR). It leverages insights from continuous mathematics (including
Stochastic Differential Equation(SDE)) which I will use to show interesting
new mechanisms for implicit regularization during training. I will finish
by presenting a new practical advance from our theoretical insights: a
robust variant of BERT (a language model at the heart of the ongoing
revolution in Natural Language Processing) called SIBERT that uses a new
scale-invariance architecture and is trainable with vanilla SGD.

*Bio: *Zhiyuan Li is a PhD candidate in the Department of Computer Science
at Princeton University, advised by Sanjeev Arora. Previously, he obtained
his bachelor’s degree in Computer Science from Tsinghua University. He has
also spent time as a research intern at Google Research. His current
research goal is to develop a mathematical theory towards a better
understanding of modern deep learning, as well as to design more efficient
and principled machine learning methods using theoretical insights. He is a
recipient of Microsoft Research PhD Fellowship in 2020.

*Host: **David McAllester* <mcallester at ttic.edu>


Mary C. Marre
Faculty Administrative Support
*Toyota Technological Institute*
*6045 S. Kenwood Avenue*
*Chicago, IL  60637*
*mmarre at ttic.edu <mmarre at ttic.edu>*


On Tue, Feb 22, 2022 at 10:11 AM Mary Marre <mmarre at ttic.edu> wrote:

> *When:*        Tuesday, February 22nd at* 11:00 am CT*
>
>
> *Where:       *Talk will be given *live, in-person* at
>
>                    TTIC, 6045 S. Kenwood Avenue
>
>                    5th Floor, Room 530
>
>
> *Where:*       Zoom Virtual Talk (*register in advance here
> <https://uchicagogroup.zoom.us/webinar/register/WN_kfIOOoD0RLCENJB7OSyIeg>*
> )
>
>
> *Who: *         Zhiyuan Li, Princeton University
>
>
> *Title:*          Toward Mathematical Understanding of Real-life Deep
> Learning
>
> *Abstract: *There is great interest in developing a mathematical
> understanding of the tremendous success of deep learning. Most of this
> understanding has been done in simplified settings (depth 2 or 3; NTK
> regime). This talk presents my recent works providing a mathematical
> understanding of real-life nets and losses, incorporating the effect of
> normalization, architectural features, stochasticity, and finite learning
> rate(LR). It leverages insights from continuous mathematics (including
> Stochastic Differential Equation(SDE)) which I will use to show interesting
> new mechanisms for implicit regularization during training. I will finish
> by presenting a new practical advance from our theoretical insights: a
> robust variant of BERT (a language model at the heart of the ongoing
> revolution in Natural Language Processing) called SIBERT that uses a new
> scale-invariance architecture and is trainable with vanilla SGD.
>
> *Bio: *Zhiyuan Li is a PhD candidate in the Department of Computer
> Science at Princeton University, advised by Sanjeev Arora. Previously, he
> obtained his bachelor’s degree in Computer Science from Tsinghua
> University. He has also spent time as a research intern at Google Research.
> His current research goal is to develop a mathematical theory towards a
> better understanding of modern deep learning, as well as to design more
> efficient and principled machine learning methods using theoretical
> insights. He is a recipient of Microsoft Research PhD Fellowship in 2020.
>
> *Host: **David McAllester* <mcallester at ttic.edu>
>
>
>
> Mary C. Marre
> Faculty Administrative Support
> *Toyota Technological Institute*
> *6045 S. Kenwood Avenue*
> *Chicago, IL  60637*
> *mmarre at ttic.edu <mmarre at ttic.edu>*
>
>
> On Mon, Feb 21, 2022 at 3:41 PM Mary Marre <mmarre at ttic.edu> wrote:
>
>> *When:*        Tuesday, February 22nd at* 11:00 am CT*
>>
>>
>> *Where:       *Talk will be given *live, in-person* at
>>
>>                    TTIC, 6045 S. Kenwood Avenue
>>
>>                    5th Floor, Room 530
>>
>>
>> *Where:*       Zoom Virtual Talk (*register in advance here
>> <https://uchicagogroup.zoom.us/webinar/register/WN_kfIOOoD0RLCENJB7OSyIeg>*
>> )
>>
>>
>> *Who: *         Zhiyuan Li, Princeton University
>>
>>
>> *Title:*          Toward Mathematical Understanding of Real-life Deep
>> Learning
>>
>> *Abstract: *There is great interest in developing a mathematical
>> understanding of the tremendous success of deep learning. Most of this
>> understanding has been done in simplified settings (depth 2 or 3; NTK
>> regime). This talk presents my recent works providing a mathematical
>> understanding of real-life nets and losses, incorporating the effect of
>> normalization, architectural features, stochasticity, and finite learning
>> rate(LR). It leverages insights from continuous mathematics (including
>> Stochastic Differential Equation(SDE)) which I will use to show interesting
>> new mechanisms for implicit regularization during training. I will finish
>> by presenting a new practical advance from our theoretical insights: a
>> robust variant of BERT (a language model at the heart of the ongoing
>> revolution in Natural Language Processing) called SIBERT that uses a new
>> scale-invariance architecture and is trainable with vanilla SGD.
>>
>> *Bio: *Zhiyuan Li is a PhD candidate in the Department of Computer
>> Science at Princeton University, advised by Sanjeev Arora. Previously, he
>> obtained his bachelor’s degree in Computer Science from Tsinghua
>> University. He has also spent time as a research intern at Google Research.
>> His current research goal is to develop a mathematical theory towards a
>> better understanding of modern deep learning, as well as to design more
>> efficient and principled machine learning methods using theoretical
>> insights. He is a recipient of Microsoft Research PhD Fellowship in 2020.
>>
>> *Host: **David McAllester* <mcallester at ttic.edu>
>>
>>
>>
>> Mary C. Marre
>> Faculty Administrative Support
>> *Toyota Technological Institute*
>> *6045 S. Kenwood Avenue*
>> *Chicago, IL  60637*
>> *mmarre at ttic.edu <mmarre at ttic.edu>*
>>
>>
>> On Wed, Feb 16, 2022 at 3:10 PM Mary Marre <mmarre at ttic.edu> wrote:
>>
>>> *When:*        Tuesday, February 22nd at* 11:00 am CT*
>>>
>>>
>>> *Where:       *Talk will be given *live, in-person* at
>>>
>>>                    TTIC, 6045 S. Kenwood Avenue
>>>
>>>                    5th Floor, Room 530
>>>
>>>
>>> *Where:*       Zoom Virtual Talk (*register in advance here
>>> <https://uchicagogroup.zoom.us/webinar/register/WN_kfIOOoD0RLCENJB7OSyIeg>*
>>> )
>>>
>>>
>>> *Who: *         Zhiyuan Li, Princeton University
>>>
>>>
>>> *Title:*          Toward Mathematical Understanding of Real-life Deep
>>> Learning
>>>
>>> *Abstract: *There is great interest in developing a mathematical
>>> understanding of the tremendous success of deep learning. Most of this
>>> understanding has been done in simplified settings (depth 2 or 3; NTK
>>> regime). This talk presents my recent works providing a mathematical
>>> understanding of real-life nets and losses, incorporating the effect of
>>> normalization, architectural features, stochasticity, and finite learning
>>> rate(LR). It leverages insights from continuous mathematics (including
>>> Stochastic Differential Equation(SDE)) which I will use to show interesting
>>> new mechanisms for implicit regularization during training. I will finish
>>> by presenting a new practical advance from our theoretical insights: a
>>> robust variant of BERT (a language model at the heart of the ongoing
>>> revolution in Natural Language Processing) called SIBERT that uses a new
>>> scale-invariance architecture and is trainable with vanilla SGD.
>>>
>>> *Bio: *Zhiyuan Li is a PhD candidate in the Department of Computer
>>> Science at Princeton University, advised by Sanjeev Arora. Previously, he
>>> obtained his bachelor’s degree in Computer Science from Tsinghua
>>> University. He has also spent time as a research intern at Google Research.
>>> His current research goal is to develop a mathematical theory towards a
>>> better understanding of modern deep learning, as well as to design more
>>> efficient and principled machine learning methods using theoretical
>>> insights. He is a recipient of Microsoft Research PhD Fellowship in 2020.
>>>
>>> *Host: **David McAllester* <mcallester at ttic.edu>
>>>
>>>
>>>
>>>
>>>
>>> Mary C. Marre
>>> Faculty Administrative Support
>>> *Toyota Technological Institute*
>>> *6045 S. Kenwood Avenue*
>>> *Chicago, IL  60637*
>>> *mmarre at ttic.edu <mmarre at ttic.edu>*
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cs.uchicago.edu/pipermail/theory/attachments/20220222/0e180c92/attachment-0001.html>