[Theory] NOW: 2/8 Talks at TTIC: Qi Lei, Princeton University

[Mary Marre]

*When:*        Tuesday, February 8th at* 11:00 am CT*


*Where:       *Talk will be given *live, in-person* at

                   TTIC, 6045 S. Kenwood Avenue

                   5th Floor, Room 530



*Where:*       Zoom Virtual Talk (*register in advance here*
<https://uchicagogroup.zoom.us/webinar/register/WN_6xGmjB73Qz2znt5Bm7IsDA>)


*Who: *         Qi Lei, Princeton University


*Title: *         Theoretical Foundations of Pre-trained Models

*Abstract:*  A pre-trained model refers to any model trained on broad data
at scale and can be adapted (e.g., fine-tuned) to a wide range of
downstream tasks. The rise of pre-trained models (e.g., BERT, GPT-3, CLIP,
Codex, MAE) transforms applications in various domains and aligns with how
humans learn. Humans and animals first establish their concepts or
impressions from different data domains and data modalities. The learned
concepts then help them learn specific tasks with minimal external
instructions. Accordingly, we argue that a pre-trained model follows a
similar procedure through the lens of deep representation learning. 1)
Learn a data representation that filters out irrelevant information from
the training tasks; 2) Transfer the data representation to downstream tasks
with few labeled samples and simple models.

This talk establishes some theoretical understanding for pre-trained models
under different settings, ranging from supervised pre-training,
meta-learning, and self-supervised learning to domain adaptation or domain
generalization. I will discuss the sufficient (and sometimes necessary)
conditions for pre-trained models to work based on the statistical relation
between training and downstream tasks. The theoretical analyses partly
answer how they work, when they fail, guide technical decisions for future
work, and inspire new methods in pre-trained models.

*Bio: *Qi Lei is an associate research scholar at the ECE department of
Princeton University. She received her Ph.D. from Oden Institute for
Computational Engineering & Sciences at UT Austin. She visited the
Institute for Advanced Study (IAS)/Princeton for the Theoretical Machine
Learning Program from 2019-2020. Before that, she was a research fellow at
Simons Institute for the Foundations of Deep Learning Program. Her research
aims to develop sample- and computationally efficient machine learning
algorithms and bridge the theoretical and empirical gap in machine
learning. Qi has received several awards, including the Outstanding
Dissertation Award, National Initiative for Modeling and Simulation
Graduate Research Fellowship, Computing Innovative Fellowship, and
Simons-Berkeley Research Fellowship.

*Host: **Nathan Srebro* <nati at ttic.edu>

Mary C. Marre
Faculty Administrative Support
*Toyota Technological Institute*
*6045 S. Kenwood Avenue*
*Chicago, IL  60637*
*mmarre at ttic.edu <mmarre at ttic.edu>*


On Tue, Feb 8, 2022 at 10:12 AM Mary Marre <mmarre at ttic.edu> wrote:

> *When:*        Tuesday, February 8th at* 11:00 am CT*
>
>
> *Where:       *Talk will be given *live, in-person* at
>
>                    TTIC, 6045 S. Kenwood Avenue
>
>                    5th Floor, Room 530
>
>
>
> *Where:*       Zoom Virtual Talk (*register in advance here*
> <https://uchicagogroup.zoom.us/webinar/register/WN_6xGmjB73Qz2znt5Bm7IsDA>
> )
>
>
> *Who: *         Qi Lei, Princeton University
>
>
> *Title: *         Theoretical Foundations of Pre-trained Models
>
> *Abstract:*  A pre-trained model refers to any model trained on broad
> data at scale and can be adapted (e.g., fine-tuned) to a wide range of
> downstream tasks. The rise of pre-trained models (e.g., BERT, GPT-3, CLIP,
> Codex, MAE) transforms applications in various domains and aligns with how
> humans learn. Humans and animals first establish their concepts or
> impressions from different data domains and data modalities. The learned
> concepts then help them learn specific tasks with minimal external
> instructions. Accordingly, we argue that a pre-trained model follows a
> similar procedure through the lens of deep representation learning. 1)
> Learn a data representation that filters out irrelevant information from
> the training tasks; 2) Transfer the data representation to downstream tasks
> with few labeled samples and simple models.
>
> This talk establishes some theoretical understanding for pre-trained
> models under different settings, ranging from supervised pre-training,
> meta-learning, and self-supervised learning to domain adaptation or domain
> generalization. I will discuss the sufficient (and sometimes necessary)
> conditions for pre-trained models to work based on the statistical relation
> between training and downstream tasks. The theoretical analyses partly
> answer how they work, when they fail, guide technical decisions for future
> work, and inspire new methods in pre-trained models.
>
> *Bio: *Qi Lei is an associate research scholar at the ECE department of
> Princeton University. She received her Ph.D. from Oden Institute for
> Computational Engineering & Sciences at UT Austin. She visited the
> Institute for Advanced Study (IAS)/Princeton for the Theoretical Machine
> Learning Program from 2019-2020. Before that, she was a research fellow at
> Simons Institute for the Foundations of Deep Learning Program. Her research
> aims to develop sample- and computationally efficient machine learning
> algorithms and bridge the theoretical and empirical gap in machine
> learning. Qi has received several awards, including the Outstanding
> Dissertation Award, National Initiative for Modeling and Simulation
> Graduate Research Fellowship, Computing Innovative Fellowship, and
> Simons-Berkeley Research Fellowship.
>
> *Host: **Nathan Srebro* <nati at ttic.edu>
>
>
>
> Mary C. Marre
> Faculty Administrative Support
> *Toyota Technological Institute*
> *6045 S. Kenwood Avenue*
> *Chicago, IL  60637*
> *mmarre at ttic.edu <mmarre at ttic.edu>*
>
>
> On Mon, Feb 7, 2022 at 4:01 PM Mary Marre <mmarre at ttic.edu> wrote:
>
>> *When:*        Tuesday, February 8th at* 11:00 am CT*
>>
>>
>> *Where:       *Talk will be given *live, in-person* at
>>
>>                    TTIC, 6045 S. Kenwood Avenue
>>
>>                    5th Floor, Room 530
>>
>>
>>
>> *Where:*       Zoom Virtual Talk (*register in advance here*
>> <https://uchicagogroup.zoom.us/webinar/register/WN_6xGmjB73Qz2znt5Bm7IsDA>
>> )
>>
>>
>> *Who: *         Qi Lei, Princeton University
>>
>>
>> *Title: *         Theoretical Foundations of Pre-trained Models
>>
>> *Abstract:*  A pre-trained model refers to any model trained on broad
>> data at scale and can be adapted (e.g., fine-tuned) to a wide range of
>> downstream tasks. The rise of pre-trained models (e.g., BERT, GPT-3, CLIP,
>> Codex, MAE) transforms applications in various domains and aligns with how
>> humans learn. Humans and animals first establish their concepts or
>> impressions from different data domains and data modalities. The learned
>> concepts then help them learn specific tasks with minimal external
>> instructions. Accordingly, we argue that a pre-trained model follows a
>> similar procedure through the lens of deep representation learning. 1)
>> Learn a data representation that filters out irrelevant information from
>> the training tasks; 2) Transfer the data representation to downstream
>> tasks with few labeled samples and simple models.
>>
>> This talk establishes some theoretical understanding for pre-trained
>> models under different settings, ranging from supervised pre-training,
>> meta-learning, and self-supervised learning to domain adaptation or domain
>> generalization. I will discuss the sufficient (and sometimes necessary)
>> conditions for pre-trained models to work based on the statistical relation
>> between training and downstream tasks. The theoretical analyses partly
>> answer how they work, when they fail, guide technical decisions for future
>> work, and inspire new methods in pre-trained models.
>>
>> *Bio: *Qi Lei is an associate research scholar at the ECE department of
>> Princeton University. She received her Ph.D. from Oden Institute for
>> Computational Engineering & Sciences at UT Austin. She visited the
>> Institute for Advanced Study (IAS)/Princeton for the Theoretical Machine
>> Learning Program from 2019-2020. Before that, she was a research fellow at
>> Simons Institute for the Foundations of Deep Learning Program. Her research
>> aims to develop sample- and computationally efficient machine learning
>> algorithms and bridge the theoretical and empirical gap in machine
>> learning. Qi has received several awards, including the Outstanding
>> Dissertation Award, National Initiative for Modeling and Simulation
>> Graduate Research Fellowship, Computing Innovative Fellowship, and
>> Simons-Berkeley Research Fellowship.
>>
>> *Host: **Nathan Srebro* <nati at ttic.edu>
>>
>>
>>
>>
>> Mary C. Marre
>> Faculty Administrative Support
>> *Toyota Technological Institute*
>> *6045 S. Kenwood Avenue*
>> *Chicago, IL  60637*
>> *mmarre at ttic.edu <mmarre at ttic.edu>*
>>
>>
>> On Thu, Feb 3, 2022 at 6:11 PM Mary Marre <mmarre at ttic.edu> wrote:
>>
>>> *When:*        Tuesday, February 8th at* 11:00 am CT*
>>>
>>>
>>> *Where:       *Talk will be given *live, in-person* at
>>>
>>>                    TTIC, 6045 S. Kenwood Avenue
>>>
>>>                    5th Floor, Room 530
>>>
>>>
>>>
>>> *Where:*      Zoom Virtual Talk (*register in advance here*
>>> <https://uchicagogroup.zoom.us/webinar/register/WN_6xGmjB73Qz2znt5Bm7IsDA>
>>> )
>>>
>>>
>>> *Who: *        Qi Lei, Princeton University
>>>
>>>
>>> *Title: *       Theoretical Foundations of Pre-trained Models
>>>
>>> *Abstract:*  A pre-trained model refers to any model trained on broad
>>> data at scale and can be adapted (e.g., fine-tuned) to a wide range of
>>> downstream tasks. The rise of pre-trained models (e.g., BERT, GPT-3, CLIP,
>>> Codex, MAE) transforms applications in various domains and aligns with how
>>> humans learn. Humans and animals first establish their concepts or
>>> impressions from different data domains and data modalities. The learned
>>> concepts then help them learn specific tasks with minimal external
>>> instructions. Accordingly, we argue that a pre-trained model follows a
>>> similar procedure through the lens of deep representation learning. 1)
>>> Learn a data representation that filters out irrelevant information from
>>> the training tasks; 2) Transfer the data representation to downstream tasks
>>> with few labeled samples and simple models.
>>>
>>> This talk establishes some theoretical understanding for pre-trained
>>> models under different settings, ranging from supervised pre-training,
>>> meta-learning, and self-supervised learning to domain adaptation or domain
>>> generalization. I will discuss the sufficient (and sometimes necessary)
>>> conditions for pre-trained models to work based on the statistical relation
>>> between training and downstream tasks. The theoretical analyses partly
>>> answer how they work, when they fail, guide technical decisions for future
>>> work, and inspire new methods in pre-trained models.
>>>
>>> *Bio: *Qi Lei is an associate research scholar at the ECE department of
>>> Princeton University. She received her Ph.D. from Oden Institute for
>>> Computational Engineering & Sciences at UT Austin. She visited the
>>> Institute for Advanced Study (IAS)/Princeton for the Theoretical Machine
>>> Learning Program from 2019-2020. Before that, she was a research fellow at
>>> Simons Institute for the Foundations of Deep Learning Program. Her research
>>> aims to develop sample- and computationally efficient machine learning
>>> algorithms and bridge the theoretical and empirical gap in machine
>>> learning. Qi has received several awards, including the Outstanding
>>> Dissertation Award, National Initiative for Modeling and Simulation
>>> Graduate Research Fellowship, Computing Innovative Fellowship, and
>>> Simons-Berkeley Research Fellowship.
>>>
>>> *Host: **Nathan Srebro* <nati at ttic.edu>
>>>
>>>
>>>
>>> Mary C. Marre
>>> Faculty Administrative Support
>>> *Toyota Technological Institute*
>>> *6045 S. Kenwood Avenue*
>>> *Chicago, IL  60637*
>>> *mmarre at ttic.edu <mmarre at ttic.edu>*
>>>
>>>
>>> On Tue, Feb 1, 2022 at 9:47 PM Mary Marre <mmarre at ttic.edu> wrote:
>>>
>>>> *When:*      Tuesday, February 8th at* 11:00 am CT*
>>>>
>>>>
>>>>
>>>> *Where:*     Zoom Virtual Talk (*register in advance here*
>>>> <https://uchicagogroup.zoom.us/webinar/register/WN_6xGmjB73Qz2znt5Bm7IsDA>
>>>> )
>>>>
>>>>
>>>> *Who: *        Qi Lei, Princeton University
>>>>
>>>>
>>>> *Title: *       Theoretical Foundations of Pre-trained Models
>>>>
>>>> *Abstract:*  A pre-trained model refers to any model trained on broad
>>>> data at scale and can be adapted (e.g., fine-tuned) to a wide range of
>>>> downstream tasks. The rise of pre-trained models (e.g., BERT, GPT-3, CLIP,
>>>> Codex, MAE) transforms applications in various domains and aligns with how
>>>> humans learn. Humans and animals first establish their concepts or
>>>> impressions from different data domains and data modalities. The learned
>>>> concepts then help them learn specific tasks with minimal external
>>>> instructions. Accordingly, we argue that a pre-trained model follows a
>>>> similar procedure through the lens of deep representation learning. 1)
>>>> Learn a data representation that filters out irrelevant information from
>>>> the training tasks; 2) Transfer the data representation to downstream tasks
>>>> with few labeled samples and simple models.
>>>>
>>>> This talk establishes some theoretical understanding for pre-trained
>>>> models under different settings, ranging from supervised pre-training,
>>>> meta-learning, and self-supervised learning to domain adaptation or domain
>>>> generalization. I will discuss the sufficient (and sometimes necessary)
>>>> conditions for pre-trained models to work based on the statistical relation
>>>> between training and downstream tasks. The theoretical analyses partly
>>>> answer how they work, when they fail, guide technical decisions for future
>>>> work, and inspire new methods in pre-trained models.
>>>>
>>>> *Bio: *Qi Lei is an associate research scholar at the ECE department
>>>> of Princeton University. She received her Ph.D. from Oden Institute for
>>>> Computational Engineering & Sciences at UT Austin. She visited the
>>>> Institute for Advanced Study (IAS)/Princeton for the Theoretical Machine
>>>> Learning Program from 2019-2020. Before that, she was a research fellow at
>>>> Simons Institute for the Foundations of Deep Learning Program. Her research
>>>> aims to develop sample- and computationally efficient machine learning
>>>> algorithms and bridge the theoretical and empirical gap in machine
>>>> learning. Qi has received several awards, including the Outstanding
>>>> Dissertation Award, National Initiative for Modeling and Simulation
>>>> Graduate Research Fellowship, Computing Innovative Fellowship, and
>>>> Simons-Berkeley Research Fellowship.
>>>>
>>>> *Host: **Nathan Srebro* <nati at ttic.edu>
>>>>
>>>>
>>>>
>>>> Mary C. Marre
>>>> Faculty Administrative Support
>>>> *Toyota Technological Institute*
>>>> *6045 S. Kenwood Avenue*
>>>> *Chicago, IL  60637*
>>>> *mmarre at ttic.edu <mmarre at ttic.edu>*
>>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cs.uchicago.edu/pipermail/theory/attachments/20220208/c8462153/attachment-0001.html>