[Colloquium] Candidate Talk 1/24: Qi Lei (Princeton) – Theoretical Foundations of Pretrained Models

[Rob Mitchum]

*Data Science Institute/Computer Science/Statistics Candidate Seminar*

*Qi Lei*
*Associate Research Scholar*
*Princeton University*

*Monday, January 24th*
*4:30 p.m. - 5:30 p.m.*
*Live Stream <http://live.cs.uchicago.edu/priyadonti/> or Zoom (details
below)*


*Theoretical Foundations of Pretrained Models*

A pre-trained model refers to any model trained on broad data at scale and
can be adapted (e.g., fine-tuned) to a wide range of downstream tasks. The
rise of pre-trained models (e.g., BERT, GPT-3, CLIP, Codex, MAE) transforms
applications in various domains and aligns with how humans learn. Humans
and animals first establish their concepts or impressions from different
data domains and data modalities. The learned concepts then help them learn
specific tasks with minimal external instructions. Accordingly, we argue
that a pre-trained model follows a similar procedure through the lens of
deep representation learning. 1) Learn a data representation that filters
out irrelevant information from the training tasks; 2) Transfer the data
representation to downstream tasks with few labeled samples and simple
models.

This talk establishes some theoretical understanding for pre-trained models
under different settings, ranging from supervised pretraining,
meta-learning, and self-supervised learning to domain adaptation or domain
generalization. I will discuss the sufficient (and sometimes necessary)
conditions for pre-trained models to work based on the statistical relation
between training and downstream tasks. The theoretical analyses partly
answer how they work, when they fail, guide technical decisions for future
work, and inspire new methods in pre-trained models.

*Bio:* Qi Lei <https://cecilialeiqi.github.io/> is an associate research
scholar at the ECE department of Princeton University. She received her
Ph.D. from Oden Institute for Computational Engineering & Sciences at UT
Austin. She visited the Institute for Advanced Study (IAS)/Princeton for
the Theoretical Machine Learning Program from 2019-2020. Before that, she
was a research fellow at Simons Institute for the Foundations of Deep
Learning Program. Her research aims to develop sample- and computationally
efficient machine learning algorithms and bridge the theoretical and
empirical gap in machine learning. Qi has received several awards,
including the Outstanding Dissertation Award, National Initiative for
Modeling and Simulation Graduate Research Fellowship, Computing Innovative
Fellowship, and Simons-Berkeley Research Fellowship..

*Zoom Info:*
https://uchicago.zoom.us/j/92427234351?pwd=cXo1aktBazZKdVlnWjE2KzhUcWFpQT09
Meeting ID: 924 2723 4351
Passcode: ds2022


-- 
*Rob Mitchum*

*Associate Director of Communications for Data Science and Computing*
*University of Chicago*
*rmitchum at uchicago.edu <rmitchum at ci.uchicago.edu>*
*773-484-9890*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cs.uchicago.edu/pipermail/colloquium/attachments/20220114/7be3efb0/attachment-0001.html>