[Colloquium] REMINDER: 5/22 Talks at TTIC: Yasaman Bahri, Google Brain

Tue May 22 10:14:07 CDT 2018

 When:     Tuesday, May 22nd at *11:00 am*

Where:    TTIC, 6045 S Kenwood Avenue, 5th Floor, Room 526

Who:       Yasaman Bahri, Google Brain

Title:        Wide, Deep Neural Networks are Gaussian Processes

Abstract: One means of better understanding seemingly complicated models
such as deep neural networks is to connect them to other objects we already
understand. For instance, Gaussian processes are well-studied models with
well-controlled analytic properties. In his seminal work, Radford Neal
suggested thinking about inference in function space, rather than parameter
space, and in doing so established a correspondence between single-layer
fully-connected neural networks with an i.i.d prior over parameters and
certain Gaussian processes (GPs), in the limit of infinite network width.
The correspondence was, however, restricted to a single-hidden layer.

We develop this line of work and build an exact correspondence between
deep, infinitely wide neural networks and Gaussian processes.
Algorithmically, this mapping also enables a route towards Bayesian
inference with deep neural networks, without needing to instantiate a
network, which we implement on MNIST and CIFAR-10. We compare to the
performance of finite-width networks trained with standard stochastic
optimization. We find that performance increases as finite-width trained
networks are made wider and more similar to a GP, and thus that GP
predictions typically outperform those of finite-width networks.

Time permitting, I will also give some brief highlights of our related
work, studying the propagation of signals through random neural networks.
This analysis informs initializations for training ultra-deep networks with
tens of thousands of layers.

Links:
Jeffrey Pennington and Yasaman Bahri. “Geometry of Neural Network Loss
Surfaces via Random Matrix Theory.” ICML 2017. http://proceedings.mlr.p
ress/v70/pennington17a

Jaehoon Lee*, Yasaman Bahri*, Roman Novak, Samuel S. Schoenholz, Jeffrey
Pennington, Jascha Sohl-Dickstein. “Deep Neural Networks as Gaussian
Processes.” ICLR 2018. https://arxiv.org/abs/1711.00165.

Bio: Yasaman Bahri is a researcher at Google Brain working on deep
learning. The goal of her research is to advance a scientific, principled
understanding of deep learning, with an eye towards theoretical analysis
informed by careful empirical work. She got a PhD in Theoretical Condensed
Matter physics from UC Berkeley, specializing in many body physics; working
on symmetry-protected topological phases, many-body localization, non-Fermi
liquids, and topological mechanics. She is also interested in the
connections between condensed matter, theoretical physics and machine
learning.

Host: Mesrob Ohannessian <mesrob at ttic.edu>

Mary C. Marre
Administrative Assistant
*Toyota Technological Institute*
*6045 S. Kenwood Avenue*
*Room 504*
*Chicago, IL  60637*
*p:(773) 834-1757*
*f: (773) 357-6970*
*mmarre at ttic.edu <mmarre at ttic.edu>*

On Mon, May 21, 2018 at 4:37 PM, Mary Marre <mmarre at ttic.edu> wrote:

> When:     Tuesday, May 22nd at *11:00 am*
>
> Where:    TTIC, 6045 S Kenwood Avenue, 5th Floor, Room 526
>
> Who:       Yasaman Bahri, Google Brain
>
>
> Title:        Wide, Deep Neural Networks are Gaussian Processes
>
> Abstract: One means of better understanding seemingly complicated models
> such as deep neural networks is to connect them to other objects we already
> understand. For instance, Gaussian processes are well-studied models with
> well-controlled analytic properties. In his seminal work, Radford Neal
> suggested thinking about inference in function space, rather than parameter
> space, and in doing so established a correspondence between single-layer
> fully-connected neural networks with an i.i.d prior over parameters and
> certain Gaussian processes (GPs), in the limit of infinite network width.
> The correspondence was, however, restricted to a single-hidden layer.
>
> We develop this line of work and build an exact correspondence between
> deep, infinitely wide neural networks and Gaussian processes.
> Algorithmically, this mapping also enables a route towards Bayesian
> inference with deep neural networks, without needing to instantiate a
> network, which we implement on MNIST and CIFAR-10. We compare to the
> performance of finite-width networks trained with standard stochastic
> optimization. We find that performance increases as finite-width trained
> networks are made wider and more similar to a GP, and thus that GP
> predictions typically outperform those of finite-width networks.
>
> Time permitting, I will also give some brief highlights of our related
> work, studying the propagation of signals through random neural networks.
> This analysis informs initializations for training ultra-deep networks with
> tens of thousands of layers.
>
> Links:
> Jeffrey Pennington and Yasaman Bahri. “Geometry of Neural Network Loss
> Surfaces via Random Matrix Theory.” ICML 2017. http://proceedings.mlr.p
> ress/v70/pennington17a
>
> Jaehoon Lee*, Yasaman Bahri*, Roman Novak, Samuel S. Schoenholz, Jeffrey
> Pennington, Jascha Sohl-Dickstein. “Deep Neural Networks as Gaussian
> Processes.” ICLR 2018. https://arxiv.org/abs/1711.00165.
>
> Bio: Yasaman Bahri is a researcher at Google Brain working on deep
> learning. The goal of her research is to advance a scientific, principled
> understanding of deep learning, with an eye towards theoretical analysis
> informed by careful empirical work. She got a PhD in Theoretical Condensed
> Matter physics from UC Berkeley, specializing in many body physics; working
> on symmetry-protected topological phases, many-body localization, non-Fermi
> liquids, and topological mechanics. She is also interested in the
> connections between condensed matter, theoretical physics and machine
> learning.
>
>
> Host: Mesrob Ohannessian <mesrob at ttic.edu>
>
>
>
>
> Mary C. Marre
> Administrative Assistant
> *Toyota Technological Institute*
> *6045 S. Kenwood Avenue*
> *Room 504*
> *Chicago, IL  60637*
> *p:(773) 834-1757*
> *f: (773) 357-6970*
> *mmarre at ttic.edu <mmarre at ttic.edu>*
>
> On Wed, May 16, 2018 at 11:20 AM, Mary Marre <mmarre at ttic.edu> wrote:
>
>> When:     Tuesday, May 22nd at *11:00 am*
>>
>> Where:    TTIC, 6045 S Kenwood Avenue, 5th Floor, Room 526
>>
>> Who:       Yasaman Bahri, Google Brain
>>
>>
>> Title:        Wide, Deep Neural Networks are Gaussian Processes
>>
>> Abstract: One means of better understanding seemingly complicated models
>> such as deep neural networks is to connect them to other objects we already
>> understand. For instance, Gaussian processes are well-studied models with
>> well-controlled analytic properties. In his seminal work, Radford Neal
>> suggested thinking about inference in function space, rather than parameter
>> space, and in doing so established a correspondence between single-layer
>> fully-connected neural networks with an i.i.d prior over parameters and
>> certain Gaussian processes (GPs), in the limit of infinite network width.
>> The correspondence was, however, restricted to a single-hidden layer.
>>
>> We develop this line of work and build an exact correspondence between
>> deep, infinitely wide neural networks and Gaussian processes.
>> Algorithmically, this mapping also enables a route towards Bayesian
>> inference with deep neural networks, without needing to instantiate a
>> network, which we implement on MNIST and CIFAR-10. We compare to the
>> performance of finite-width networks trained with standard stochastic
>> optimization. We find that performance increases as finite-width trained
>> networks are made wider and more similar to a GP, and thus that GP
>> predictions typically outperform those of finite-width networks.
>>
>> Time permitting, I will also give some brief highlights of our related
>> work, studying the propagation of signals through random neural networks.
>> This analysis informs initializations for training ultra-deep networks with
>> tens of thousands of layers.
>>
>> Links:
>> Jeffrey Pennington and Yasaman Bahri. “Geometry of Neural Network Loss
>> Surfaces via Random Matrix Theory.” ICML 2017. http://proceedings.mlr.p
>> ress/v70/pennington17a
>>
>> Jaehoon Lee*, Yasaman Bahri*, Roman Novak, Samuel S. Schoenholz, Jeffrey
>> Pennington, Jascha Sohl-Dickstein. “Deep Neural Networks as Gaussian
>> Processes.” ICLR 2018. https://arxiv.org/abs/1711.00165.
>>
>> Bio: Yasaman Bahri is a researcher at Google Brain working on deep
>> learning. The goal of her research is to advance a scientific, principled
>> understanding of deep learning, with an eye towards theoretical analysis
>> informed by careful empirical work. She got a PhD in Theoretical Condensed
>> Matter physics from UC Berkeley, specializing in many body physics; working
>> on symmetry-protected topological phases, many-body localization, non-Fermi
>> liquids, and topological mechanics. She is also interested in the
>> connections between condensed matter, theoretical physics and machine
>> learning.
>>
>>
>> Host: Mesrob Ohannessian <mesrob at ttic.edu>
>>
>>
>> Mary C. Marre
>> Administrative Assistant
>> *Toyota Technological Institute*
>> *6045 S. Kenwood Avenue*
>> *Room 504*
>> *Chicago, IL  60637*
>> *p:(773) 834-1757*
>> *f: (773) 357-6970*
>> *mmarre at ttic.edu <mmarre at ttic.edu>*
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cs.uchicago.edu/pipermail/colloquium/attachments/20180522/972545a5/attachment-0001.html>