[Colloquium] REMINDER: 5/22 Talks at TTIC: Yasaman Bahri, Google Brain

[Mary Marre via Colloquium]

 When:     Tuesday, May 22nd at *11:00 am*

Where:    TTIC, 6045 S Kenwood Avenue, 5th Floor, Room 526

Who:       Yasaman Bahri, Google Brain


Title:        Wide, Deep Neural Networks are Gaussian Processes

Abstract: One means of better understanding seemingly complicated models
such as deep neural networks is to connect them to other objects we already
understand. For instance, Gaussian processes are well-studied models with
well-controlled analytic properties. In his seminal work, Radford Neal
suggested thinking about inference in function space, rather than parameter
space, and in doing so established a correspondence between single-layer
fully-connected neural networks with an i.i.d prior over parameters and
certain Gaussian processes (GPs), in the limit of infinite network width.
The correspondence was, however, restricted to a single-hidden layer.

We develop this line of work and build an exact correspondence between
deep, infinitely wide neural networks and Gaussian processes.
Algorithmically, this mapping also enables a route towards Bayesian
inference with deep neural networks, without needing to instantiate a
network, which we implement on MNIST and CIFAR-10. We compare to the
performance of finite-width networks trained with standard stochastic
optimization. We find that performance increases as finite-width trained
networks are made wider and more similar to a GP, and thus that GP
predictions typically outperform those of finite-width networks.

Time permitting, I will also give some brief highlights of our related
work, studying the propagation of signals through random neural networks.
This analysis informs initializations for training ultra-deep networks with
tens of thousands of layers.

Links:
Jeffrey Pennington and Yasaman Bahri. “Geometry of Neural Network Loss
Surfaces via Random Matrix Theory.” ICML 2017. http://proceedings.mlr.p
ress/v70/pennington17a

Jaehoon Lee*, Yasaman Bahri*, Roman Novak, Samuel S. Schoenholz, Jeffrey
Pennington, Jascha Sohl-Dickstein. “Deep Neural Networks as Gaussian
Processes.” ICLR 2018. https://arxiv.org/abs/1711.00165.

Bio: Yasaman Bahri is a researcher at Google Brain working on deep
learning. The goal of her research is to advance a scientific, principled
understanding of deep learning, with an eye towards theoretical analysis
informed by careful empirical work. She got a PhD in Theoretical Condensed
Matter physics from UC Berkeley, specializing in many body physics; working
on symmetry-protected topological phases, many-body localization, non-Fermi
liquids, and topological mechanics. She is also interested in the
connections between condensed matter, theoretical physics and machine
learning.


Host: Mesrob Ohannessian <mesrob at ttic.edu>




Mary C. Marre
Administrative Assistant
*Toyota Technological Institute*
*6045 S. Kenwood Avenue*
*Room 504*
*Chicago, IL  60637*
*p:(773) 834-1757*
*f: (773) 357-6970*
*mmarre at ttic.edu <mmarre at ttic.edu>*

On Wed, May 16, 2018 at 11:20 AM, Mary Marre <mmarre at ttic.edu> wrote:

> When:     Tuesday, May 22nd at *11:00 am*
>
> Where:    TTIC, 6045 S Kenwood Avenue, 5th Floor, Room 526
>
> Who:       Yasaman Bahri, Google Brain
>
>
> Title:        Wide, Deep Neural Networks are Gaussian Processes
>
> Abstract: One means of better understanding seemingly complicated models
> such as deep neural networks is to connect them to other objects we already
> understand. For instance, Gaussian processes are well-studied models with
> well-controlled analytic properties. In his seminal work, Radford Neal
> suggested thinking about inference in function space, rather than parameter
> space, and in doing so established a correspondence between single-layer
> fully-connected neural networks with an i.i.d prior over parameters and
> certain Gaussian processes (GPs), in the limit of infinite network width.
> The correspondence was, however, restricted to a single-hidden layer.
>
> We develop this line of work and build an exact correspondence between
> deep, infinitely wide neural networks and Gaussian processes.
> Algorithmically, this mapping also enables a route towards Bayesian
> inference with deep neural networks, without needing to instantiate a
> network, which we implement on MNIST and CIFAR-10. We compare to the
> performance of finite-width networks trained with standard stochastic
> optimization. We find that performance increases as finite-width trained
> networks are made wider and more similar to a GP, and thus that GP
> predictions typically outperform those of finite-width networks.
>
> Time permitting, I will also give some brief highlights of our related
> work, studying the propagation of signals through random neural networks.
> This analysis informs initializations for training ultra-deep networks with
> tens of thousands of layers.
>
> Links:
> Jeffrey Pennington and Yasaman Bahri. “Geometry of Neural Network Loss
> Surfaces via Random Matrix Theory.” ICML 2017. http://proceedings.mlr.p
> ress/v70/pennington17a
>
> Jaehoon Lee*, Yasaman Bahri*, Roman Novak, Samuel S. Schoenholz, Jeffrey
> Pennington, Jascha Sohl-Dickstein. “Deep Neural Networks as Gaussian
> Processes.” ICLR 2018. https://arxiv.org/abs/1711.00165.
>
> Bio: Yasaman Bahri is a researcher at Google Brain working on deep
> learning. The goal of her research is to advance a scientific, principled
> understanding of deep learning, with an eye towards theoretical analysis
> informed by careful empirical work. She got a PhD in Theoretical Condensed
> Matter physics from UC Berkeley, specializing in many body physics; working
> on symmetry-protected topological phases, many-body localization, non-Fermi
> liquids, and topological mechanics. She is also interested in the
> connections between condensed matter, theoretical physics and machine
> learning.
>
>
> Host: Mesrob Ohannessian <mesrob at ttic.edu>
>
>
> Mary C. Marre
> Administrative Assistant
> *Toyota Technological Institute*
> *6045 S. Kenwood Avenue*
> *Room 504*
> *Chicago, IL  60637*
> *p:(773) 834-1757*
> *f: (773) 357-6970*
> *mmarre at ttic.edu <mmarre at ttic.edu>*
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cs.uchicago.edu/pipermail/colloquium/attachments/20180521/d1d0328b/attachment-0001.html>