[Theory] [Talks at TTIC] 4/9 Young Researcher Seminar Series: Daniel Kunin, Stanford University

[Brandie Jones via Theory]

*When:    *Wednesday, April 9th* at **11AM CT*

*Where:   *Talk will be given *live, in-person* at

                    TTIC, 6045 S. Kenwood Avenue

                    5th Floor, Room 530


*Virtually: *via Panopto (Livestream
<https://uchicago.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=4803398c-95cb-4dc2-9632-b26e0116ffba>
)

*Who:      *Daniel Kunin, Stanford University

*Title:*       Learning Mechanics of Neural Networks: Conservation Laws,
Implicit Biases, and Feature Learning

*Abstract: *The success of neural networks is often attributed to their
ability to extract task-relevant features from data through training, yet a
precise understanding of this process remains elusive. In this talk, I will
explore the learning dynamics of neural networks, focusing on when and how
they learn features. First, I will review how the parameter initialization
scale influences learning -- large scales result in a "kernel regime" that
stays close to its initialization, while small scales lead to an "active
regime" traversing between saddle points. In the second part of the talk, I
will present ongoing work describing the saddle-to-saddle dynamics for
two-layer neural networks with vanishing initializations. We conjecture
that training follows a recursive optimization process, alternating between
maximizing a utility function over "dormant neurons" and minimizing a cost
function over "active neurons". We demonstrate how this framework unifies
existing theories of feature learning, such as those for diagonal linear
networks and matrix factorization, and extends to new settings, such as
quadratic networks for modular addition.

*Bio:* Daniel is currently a PhD candidate at the Institute for
Computational and Mathematical Engineering at Stanford University, advised
by Surya Ganguli. Daniel will join UC Berkeley as a Miller Fellow in the
neuroscience and statistics departments this fall. His research focuses on
understanding the learning dynamics of neural networks, particularly how
inductive biases emerge during training and how networks extract meaningful
representations from data.

*Host: Nati Srebro <nati at ttic.edu>*


-- 
*Brandie Jones *
*Executive **Administrative Assistant*
Toyota Technological Institute
6045 S. Kenwood Avenue
Chicago, IL  60637
www.ttic.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cs.uchicago.edu/pipermail/theory/attachments/20250402/c31f753d/attachment.html>