[CS] Zixuan Zhao MS Presentation/Dec.2nd

[via cs]

This is an announcement of Zixuan Zhao's MS Presentation
===============================================
Candidate: Zixuan Zhao

Date: Monday, December 2nd

Time: 9AM -10AM CST

Location: JCL 298 

Remote Location:  https://uchicago.zoom.us/j/94031362675?pwd=byDRZuWhe1ifTIltIuLjlJBSPvQTJ2.1

Meeting ID: 940 3136 2675
Passcode: 071540

Title: Exploring Non-Contrastive Self-Supervised Learning

Abstract: Self-supervised learning (SSL) is a machine learning paradigm where models learn meaningful representations from unlabeled data by solving algorithmically-generated prediction tasks using inherent structure within the data itself. This method has been crucial in training large language models (LLMs) through next-token prediction and powering image models like DINO. Most vision SSL algorithms generate two augmented views of an image, learning representations by pulling these views closer while pushing apart those from different images. Non-contrastive SSL (nc-SSL) algorithms avoid comparing different image pairs, which risks a collapse where all images degenerates to the same representation. Many nc-SSL methods address this using a predictor and stop-gradient operation, though the reasons for their effectiveness are not fully understood. We developed a codebase to replicate various SSL algorithms and studied their properties. Focusing on DirectCopy, which simplifies the predictor with a non-trainable, closed-form solution, we found a counterintuitive result: an increasing training loss often correlated with better representation learning in 92% of high-performing cases. We also explored biologically plausible adaptations of DirectCopy, showing their performance is comparable to contrastive methods. Finally, we propose FreeCopy, a DirectCopy variation that uses cross-correlation to make the two branches free of data-vectors, bridging nc-SSL methods like BYOL and regularization-based approaches such as Barlow Twins. Our results offer new insights into nc-SSL mechanisms and suggest paths for further study.


Advisors: Rick Stevens

Committee Members: Rick Stevens, Yali Amit, Fangfang Xia