[CS] Yibo Jiang Dissertation Defense May 21, 2025

[via cs]

This is an announcement of Yibo Jiang's Dissertation Defense.
===============================================
Candidate: Yibo Jiang

Date: Wednesday, May 21, 2025

Time:  1 pm CST

Remote Location: https://uchicago.zoom.us/j/98115267085?pwd=PspaiWHB3ZFIxTLvbjLnGkdZBO5mVH.1  Meeting ID: 981 1526 7085 Passcode: 107782

Location: JCL 298

Title: Geometric and Algebraic Structures in Foundation Model Representations

Abstract: Foundation models, such as large language models (LLMs), operate within vector spaces, whereas human perception of concepts does not naturally align with this framework. This raises a fundamental question: how do these models internalize the structure of concepts within a vector space and how do they use it? To address this, the thesis investigates two structural properties—partial orthogonality and linearity—and also studies how information in representations can be effectively leveraged by the self-attention architecture. The first part examines how models represent the intuitive notion of "semantic independence." Rather than formally defining semantic independence, the focus is on the algebraic axioms of independence and how they can be represented in the forms of partial orthogonality in the embedding space. The second part investigates linear representations. While the concept of linearity appears straightforward, its underlying basis—especially in LLMs trained solely on next-token prediction—remains a largely unresolved mystery. This thesis provides new insights into this phenomenon by showing the connection between linear representations and the implicit bias of gradient descent. Finally, the third part examines representations in a practical setting—fact retrieval—and explores how self-attention can effectively combine stored information in representations to retrieve the most relevant outputs, functioning like associative memory.

Advisors: Victor Veitch

Commitee: Victor Veitch, Bryon Aragam, Ari Holtzman, Yuxin Chen