[CS] Yibo Jiang Candidacy Exam

[via cs]

This is an announcement of Yibo Jiang's Candidacy Exam.
===============================================
Candidate: Yibo Jiang

Date: January 6, 2025

Time: 2:00 p.m.

Location: JCL 354

Title: Geometric and Algebraic Structures in Foundation Model Representations

Abstract:  Foundation models, such as large language models (LLMs), operate within vector spaces, whereas human perception of concepts does not naturally align with this framework. This raises a fundamental question: how do these models internalize the structure of concepts within a vector space? To address this, this thesis explores two specific structures: partial orthogonality and linearity. The first part examines how models represent the intuitive notion of "semantic independence." Rather than formally defining semantic independence, the focus is on the algebraic axioms of independence and how they can be represented in the forms of partial orthogonality in the embedding space. The second part investigates linear representations. While the concept of linearity appears straightforward, its underlying basis—especially in LLMs trained solely on next-token prediction—remains a largely unresolved mystery. This thesis provides new insights into this phenomenon by showing the connection between linear representations and the implicit bias of gradient descent.

Advisor: Victor Vietch

Advisors: Yuxin Chen and Victor Veitch

Committee Members: Yuxin Chen and Victor Veitch, Bryon Aragam, Ari Holtzman