<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div class="elementToProof" style="text-align: left; text-indent: 0px; margin: 0px; font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
This is an announcement of Yuhan Liu's MS Presentation</div>
<div style="text-align: left; text-indent: 0px; margin: 0px; font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
===============================================</div>
<div style="text-align: left; text-indent: 0px; margin: 0px; font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Candidate: Yuhan Liu</div>
<div style="text-align: left; text-indent: 0px; margin: 0px; font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="text-align: left; text-indent: 0px; margin: 0px; font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Date: Wednesday, May 29, 2024</div>
<div style="text-align: left; text-indent: 0px; margin: 0px; font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="text-align: left; text-indent: 0px; margin: 0px; font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Time: 2:30 pm CT</div>
<div style="text-align: left; text-indent: 0px; margin: 0px; font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="text-align: left; text-indent: 0px; margin: 0px; font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Remote Location: <a href="https://urldefense.com/v3/__https://uchicago.zoom.us/j/6603596916?pwd=Z1E5MDRWUSt2am5XbEt4dTFkNGx6QT09__;!!BpyFHLRN4TMTrA!6bzqYITO8ttrkZnjL4DIVNh7nz4DzSLWnLSTMCEQw7iWSlbimsyugjjenT01KNIO6gwq0oW4cAyC3ltjrvz0m1LSl-24FUFhN30G8N0XSw$" id="OWA0bfb081b-4c24-62b5-d79a-a68cffa034d4" class="OWAAutoLink" data-loopstyle="linkonly">https://uchicago.zoom.us/j/6603596916?pwd=Z1E5MDRWUSt2am5XbEt4dTFkNGx6QT09</a></div>
<div style="text-align: left; text-indent: 0px; margin: 0px; font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="text-align: left; text-indent: 0px; margin: 0px; font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Location: JCL 298</div>
<div style="text-align: left; text-indent: 0px; margin: 0px; font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="text-align: left; text-indent: 0px; margin: 0px; font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Title: KV Cache Compression and Streaming for Fast Language Model Serving</div>
<div style="text-align: left; text-indent: 0px; margin: 0px; font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="text-align: left; text-indent: 0px; margin: 0px; font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Abstract: As large language models (LLMs) take on complex tasks, their inputs are supplemented with longer contexts that incorporate domain knowledge or user-specific information. Yet using long contexts poses a challenge for responsive LLM systems, as nothing
can be generated until the whole context is processed by the LLM. While the context-processing delay can be reduced by reusing the KV cache of a context across different inputs, fetching the KV cache, which contains large tensors, over the network can cause
extra network delays.</div>
<div style="text-align: left; text-indent: 0px; margin: 0px; font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
CacheGen is a fast context-loading module for LLM systems. First, CacheGen uses a custom tensor encoder, which embraces KV cache’s distributional properties, to encode a KV cache into more compact bitstream representations with negligible encoding/decoding
overhead. This reduces the bandwidth demand to fetch the KV cache. Second, to maintain low context-loading delay and high generation quality, CacheGen adapts the streaming strategies to cope with changes in available bandwidth. When available bandwidth drops,
CacheGen may raise the compression level for a part of the context or choose to recompute its KV cache on the fly. We test CacheGen on four popular LLMs of various sizes and four datasets (662 contexts in total). Compared to the recent systems that reuse the
KV cache, CacheGen reduces the KV cache size by 3.5-4.3x and the total delay in fetching and processing contexts by 3.2-3.7x while having negligible impact on the LLM response quality in accuracy or perplexity.</div>
<div style="text-align: left; text-indent: 0px; margin: 0px; font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="text-align: left; text-indent: 0px; margin: 0px; font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Advisors: Junchen Jiang and Shan Lu</div>
<div style="text-align: left; text-indent: 0px; margin: 0px; font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="text-align: left; text-indent: 0px; margin: 0px; font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Committee Members: Shan Lu, Junchen Jiang, and Arvind Krishnamurthy</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
<br>
</div>
<div id="Signature">
<div style="background-color: rgb(255, 255, 255);"></div>
</div>
</body>
</html>