<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<div class="" style="word-wrap:break-word; line-break:after-white-space">
<div class="x_x_ContentPasted0" style="background-color:rgb(255,255,255); border:0px; font-stretch:inherit; line-height:inherit; margin:0px; padding:0px; vertical-align:baseline">
This is an announcement of Yuhan Liu's MS Presentation</div>
<div class="x_x_ContentPasted0" style="background-color:rgb(255,255,255); border:0px; font-stretch:inherit; line-height:inherit; margin:0px; padding:0px; vertical-align:baseline">
===============================================</div>
<div class="x_x_ContentPasted0" style="background-color:rgb(255,255,255); border:0px; font-stretch:inherit; line-height:inherit; margin:0px; padding:0px; vertical-align:baseline">
Candidate: Yuhan Liu</div>
<div class="x_x_ContentPasted0" style="background-color:rgb(255,255,255); border:0px; font-stretch:inherit; line-height:inherit; margin:0px; padding:0px; vertical-align:baseline">
<br class="">
</div>
<div class="x_x_ContentPasted0" style="background-color:rgb(255,255,255); border:0px; font-stretch:inherit; line-height:inherit; margin:0px; padding:0px; vertical-align:baseline">
Date: Wednesday, May 29, 2024</div>
<div class="x_x_ContentPasted0" style="background-color:rgb(255,255,255); border:0px; font-stretch:inherit; line-height:inherit; margin:0px; padding:0px; vertical-align:baseline">
<br class="">
</div>
<div class="x_x_ContentPasted0" style="background-color:rgb(255,255,255); border:0px; font-stretch:inherit; line-height:inherit; margin:0px; padding:0px; vertical-align:baseline">
Time: 2:30 pm CT</div>
<div class="x_x_ContentPasted0" style="background-color:rgb(255,255,255); border:0px; font-stretch:inherit; line-height:inherit; margin:0px; padding:0px; vertical-align:baseline">
<br class="">
</div>
<div class="x_x_ContentPasted0" style="background-color:rgb(255,255,255); border:0px; font-stretch:inherit; line-height:inherit; margin:0px; padding:0px; vertical-align:baseline">
Remote Location: <a href="https://urldefense.com/v3/__https://uchicago.zoom.us/j/6603596916?pwd=Z1E5MDRWUSt2am5XbEt4dTFkNGx6QT09__;!!BpyFHLRN4TMTrA!6bzqYITO8ttrkZnjL4DIVNh7nz4DzSLWnLSTMCEQw7iWSlbimsyugjjenT01KNIO6gwq0oW4cAyC3ltjrvz0m1LSl-24FUFhN30G8N0XSw$" class="">https://uchicago.zoom.us/j/6603596916?pwd=Z1E5MDRWUSt2am5XbEt4dTFkNGx6QT09</a></div>
<div class="x_x_ContentPasted0" style="background-color:rgb(255,255,255); border:0px; font-stretch:inherit; line-height:inherit; margin:0px; padding:0px; vertical-align:baseline">
<br class="">
</div>
<div class="x_x_ContentPasted0" style="background-color:rgb(255,255,255); border:0px; font-stretch:inherit; line-height:inherit; margin:0px; padding:0px; vertical-align:baseline">
Location: JCL 298</div>
<div class="x_x_ContentPasted0" style="background-color:rgb(255,255,255); border:0px; font-stretch:inherit; line-height:inherit; margin:0px; padding:0px; vertical-align:baseline">
<br class="">
</div>
<div class="x_x_ContentPasted0" style="background-color:rgb(255,255,255); border:0px; font-stretch:inherit; line-height:inherit; margin:0px; padding:0px; vertical-align:baseline">
Title: KV Cache Compression and Streaming for Fast Language Model Serving</div>
<div class="x_x_ContentPasted0" style="background-color:rgb(255,255,255); border:0px; font-stretch:inherit; line-height:inherit; margin:0px; padding:0px; vertical-align:baseline">
<br class="">
</div>
<div class="x_x_ContentPasted0" style="background-color:rgb(255,255,255); border:0px; font-stretch:inherit; line-height:inherit; margin:0px; padding:0px; vertical-align:baseline">
Abstract: As large language models (LLMs) take on complex tasks, their inputs are supplemented with longer contexts that incorporate domain knowledge or user-specific information. Yet using long contexts poses a challenge for responsive LLM systems, as nothing
can be generated until the whole context is processed by the LLM. While the context-processing delay can be reduced by reusing the KV cache of a context across different inputs, fetching the KV cache, which contains large tensors, over the network can cause
extra network delays.</div>
<div class="x_x_ContentPasted0" style="background-color:rgb(255,255,255); border:0px; font-stretch:inherit; line-height:inherit; margin:0px; padding:0px; vertical-align:baseline">
CacheGen is a fast context-loading module for LLM systems. First, CacheGen uses a custom tensor encoder, which embraces KV cache’s distributional properties, to encode a KV cache into more compact bitstream representations with negligible encoding/decoding
overhead. This reduces the bandwidth demand to fetch the KV cache. Second, to maintain low context-loading delay and high generation quality, CacheGen adapts the streaming strategies to cope with changes in available bandwidth. When available bandwidth drops,
CacheGen may raise the compression level for a part of the context or choose to recompute its KV cache on the fly. We test CacheGen on four popular LLMs of various sizes and four datasets (662 contexts in total). Compared to the recent systems that reuse the
KV cache, CacheGen reduces the KV cache size by 3.5-4.3x and the total delay in fetching and processing contexts by 3.2-3.7x while having negligible impact on the LLM response quality in accuracy or perplexity.</div>
<div class="x_x_ContentPasted0" style="background-color:rgb(255,255,255); border:0px; font-stretch:inherit; line-height:inherit; margin:0px; padding:0px; vertical-align:baseline">
<br class="">
</div>
<div class="x_x_ContentPasted0" style="background-color:rgb(255,255,255); border:0px; font-stretch:inherit; line-height:inherit; margin:0px; padding:0px; vertical-align:baseline">
Advisors: Junchen Jiang and Shan Lu</div>
<div class="x_x_ContentPasted0" style="background-color:rgb(255,255,255); border:0px; font-stretch:inherit; line-height:inherit; margin:0px; padding:0px; vertical-align:baseline">
<br class="">
</div>
<div class="x_x_ContentPasted0" style="background-color:rgb(255,255,255); border:0px; font-stretch:inherit; line-height:inherit; margin:0px; padding:0px; vertical-align:baseline">
Committee Members: Shan Lu, Junchen Jiang, and Arvind Krishnamurthy</div>
<div class="x_x_ContentPasted0" style="background-color:rgb(255,255,255); border:0px; font-stretch:inherit; line-height:inherit; margin:0px; padding:0px; vertical-align:baseline">
<br class="">
</div>
<div class="x_x_ContentPasted0" style="background-color:rgb(255,255,255); border:0px; font-stretch:inherit; line-height:inherit; margin:0px; padding:0px; vertical-align:baseline">
<br class="">
</div>
<div class="x_x_ContentPasted0" style="background-color:rgb(255,255,255); border:0px; font-stretch:inherit; line-height:inherit; margin:0px; padding:0px; vertical-align:baseline">
</div>
</div>
<div class="" style="word-wrap:break-word; line-break:after-white-space">
<div class="x_x_ContentPasted0" style="background-color:rgb(255,255,255); border:0px; font-stretch:inherit; line-height:inherit; margin:0px; padding:0px; vertical-align:baseline">
</div>
<div class="">
<div dir="auto" class="" style="color:rgb(0,0,0); letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<div dir="auto" class="" style="color:rgb(0,0,0); letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<div dir="auto" class="" style="color:rgb(0,0,0); letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<div dir="auto" class="" style="color:rgb(0,0,0); letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<div dir="auto" class="" style="color:rgb(0,0,0); letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<div dir="auto" class="" style="color:rgb(0,0,0); letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<div dir="auto" class="" style="color:rgb(0,0,0); letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
<br class="x_Apple-interchange-newline">
</div>
<br class="x_Apple-interchange-newline">
</div>
<br class="x_Apple-interchange-newline">
</div>
<br class="x_Apple-interchange-newline">
</div>
<br class="x_Apple-interchange-newline">
</div>
<br class="x_Apple-interchange-newline">
</div>
<br class="x_Apple-interchange-newline">
</div>
<br class="x_Apple-interchange-newline">
<br class="x_Apple-interchange-newline">
</div>
<br class="">
</div>
</body>
</html>