<div dir="ltr"><div dir="ltr"><div class="gmail_default" style="font-size:small"><div style="color:rgb(80,0,80)"><b>When</b>: <span style="color:rgb(34,34,34)">Wednesday, August 23, at 3 PM CST</span></div><div style="color:rgb(80,0,80)"><span style="color:rgb(34,34,34)"><br></span></div><div style="color:rgb(80,0,80)"><b>Virtually</b>: Talk will be held via Zoom <b><font color="#0000ff"><a href="https://uchicago.zoom.us/j/99792875029?pwd=aHIwZERjTHhTeHQwTXhkOU92T2o2UT09" target="_blank">here</a></font></b><br></div><div style="color:rgb(80,0,80)"><br><b>Who</b>: <span style="color:rgb(34,34,34)">Davis Yoshida</span>, TTIC</div><div style="color:rgb(80,0,80)"><br></div><div style="color:rgb(80,0,80)"><div class="MsoNormal" align="center" style="margin:0in 0in 8pt;text-align:center;line-height:15.6933px;font-size:11pt;font-family:Calibri,sans-serif"><hr size="2" width="100%" align="center"></div></div><div style="color:rgb(80,0,80)"><div><b>Title:</b> Making the Most of your Model: Methods for Finetuning and Applying Pretrained Transformers</div><div><br style="color:rgb(34,34,34)"></div><div><b>Abstract:</b> <span style="color:rgb(34,34,34)">Recent progress in NLP has been dominated by large pretrained models. While the rate of improvement has been astounding, we are far from knowing how to optimally use even the models we already have. Improvements to our knowledge about how to make use of pretrained models have a multiplicative benefit, because they increase the utility of all extant pretrained models, even those trained for niche domains which are not well covered by even state of the art large language models (LLMs). </span></div><div style="color:rgb(34,34,34)"><br>This thesis provides methods and analysis of models which make progress on this goal. The techniques outlined are task agnostic, and should provide benefit when used with nearly any transformer LM. We introduce two new finetuning methods which add new capabilities to the models they are used on. The first adds a recurrence mechanism, which removes the fixed-window sized constraint and improves the efficiency of a transformer decoder. The second allows MLMs to be used for initialization of both the encoder and decoder of a non-autoregressive sequence-to-sequence transformer, opening up generative applications of models which were previously only used for natural language understanding tasks.<br><br>We also introduce two new techniques for improving the quality of predictions of any transformer decoder without additional finetuning. One, hidden state optimization, can be applied to any transformer decoder to improve the quality of predictions at inference time, especially for few-shot classification. The other, conditional beam search, allows practitioners to search for NLG model outputs with high likelihood while conditioning on the event that the output is not degenerate (e.g. empty, repetitive, etc.).<br><br>Finally, we provide theoretical and empirical insights on the divergence of model-likelihood and output quality which has widely been observed in prior work. These insights apply to any model which represents a distribution over text, and apply to language models which are not transformers or even autoregressive. We argue that the NLP community has, to some extent, misunderstood the implications of these findings, and encourage a point of view which has more nuance. <br><br></div><div style="color:rgb(34,34,34)">Taken together, the findings in this thesis should allow NLP practitioners to make much more effective use of pretrained models, either those that already exist or ones that will be created in the future.<br></div><div style="color:rgb(34,34,34)"><br></div><div style="color:rgb(34,34,34)"><b>Advisor</b>: <b><a href="mailto:kgimpel@ttic.edu" target="_blank">Kevin Gimpel</a> </b></div><div style="color:rgb(34,34,34)"><br></div><div style="color:rgb(34,34,34)"><br></div></div></div><div><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><span style="font-family:arial,helvetica,sans-serif;font-size:x-small">Mary C. Marre</span><br></div><div><div><font face="arial, helvetica, sans-serif" size="1">Faculty Administrative Support</font></div><div><i><font face="arial, helvetica, sans-serif" color="#3d85c6" size="1"><b>Toyota Technological Institute</b></font></i></div><div><i><font face="arial, helvetica, sans-serif" color="#3d85c6" size="1">6045 S. Kenwood Avenue, Rm 517</font></i></div><div><font size="1"><i><font face="arial, helvetica, sans-serif" color="#3d85c6">Chicago, IL 60637</font></i><br></font></div><div><font size="1"><i><font face="arial, helvetica, sans-serif" color="#3d85c6">773-834-1757</font></i></font></div><div><b><i><a href="mailto:mmarre@ttic.edu" target="_blank"><font face="arial, helvetica, sans-serif" size="1">mmarre@ttic.edu</font></a></i></b></div></div></div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Aug 22, 2023 at 4:32 PM Mary Marre <<a href="mailto:mmarre@ttic.edu">mmarre@ttic.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div style="font-size:small"><div style="color:rgb(80,0,80)"><b>When</b>: <span style="color:rgb(34,34,34)">Wednesday, August 23, at 3 PM CST</span></div><div style="color:rgb(80,0,80)"><span style="color:rgb(34,34,34)"><br></span></div><div style="color:rgb(80,0,80)"><b>Virtually</b>: Talk will be held via Zoom <b><font color="#0000ff"><a href="https://uchicago.zoom.us/j/99792875029?pwd=aHIwZERjTHhTeHQwTXhkOU92T2o2UT09" target="_blank">here</a></font></b><br></div><div style="color:rgb(80,0,80)"><br><b>Who</b>: <span style="color:rgb(34,34,34)">Davis Yoshida</span>, TTIC</div><div style="color:rgb(80,0,80)"><br></div><div style="color:rgb(80,0,80)"><div class="MsoNormal" align="center" style="margin:0in 0in 8pt;text-align:center;line-height:15.6933px;font-size:11pt;font-family:Calibri,sans-serif"><hr size="2" width="100%" align="center"></div></div><div style="color:rgb(80,0,80)"><div><b>Title:</b> Making the Most of your Model: Methods for Finetuning and Applying Pretrained Transformers</div><div><br style="color:rgb(34,34,34)"></div><div><b>Abstract:</b> <span style="color:rgb(34,34,34)">Recent progress in NLP has been dominated by large pretrained models. While the rate of improvement has been astounding, we are far from knowing how to optimally use even the models we already have. Improvements to our knowledge about how to make use of pretrained models have a multiplicative benefit, because they increase the utility of all extant pretrained models, even those trained for niche domains which are not well covered by even state of the art large language models (LLMs). </span></div><div style="color:rgb(34,34,34)"><br>This thesis provides methods and analysis of models which make progress on this goal. The techniques outlined are task agnostic, and should provide benefit when used with nearly any transformer LM. We introduce two new finetuning methods which add new capabilities to the models they are used on. The first adds a recurrence mechanism, which removes the fixed-window sized constraint and improves the efficiency of a transformer decoder. The second allows MLMs to be used for initialization of both the encoder and decoder of a non-autoregressive sequence-to-sequence transformer, opening up generative applications of models which were previously only used for natural language understanding tasks.<br><br>We also introduce two new techniques for improving the quality of predictions of any transformer decoder without additional finetuning. One, hidden state optimization, can be applied to any transformer decoder to improve the quality of predictions at inference time, especially for few-shot classification. The other, conditional beam search, allows practitioners to search for NLG model outputs with high likelihood while conditioning on the event that the output is not degenerate (e.g. empty, repetitive, etc.).<br><br>Finally, we provide theoretical and empirical insights on the divergence of model-likelihood and output quality which has widely been observed in prior work. These insights apply to any model which represents a distribution over text, and apply to language models which are not transformers or even autoregressive. We argue that the NLP community has, to some extent, misunderstood the implications of these findings, and encourage a point of view which has more nuance. <br><br></div><div style="color:rgb(34,34,34)">Taken together, the findings in this thesis should allow NLP practitioners to make much more effective use of pretrained models, either those that already exist or ones that will be created in the future.<br></div><div style="color:rgb(34,34,34)"><br></div><div style="color:rgb(34,34,34)"><b>Advisor</b>: <b><a href="mailto:kgimpel@ttic.edu" target="_blank">Kevin Gimpel</a> </b></div><div style="color:rgb(34,34,34)"><br></div><div style="color:rgb(34,34,34)"><br></div><div style="color:rgb(34,34,34)"><br></div><div style="color:rgb(34,34,34)"><br></div><div style="color:rgb(34,34,34)"><br></div></div></div><div><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><span style="font-family:arial,helvetica,sans-serif;font-size:x-small">Mary C. Marre</span><br></div><div><div><font face="arial, helvetica, sans-serif" size="1">Faculty Administrative Support</font></div><div><i><font face="arial, helvetica, sans-serif" color="#3d85c6" size="1"><b>Toyota Technological Institute</b></font></i></div><div><i><font face="arial, helvetica, sans-serif" color="#3d85c6" size="1">6045 S. Kenwood Avenue, Rm 517</font></i></div><div><font size="1"><i><font face="arial, helvetica, sans-serif" color="#3d85c6">Chicago, IL 60637</font></i><br></font></div><div><font size="1"><i><font face="arial, helvetica, sans-serif" color="#3d85c6">773-834-1757</font></i></font></div><div><b><i><a href="mailto:mmarre@ttic.edu" target="_blank"><font face="arial, helvetica, sans-serif" size="1">mmarre@ttic.edu</font></a></i></b></div></div></div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Aug 19, 2023 at 3:21 PM Mary Marre <<a href="mailto:mmarre@ttic.edu" target="_blank">mmarre@ttic.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="font-size:small"><div style="color:rgb(80,0,80)"><b>When</b>: <span style="color:rgb(34,34,34)">Wednesday, August 23, at 3 PM CST</span></div><div style="color:rgb(80,0,80)"><span style="color:rgb(34,34,34)"><br></span></div><div style="color:rgb(80,0,80)"><b>Virtually</b>: Talk will be held via Zoom <b><font color="#0000ff"><a href="https://uchicago.zoom.us/j/99792875029?pwd=aHIwZERjTHhTeHQwTXhkOU92T2o2UT09" target="_blank">here</a></font></b><br></div><div style="color:rgb(80,0,80)"><br><b>Who</b>: <span style="color:rgb(34,34,34)">Davis Yoshida</span>, TTIC</div><div style="color:rgb(80,0,80)"><br></div><div style="color:rgb(80,0,80)"><div class="MsoNormal" align="center" style="margin:0in 0in 8pt;text-align:center;line-height:15.6933px;font-size:11pt;font-family:Calibri,sans-serif"><hr size="2" width="100%" align="center"></div></div><div style="color:rgb(80,0,80)"><div><b>Title:</b> Making the Most of your Model: Methods for Finetuning and Applying Pretrained Transformers</div><div><br style="color:rgb(34,34,34)"></div><div><b>Abstract:</b> <span style="color:rgb(34,34,34)">Recent progress in NLP has been dominated by large pretrained models. While the rate of improvement has been astounding, we are far from knowing how to optimally use even the models we already have. Improvements to our knowledge about how to make use of pretrained models have a multiplicative benefit, because they increase the utility of all extant pretrained models, even those trained for niche domains which are not well covered by even state of the art large language models (LLMs). </span></div><div style="color:rgb(34,34,34)"><br>This thesis provides methods and analysis of models which make progress on this goal. The techniques outlined are task agnostic, and should provide benefit when used with nearly any transformer LM. We introduce two new finetuning methods which add new capabilities to the models they are used on. The first adds a recurrence mechanism, which removes the fixed-window sized constraint and improves the efficiency of a transformer decoder. The second allows MLMs to be used for initialization of both the encoder and decoder of a non-autoregressive sequence-to-sequence transformer, opening up generative applications of models which were previously only used for natural language understanding tasks.<br><br>We also introduce two new techniques for improving the quality of predictions of any transformer decoder without additional finetuning. One, hidden state optimization, can be applied to any transformer decoder to improve the quality of predictions at inference time, especially for few-shot classification. The other, conditional beam search, allows practitioners to search for NLG model outputs with high likelihood while conditioning on the event that the output is not degenerate (e.g. empty, repetitive, etc.).<br><br>Finally, we provide theoretical and empirical insights on the divergence of model-likelihood and output quality which has widely been observed in prior work. These insights apply to any model which represents a distribution over text, and apply to language models which are not transformers or even autoregressive. We argue that the NLP community has, to some extent, misunderstood the implications of these findings, and encourage a point of view which has more nuance. <br><br></div><div style="color:rgb(34,34,34)">Taken together, the findings in this thesis should allow NLP practitioners to make much more effective use of pretrained models, either those that already exist or ones that will be created in the future.<br></div><div style="color:rgb(34,34,34)"><br></div><div style="color:rgb(34,34,34)"><b>Advisor</b>: <a href="mailto:kgimpel@ttic.edu" target="_blank">Kevin Gimpel</a> </div><div style="color:rgb(34,34,34)"><br></div><div style="color:rgb(34,34,34)"><br></div><div style="color:rgb(34,34,34)"><br></div></div></div><div><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><span style="font-family:arial,helvetica,sans-serif;font-size:x-small">Mary C. Marre</span><br></div><div><div><font face="arial, helvetica, sans-serif" size="1">Faculty Administrative Support</font></div><div><i><font face="arial, helvetica, sans-serif" color="#3d85c6" size="1"><b>Toyota Technological Institute</b></font></i></div><div><i><font face="arial, helvetica, sans-serif" color="#3d85c6" size="1">6045 S. Kenwood Avenue, Rm 517</font></i></div><div><font size="1"><i><font face="arial, helvetica, sans-serif" color="#3d85c6">Chicago, IL 60637</font></i><br></font></div><div><font size="1"><i><font face="arial, helvetica, sans-serif" color="#3d85c6">773-834-1757</font></i></font></div><div><b><i><a href="mailto:mmarre@ttic.edu" target="_blank"><font face="arial, helvetica, sans-serif" size="1">mmarre@ttic.edu</font></a></i></b></div></div></div></div></div></div>
</blockquote></div></div>
</blockquote></div></div>