<div dir="ltr"><div dir="ltr"><div><div class="gmail_default" style="font-family:georgia,serif;font-size:small;color:rgb(0,0,0)"><font style="vertical-align:inherit"><font style="vertical-align:inherit"><b>When:</b> </font></font><font style="vertical-align:inherit"><font style="vertical-align:inherit"> Thursday, May 15th at <b style="background-color:rgb(255,255,0)">11am CT</b><b> </b></font></font></div><div class="gmail_default"><font face="georgia, serif"><font style="vertical-align:inherit"><font color="#000000" style="vertical-align:inherit"><b><br></b></font></font></font></div><div class="gmail_default"><font face="georgia, serif" color="#000000"></font></div><div class="gmail_default"><font face="georgia, serif" color="#000000"><b>Where: </b>Talk will be given <font style="font-weight:bold"><u>live, in-person</u></font><font style="font-weight:bold"> </font>at</font></div><p class="MsoNormal" style="margin:0in;line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="georgia, serif" color="#000000"> TTIC, 6045 S. Kenwood Avenue</font></p><p class="MsoNormal" style="margin:0in;line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="georgia, serif" color="#000000"> 5th Floor, Room 5<span class="gmail_default" style="font-family:georgia,serif;font-size:small;color:rgb(0,0,0)">29</span></font></p><p class="MsoNormal" style="margin:0in;line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><b><font face="georgia, serif" color="#000000"><br></font></b></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="georgia, serif" color="#000000"><b style="letter-spacing:0.2px">Virtually:</b><span style="letter-spacing:0.2px"> via Panopto </span>(<a href="https://uchicago.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=dc3e2b16-3fae-4961-8dc4-b2d800fb6856" target="_blank">livestream</a><span style="letter-spacing:0.2px"><span class="gmail_default"></span>)</span><br></font></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font face="georgia, serif" color="#000000"><br></font></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><font color="#000000"><font face="georgia, serif"><font style="vertical-align:inherit"><font style="vertical-align:inherit"><b>Who: </b> </font></font><span class="gmail_default"></span><span class="gmail_default"></span><span class="gmail_default"></span><span class="gmail_default"></span></font><font face="georgia, serif">Simon Du, University of Washington</font></font></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;line-height:normal;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"></p><div><p style="letter-spacing:0.2px"><font face="georgia, serif" color="#000000"><span style="letter-spacing:normal"><b>Title:</b> </span><span style="letter-spacing:normal"><span class="gmail_default"></span></span><span style="letter-spacing:normal">Reinforcement Learning for Reasoning in Large Language Models with One Training Example</span></font></p><div><font face="georgia, serif" color="#000000"><b>Abstract: </b><span class="gmail_default"></span>We show that reinforcement learning with verifiable reward using one training example (1-shot RLVR) is effective in incentivizing the math reasoning capabilities of large language models (LLMs). Applying RLVR to the base model Qwen2.5-Math-1.5B, we identify a single example that elevates model performance on MATH500 from 36.0% to 73.6%, and improves the average performance across six common mathematical reasoning benchmarks from 17.6% to 35.7%. This result matches the performance obtained using the 1.2k DeepScaleR subset (MATH500: 73.6%, average: 35.9%), which includes the aforementioned example. Similar substantial improvements are observed across various models (Qwen2.5-Math-7B, Llama3.2-3B-Instruct, DeepSeek-R1-Distill-Qwen-1.5B), RL algorithms (GRPO and PPO), and different math examples (many of which yield approximately 30% or greater improvement on MATH500 when employed as a single training example). </font></div><div><font face="georgia, serif" color="#000000"><br></font></div><div><font face="georgia, serif" color="#000000">In addition, we identify some interesting phenomena during 1-shot RLVR, including cross-domain generalization, increased frequency of self-reflection, and sustained test performance improvement even after the training accuracy has saturated, a phenomenon we term post-saturation generalization. Moreover, we verify that the effectiveness of 1-shot RLVR primarily arises from the policy gradient loss, distinguishing it from the "grokking" phenomenon. We also show the critical role of promoting exploration (e.g., by adding entropy loss with an appropriate coefficient) in 1-shot RLVR training. As a bonus, we observe that applying entropy loss alone, without any outcome reward, significantly enhances Qwen2.5-Math-1.5B's performance on MATH500 by 27.4%. These findings can inspire future work on RLVR data efficiency and encourage a re-examination of both recent progress and the underlying mechanisms in RLVR.</font></div></div><div><div><font face="georgia, serif" color="#000000"><br></font></div></div><div><font face="georgia, serif" color="#000000"><br><b>Host:<span class="gmail_default"> <a href="mailto:zhiyuanli@ttic.edu" target="_blank">Zhiyuan Li</a></span></b></font></div><br clear="all"></div><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><b style="background-color:rgb(255,255,255)"><font color="#3d85c6">Brandie Jones </font></b><div><div><div><font color="#3d85c6"><b><i>Executive </i></b></font><b style="color:rgb(61,133,198)"><i>Administrative Assistant</i></b></div></div><div><span style="background-color:rgb(255,255,255)"><font color="#3d85c6">Toyota Technological Institute</font></span></div><div><span style="background-color:rgb(255,255,255)"><font color="#3d85c6">6045 S. Kenwood Avenue</font></span></div><div><span style="background-color:rgb(255,255,255)"><font color="#3d85c6">Chicago, IL 60637</font></span></div></div><div><span style="background-color:rgb(255,255,255)"><font color="#3d85c6"><a href="http://www.ttic.edu" target="_blank">www.ttic.edu</a> </font></span></div><div><span style="background-color:rgb(255,255,255)"><font color="#3d85c6"><br></font></span></div></div></div></div>
</div>