[Colloquium] Madison Pickering MS Presentation/May 7, 2024

[Megan Woodward]

This is an announcement of Madison Pickering's MS Presentation
===============================================
Candidate: Madison Pickering

Date: Tuesday, May 07, 2024

Time: 12 pm CT

Location: JCL 390

Title: UNPACKING HOW HUMANS COMMUNICATE PROGRAMMING TASKS IN NATURAL LANGUAGE TO SUPPORT END-USER PROGRAMMING WITH LLMS

Abstract: Large language models (LLMs) like GPT-4 and Code Llama can convert natural-language descriptions into computer code, inspiring many people to see them as a promising interface for end-user programming. We undertake a systematic analysis of how people without programming experience describe information-processing tasks (IPTs) in natural language, focusing on the characteristics that make descriptions successful. In an online study, we paired 242 crowdworkers and asked one member of each pair (the sender) to communicate IPTs to the other (the receiver) in natural language. Participants answered test cases (producing outputs for given inputs) based on these descriptions. We also fed these same descriptions to six LLMs to see how they would answer the test cases. We found that participants both with and without prior programming experience successfully communicated IPTs, though descriptions written by senders with programming experience enabled both human receivers and LLMs to answer more test cases correctly. Descriptions containing example test cases led human receivers to answer 16.8% more test cases correctly, with similar gains by LLMs. Receivers who asked certain clarifying questions (e.g., about edge cases) answered slightly more test cases correctly. The best LLMs we tested narrowly outperformed humans in answering test cases.


Advisors: Blase Ur

Committee Members: Blase Ur, Ari Holtzman, and Mina Lee







-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cs.uchicago.edu/pipermail/colloquium/attachments/20240423/f3bea29a/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: MS_Formatted__UIST_paper.pdf
Type: application/pdf
Size: 4875594 bytes
Desc: MS_Formatted__UIST_paper.pdf
URL: <http://mailman.cs.uchicago.edu/pipermail/colloquium/attachments/20240423/f3bea29a/attachment-0001.pdf>