Evaluating LLM-Generated Contextualized Algorithm Design Problems
\textbf{Background:} Context personalization, the practice of adapting learning materials to students’ personal interests, has been shown to increase student learning and engagement. Within computer science education, research has found that LLMs can successfully generate contextualized introductory programming exercises. \textbf{Objective:} In this paper, we evaluate the capability of LLMs to generate technically correct and thematically integrated contextualized algorithm design problems. \textbf{Methods:} In a series of three iterative studies, we use LLMs to generate contextualized algorithm design problems from a given base problem and theme, evaluating over 500 generated problems for technical and thematic alignment. \textbf{Results:} We find that LLMs encounter significantly more challenges writing algorithm design problems than prior work has found with introductory programming problems. We identify issues specific to the algorithm design context and then mitigate these issues with prompt engineering techniques and model choice. With these adjustments, we produce LLM-generated contextualized algorithm design problems that are technically strong, deeply themed, and largely realistic, though realism drops with more culturally and locally specific themes. \textbf{Implications:} Our results indicate that with intentional prompting, advanced LLMs can be a useful tool for generating contextualized problems for algorithms courses. While the generated problems were largely directly usable, the importance of ensuring technical correctness and thematic authenticity warrants instructor review before presenting LLM-generated problems to students.
Fri 14 AugDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
13:45 - 14:35 | GenAI beyond CS1Research Papers at Main conference room Chair(s): James Prather Abilene Christian University | ||
13:45 25mTalk | Characterization and Effects of CS2 Learning with GenAI, Visualization, and Human Support Research Papers Quinton Yong University of Victoria, Miguel A. Nacenta University of Victoria, Anthony Estey University of British Columbia | ||
14:10 25mTalk | Evaluating LLM-Generated Contextualized Algorithm Design Problems Research Papers Erica Goodwin University of Chicago, Katherine Braught University of Illinois at Urbana-Champaign, Jonathan Liu University of Chicago, Dip Kiran Pradhan Newar pc, Yael Gertner University of Illinois Urbana-Champaign, Seth Poulsen Utah State University, Diana Franklin University of Chicago | ||