Benchmarking Large Language Models for Automated Code Changes: A Systematic Review

Large Language Models (LLMs) have been employed more and more to automate code changes (ACC) for a range of development projects, such as enhancements in features, refining and error remediation. Recently, a number of LLM based automate code change paradigms have been created and reviewed. Although LLMs in ACC tasks have been studied in past research, this domain still needs to evaluate uniform procedures, a standardized dataset and reducibility challenges. Therefore, this study provides a systematic analysis of existing literature on LLM based ACC tasks to examine the core data sets, frequently employed assessment metrics and persistent reducibility issues. Findings of this review study revealed that the most frequently used dataset to train an LLM model for code change is Defects4J. Similarly, repair success rate, pass@k and patch accuracy were the main evaluation measured utilized in the articles to evaluate the performance of the developed model. Moreover, this study also examined various reproducibility challenges that can be faced when recreating a similar ACC model. These results open the door to more investigation into how to fully utilized LLMs to transform automated code changes.