Running students' software tests against each others' code

At SIGCSE 2002, Michael Goldwasser suggested a strategy for adding software testing practices to programming courses by requiring students to turn in tests along with their solutions, and then running every student's tests against every other student's program. This approach provides a much more robust environment for assessing the quality of student-written tests, and also provides more thorough testing of student solutions. Although software testing is included as a regular part of many more programming courses today, the all-pairs model of executing tests is still a rarity. This is because student-written tests, such as JUnit tests written for Java programs, are now more commonly written in the form of program code themselves, and they may depend on virtually any aspect of their author's own solution. These dependencies may keep one student's tests from even compiling against another student's program. This paper discusses the problem and presents a novel solution for Java that uses bytecode rewriting to transform a student's tests into a form that uses reflection to run against any other solution, regardless of any compile-time dependencies that may have been present in the original tests. Results of applying this technique to two assignments, encompassing 147 student programs and 240,158 individual test case runs, shows the feasibility of the approach and provides some insight into the quality of both student tests and student programs. An analysis of these results is presented.