Search for a command to run...
Infinite looping problems that keep student programs from termi-nating may occur in many kinds of programming assignments. While non-terminating code is easier to diagnose interactively, it poses different concerns when software tests are being run auto-matically in batch. The common strategy of using a timeout to preemptively kill test runs that execute for too long has limita-tions, however. When one test case gets stuck in an infinite loop, forcible termination prevents any later test cases from running. Worse, when test results are buffered inside a test execution framework, forcible termination may prevent any information from being produced. Further, overly generous timeouts can de-lay the availability of results, and when tests are executed on a shared server, one non-terminating program can delay results for many people. This paper describes an alternative strategy that uses a fine-grained timeout on the execution of each individual test case in a test suite, and that adaptively adjusts this timeout dynamically based on the termination behavior of test cases com-pleted so far. By avoiding forcible termination of test runs, this approach allows all test cases an opportunity to run and produce results, even when infinite looping behaviors occur. Such fine-grained timeouts also result in faster completion of entire test runs when non-terminating code is present. Experimental results from applying this strategy to 4,214 student-written programs are dis-cussed, along with experiences from live deployment in the class-room where 8,926 non-termination events were detected over the course of one academic year.