Comparative studies of the model evaluation criterions mmre and pred in software cost estimation research

Software cost model research results depend on model accuracy criteria such as MMRE and PRED. Despite criticism, MMRE has emerged as the de facto standard criterion. Many alternatives have been proposed and studied, surprisingly however PRED, the second most popular criterion, has not been extensively studied. This work attempts to fill this gap in the literature and expand the understanding and use of evaluation criterion in general. The majority of this work is empirically based, applying MMRE and PRED to a number of COCOMO model variations with respect to a simulated data set and four publicly available cost estimation data sets. We replicate a number of results based on MMRE and extend them to PRED. We study qualities of MMRE and PRED as sample estimator statistics for parameters of a cost model error distribution. Standard error is used to ensure greater confidence in replicated and new results based on sample data.