Benchmarking autonomic capabilities: promises and pitfalls

Benchmarks provide a way to quantify progress in a field. Our goal is to produce a suite of benchmarks covering the four categories of autonomic capabilities: self-configuring, self-healing, self-optimizing, and self-protecting (IBM, 2003). This is not an easy task, however, and in this paper we identify several of the challenges and pitfalls that must be confronted to extend benchmarking technology beyond its traditional basis in performance evaluation.