A comprehensive, application-oriented study of catastrophic forgetting\n in DNNs

We present a large-scale empirical study of catastrophic forgetting (CF) in\nmodern Deep Neural Network (DNN) models that perform sequential (or:\nincremental) learning. A new experimental protocol is proposed that enforces\ntypical constraints encountered in application scenarios. As the investigation\nis empirical, we evaluate CF behavior on the hitherto largest number of visual\nclassification datasets, from each of which we construct a representative\nnumber of Sequential Learning Tasks (SLTs) in close alignment to previous works\non CF. Our results clearly indicate that there is no model that avoids CF for\nall investigated datasets and SLTs under application conditions. We conclude\nwith a discussion of potential solutions and workarounds to CF, notably for the\nEWC and IMM models.\n