CMKD: CNN/Transformer-Based Cross-Model Knowledge Distillation for Audio\n Classification

Audio classification is an active research area with a wide range of\napplications. Over the past decade, convolutional neural networks (CNNs) have\nbeen the de-facto standard building block for end-to-end audio classification\nmodels. Recently, neural networks based solely on self-attention mechanisms\nsuch as the Audio Spectrogram Transformer (AST) have been shown to outperform\nCNNs. In this paper, we find an intriguing interaction between the two very\ndifferent models - CNN and AST models are good teachers for each other. When we\nuse either of them as the teacher and train the other model as the student via\nknowledge distillation (KD), the performance of the student model noticeably\nimproves, and in many cases, is better than the teacher model. In our\nexperiments with this CNN/Transformer Cross-Model Knowledge Distillation (CMKD)\nmethod we achieve new state-of-the-art performance on FSD50K, AudioSet, and\nESC-50.\n