«上一篇/Previous Article|本期目录/Table of Contents|下一篇/Next Article»

《计算机技术与发展》[ISSN:1006-6977/CN:61-1281/TN]

卷:

34

期数:

2024年12期

页码:

125-131

栏目:

人工智能

出版日期:

2024-12-10

文章信息/Info

Title:

Class Attention Knowledge Distillation Based on Channel Correlation

文章编号:

1673-629X(2024)12-0125-07

作者:

吴华涛; 朱子奇

武汉科技大学 计算机科学与技术学院,湖北 武汉 430065

Author(s):

WU Hua-tao; ZHU Zi-qi

School of Computer Science & Technology,Wuhan University of Science and Technology,Wuhan 430065,China

关键词:

知识蒸馏; 通道相关性; 类激活图; 通道知识; 注意力蒸馏

Keywords:

knowledge distillation; channel correlation; class activation maps; channel knowledge; attention distillation

分类号:

TP391.4

DOI:

10.20165/j.cnki.ISSN1673-629X.2024.0242

摘要:

以前的知识蒸馏方法在模型压缩上表现出了令人印象深刻的性能,其中在基于类注意力转移的知识蒸馏(CAT-KD)这项工作中证明了通过转移类激活图可以使学生模型获得和增强识别输入分类区域的能力,这种能力是当前主流 CNN 模型进行分类的关键。 其通过将类激活图平均池化和 l2 归一化的方式来转移类激活图,从而增强学生模型识别输入分类区域的能力,提高蒸馏性能。 然而,这种方式忽略了类激活图中的通道相关的知识,这对于学生模型学习识别输入分类区域的能力至关重要。 为了解决上述问题,该文提出了基于通道相关性的类注意力转移方法。 具体来说,为了从类激活图中提取丰富的知识,该方法不仅考虑了样本内的类激活图中不同通道的特征知识,还考虑了不同样本的类激活图中基于每通道特征的关系知识。 实验表明,该方法在 CIFAR-100 数据集上比基准方法提升了 0. 96 百分点,优于对比方法。

Abstract:

Previous knowledge distillation methods have shown impressive performance in model compression,among which in the work of Class Attention Transfer Based Knowledge Distillation (CAT-KD),it has been proven that the transfer class activation graph can enable student models to acquire and enhance the ability to recognize input classification regions,which is the key to current mainstream CNN models for classification. It enhances the ability of the student model to recognize input classification regions and improves distillation performance by transferring class activation maps through average pooling and l2 normalization. However, this approach ignores the channel related knowledge in the class activation maps,which is crucial for the student model’s ability to learn and recognize input classification regions. To address the aforementioned issues, we propose a class attention transfer method based on channel correlation. Specifically,in order to extract rich knowledge from class activation maps,the proposed method not only considers the feature knowledge of different channels in the class activation maps within the samples,but also considers the relationship knowledge based on each channel feature in the class activation maps of different samples. The experiment shows that the proposed method has improved by 0. 96 percentage points compared to the benchmark method on the CIFAR-100 dataset,which is better than the comparison method.

常用功能

更新日期/Last Update: 2024-12-10