论文部分内容阅读
深层神经网络(DNN)的参数量巨大,限制了其在一些计算资源受限或是注重速度的应用场景中的应用。为了降低DNN参数量,有学者提出利用奇异值分解(SVD)对DNN进行裁剪,然而其方法缺乏自适应性,因为它会从所有隐层裁减掉同样数量的奇异值。该文提出了一种基于奇异值比率裁剪因子(singular rate pruning factor,SRPF)的DNN裁剪方法。该方法以数据驱动的方式分别为DNN的各个隐层计算出SRPF,然后以不同的裁剪因子对各隐层进行裁剪,这充分利用了各隐层权值矩阵的奇异值分布特性。与固定数量裁剪法相比,该方法具有自适应性。实验表明:在同样裁剪力度下,该方法给DNN造成的性能损失更小。另外,该文还提出了一种适合裁剪后的DNN的重训练方法。
Deep Neural Network (DNN) has a huge amount of parameters, which limits its application in some applications where computing resources are limited or speed-oriented. In order to reduce the amount of DNN parameters, some scholars propose to singulate DNN by using singular value decomposition (SVD). However, its method lacks self-adaptability because it subtracts the same number of singular values from all hidden layers. This paper proposes a DNN clipping method based on singular rate pruning factor (SRPF). In this method, SRPF is calculated for each hidden layer of DNN by data-driven method, and each hidden layer is clipped with different clipping factors, which makes full use of the singular value distribution of each hidden layer weight matrix. Compared with the fixed number clipping method, this method is adaptive. Experiments show that, under the same cutting force, the performance loss caused by this method to DNN is smaller. In addition, the paper also proposed a training method suitable for tailored DNN.