Taking Jiuhong Modern Agriculture Demonstration Park of Heilongjiang Province as the base for rice disease image acquisition, a total of 841 images of the four different diseases, including rice blast, stripe leaf blight, red blight and bacterial brown spot, were obtained. In this study, an interleaved attention neural network (IANN) was proposed to realize the recognition of rice disease images and an interleaved group convolutions (IGC) network was introduced to reduce the number of convolutional parameters, which realized the information interaction between channels. Based on the convolutional block attention module (CBAM), attention was paid to the features of results of the primary group convolution in the cross-group convolution to improve the classification performance of the deep learning model. The results showed that the classification accuracy of IANN was 96.14％, which was 4.72％ higher than that of the classical convolutional neural network (CNN). This study showed a new idea for the efficient training of neural networks in the case of small samples and provided a reference for the image recognition and diagnosis of rice and other crop diseases.