To understand the content of ultrasound images more conveniently and more quickly, in this paper, we propose a coarse-to-fine ultrasound image captioning ensemble model, which can automatically generate the annotation text that is composed of relevant n-grams to describe the disease information in the ultrasound images. First, the organs in the ultrasound images are detected by the coarse classification model. Second, the ultrasound images are encoded by the corresponding fine-grained classification model according to the organ labels. Finally, we input the encoding vectors to the language generation model, and the language generation model generates automatically annotation text to describe the disease information in the ultrasound images. In our experiments, the encoding model can obtain the high accuracy rate in the ultrasound image recognition. And the language generation model can automatically generate high-quality annotation text. In practical applications, the coarse-to-fine ultrasound image captioning ensemble model can help patients and doctors obtain the well understanding of the contents of ultrasound images.