Damage detection is a key procedure in maintenance throughout structures\' life cycles and post-disaster loss assessment. Due to the complex types of structural damages and the low efficiency and safety of manual detection, detecting damages with high efficiency and accuracy is the most popular research direction in civil engineering. Computer vision (CV) technology and deep learning (DL) algorithms are considered as promising tools to address the aforementioned challenges. The paper aims to systematically summarized the research and applications of DL-based CV technology in the field of damage detection in recent years. The basic concepts of DL-based CV technology are introduced first. The implementation steps of creating a damage detection dataset and some typical datasets are reviewed. CV-based structural damage detection algorithms are divided into three categories, namely, image classification-based (IC-based) algorithms, object detection-based (OD-based) algorithms, and semantic segmentation-based (SS-based) algorithms. Finally, the problems to be solved and future research directions are discussed. The foundation for promoting the deep integration of DL-based CV technology in structural damage detection and structural seismic damage identification has been laid.