This paper presents a hybrid ensemble classifier combined synthetic minority oversampling technique（SMOTE）, random search（RS） hyper-parameters optimization algorithm and gradient boosting tree（GBT） to achieve efficient and accurate rock trace identification. A thirteen-dimensional database consisting of basic, vector, and discontinuity features is established from image samples. All data points are classified as either ‘‘trace” or ‘‘non-trace” to divide the ultimate results into candidate trace samples. It is found that the SMOTE technology can effectively improve classification performance by recommending an optimized imbalance ratio of 1:5 to 1:4. Then, sixteen classifiers generated from four basic machine learning（ML） models are applied for performance comparison. The results reveal that the proposed RS-SMOTE-GBT classifier outperforms the other fifteen hybrid ML algorithms for both trace and nontrace classifications. Finally, discussions on feature importance, generalization ability and classification error are conducted for the proposed classifier. The experimental results indicate that more critical features affecting the trace classification are primarily from the discontinuity features. Besides, cleaning up the sedimentary pumice and reducing the area of fractured rock contribute to improving the overall classification performance. The proposed method provides a new alternative approach for the identification of 3D rock trace.