In this paper, two new similarity measure methods based on set theory were proposed. Firstly, similarity measure of two sets based on set theory and set operation was discussed. This principle was used to spectral vectors, and two approaches were proposed. The first method was to create a spectral polygon corresponding to spectral curve, and similarity of two spectral vectors can be replaced by that of two polygons. Area of spectral polygon was used as quantification function and some effective indexes for similarity and dissimilarity were computed. The second method was to transform the original spectralvector to encoding vector according to absorption or reflectance feature bands, and similarity measure was conducted to encoding vectors. It proved that the spectral polygon-based approach was effective and can be used to hyperspectral RS image retrieval.