This paper proposes a novel algorithm for extracting key frames to represent video shots. Regarding whether, or how well, a key frame represents a shot, different interpretations have been suggested. We develop our algorithm on the assumption that more important content may demand more attention and may last relatively more frames. Unsupervised clustering is used to divide the frames into clusters within a shot, and then a key frame is selected from each candidate cluster. To make the algorithm independent of video sequences, we employ a statistical model to calculate the clustering threshold. The proposed algorithm can capture the important yet salient content as the key frame. Its robustness and adaptability are validated by experiments with various kinds of video sequences.