Large-scale semantic concept detection from large video database suffers from the large variations among different semantic concepts as well as their corresponding effective low-level features. In this paper, we propose a novel framework to deal with this obstacle. The proposed framework consists of four major components: feature pool construction, pre-filtering, modeling, and classification. First, a large low-level feature pool is constructed, from which a specific set of features are selected for the latter steps automatically or semi-automatically. Then, to deal with the unbalance problem in training set, a pre-filtering classifier is generated, which aim at achieving a high recall rate and a certain precision rate nearly 50％ for a certain concept. Thereafter, from the pre-filtered training samples, a SVM classifier is built based on the selected features in the feature pool. After that, the SVM classifier is applied to classification of semantic concept. This framework is flexible and extensible in terms of adding new features into the feature pool, introducing human interactions on selecting features, building models for new concepts and adopting active learning.