We presented a novel framework for au-tomatic behavior clustering and unsupervised anom-aly detection in a large video set. The framework consisted of the following key components: 1) Drawing from natural language processing, we in-troduced a compact and effective behavior repre-sentation method as a stochastic sequence of spatio-temporal events, where we analyzed the global structural information of behaviors using their local action statistics. 2) The natural grouping of behavior patterns was discovered through a novel clustering algorithm. 3) A run-time accumulative anomaly measure was introduced to detect abnormal behav-ior, whereas normal behavior patterns were recog-nized when sufficient visual evidence had become a-vailable based on an online Likelihood Ratio Test (LRT) method. This ensured robust and reliable a-nomaly detection and normal behavior recognition at the shortest possible time. Experimental results demonstrated the effectiveness and robustness of our approach using noisy and sparse data sets col-lected from a real surveillance scenario.