In YCbCr colorspace, a method is proposed to reconstruct the background and extract moving objects based on the Gaussian model of chroma components. Background model is updated according to changes of chroma components. In order to eliminate the disturbance of shadow, a shadow detecting principle is proposed in YCbCr colorspace. A Kalman filter is introduced to estimate objects’ positions in the image and then the pedestrian is tracked according to its information of shape. Experiments show that the background reconstruction and updating are successful, object extraction and shadow suppression are satisfactory, and real-time and reliable tracking is realized.