A new algorithm of background reconstruction and moving foreground extraction for stereo video with static backgrounds is proposed. The background is reconstructed based on blocks and the moving foregrounds are separated by comparing the reconstruction of the backgrounds information with every frame. The result shows that it can accurately achieve the division between foreground and background, and significantly reduce the stereo matching time and the information volume.