This paper proposes a novel video object tracking approach using birdirectional projection. Forward projection is exploited to locate the current video object with rough boundary information. Watershed segmentation is applied to the simplified gradient image of the current frame to obtain a reasonable partition. An improved backward projection, which incorporates pixel classification with region classification, is performed on some segmented regions in a rather small search range, and the tracking performance is enhanced in respect to both reliability and efficiency. Experimental results for various types of the MPEG-4 (moving picture experts group) test sequences demonstrate an efficient and faithful segmentation performance of the proposed approach.