A method based on local HSV image and the shape of object to recognize object is proposed for robot tracking. After the color segment, the knowledge of the shape of objects is used to recognize objects. The robot tracking result testifies the avail-ability of the method. A method based on local HSV image and the shape of object to recognize object is proposed for robot tracking. After the color segment, the knowledge of the shape of objects is used to recognize objects. The robot tracking result testifies the avail-ability of the method.