Machine vision based automated inspection and monitoring for railway Infrastructures, such as pantograph overhead lines, is a promising technical trend to increase the efficiency and ease the manpower burdens.Vision data are naturally informative and comply with human sensing and cognition.However, automatically detecting and locate the infrastructure facilities from variance cluttered background in rail-lines inspection videos is still a challenging task due to the ill-pose essential.We propose a cantenary poles and gantries segmentation framework by combining the appearance and motion patterns via a sequential Bayesian approach.The poles and supporting arms of power supply lines are detected firstly to yield the region of interest for detailed processing.Then the motion hypothesis of foreground and background are estimated from the edge flow extracted from local curve and line segments,so as to occlusion reasoning.After that, the pole model and background model are implemented to classify the candidates in hypothesis.Finally, promising experimental results demonstrate the potentials of the proposed poles segmentation method with respect to various insignificant patterns and cluttered backgrounds.