Stability and dynamic characteristics of a ball bearing-rotor system are investigated under the effect of the clearance in the ball bearing. Different clearance values are assumed to calculate the nonlinear stability of periodic solution with the aid of the Floquet theory. Bifurcation and chaos behavior are analyzed with variation of the clearance and rotational speed. It is found that there are three routes to unstable periodic solution.The period-doubling bifurcation and the secondary Hopf bifurcation are two usual routes to instability. The third route is the boundary crisis, a chaotic attractor occurs suddenly as the speed passes through its critical value. At last, the instable ranges for different internal clearance values are described. It is useful to investigate the stability property of ball bearing rotor system.