This work deals with super-harmonic responses and the stabilities of a gear transmission system of a high-speed train under the stick-slip oscillation of the wheel-set. The dynamic model of the system is developed with consideration on the factors including the time-varying system stiffness, the transmission error, the tooth backlash and the self-excited excitation of the wheel-set. The frequency-response equation of the system at super-harmonic resonance is obtained by the multiple scales method, and the stabilities of the system are analyzed using the perturbation theory. Complex nonlinear behaviors of the system including multi-valued solutions, jump phenomenon, hardening stiffness are found. The effects of the equivalent damping and the loads of the system under the stick-slip oscillation are analyzed. It shows that the change of the load can obviously influence the resonance frequency of the system and have little effect on the steady-state response amplitude of the system. The damping of the system has a negative effect, opposite to the load. The synthetic damping of the system composed of meshing damping and equivalent damping may be less than zero when the wheel-set has a large slippage, and the system loses its stability owing to the Hopf bifurcation. Analytical results are validated by numerical simulations.