Due to the calculation problem of classical methods (such as Lyapunov exponent) for chaotic behavior, a new method of identifying nonlinear dynamics with higher-order time-frequency entropy (HOTFE) based on time-frequency analysis and information theorem is proposed. Firstly, the meaning of HOTFE is defined, and then its validity is testified by numerical simulation. In the end vibration data from rotors are analyzed by HOTFE. The results demonstrate that it can indeed identify the early rub-impact chaotic behavior in rotors and also is simpler to calculate than previous methods.