Through adding a nonlinear self-feedback term in the evolution equations of neural network, we introduced a transiently chaotic neural network model. In order to utilize the transiently chaotic dynamics mechanism in optimization problem efficiently, we have analyzed the dynamical procedure of the transiently chaotic neural network rnodel and studied the function of the crucial bifurcation parameter which governs the chaotic behavior of the system. Based on the dynamical analysis of the transiently chaotic neural network model, chaotic annealing algorithm is also examined and improved. As an example, we applied chaotic annealing method to the traveling salesman problem and obtained good results.``