We present an ameliorated arctangent algorithm based on phase-locked loop for digital Doppler signal processing,utilized within the heterodyne detection system.We define the error gain factor given by the approximation of Taylor expansion by means of a comparison of the measured values and true values.Exact expressions are derived for the amplitude error of two in-phase & quadrature signals and the frequency error of the acousto-optic modulator.Numerical simulation results and experimental results make it clear that the dynamic instability of the intermediate frequency signals leads to cumulative errors,which will spiral upward.An improved arctangent algorithm for the heterodyne detection is proposed to eliminate the cumulative errors and harmonic components.Depending on the narrow-band filter,our experiments were performed to realize the detectable displacement of 20 nm at a detection distance of 20 m.The aim of this paper is the demonstration of the optimized arctangent algorithm as a powerful approach to the demodulation algorithm,which will advance the signal-to-noise ratio and measurement accuracy of the heterodyne detection system.