In order to achieve the accurate measurement of displacement, this Letter presents a self-mixing interference displacement measurement method suitable for the speckle effect. Because of the speckle effect, the amplitude of the self-mixing interference signal fluctuates greatly, which will affect the measurement accuracy of displacement. The ensemble empirical mode decomposition is used to process the interference signal, which can filter out high-frequency noise and low-frequency noise at the same time. The envelope of the self-mixing interference signal is extracted by Hilbert transform, and it is used to realize the normalization of the signal. Through a series of signal processing, the influence of speckle can be effectively reduced, and the self-mixing interference signal can be transformed into standard form. The displacement can be reconstructed by fringe counting and the interpolation method. The experimental results show that the method is successfully applied to the displacement measurement in the presence of speckle, which verifies the effectiveness and feasibility of the method.