Significant and persistent trajectory-to-trajectory variance are commonly observed in particle tracking experiments, which have become a major challenge for the experimental data analysis. In this theoretical paper we investigate the ergodicity recovery behavior, which helps clarify the origin and the convergence of trajectory-to-trajectory fluctuation in various heterogeneous disordered media. The concepts of self-averaging and ergodicity are revisited in the context of trajectory analysis. The slow ergodicity recovery and the non-Gaussian diffusion in the annealed disordered media are shown as the consequences of the central limit theorem in different situations. The strange ergodicity recovery behavior is reported in the quenched disordered case, which arises from a localization mechanism. The first-passage approach is introduced to the ergodicity analysis for this case, of which the central limit theorem can be employed and the ergodicity is recovered in the length scale of diffusivity correlation.