Relative orbit determination is widely used for autonomously controlling the Satellite Formation Flying (SFF).Due to the fact that some traditional techniques cannot meet the high requirement on the accuracy of relative orbit determination for specific space missions,we intend to fulfil the mission by increasing the accuracy of inter-satellite distance measurement of SFF.Thus,a novel quantum ranging scheme is proposed,which utilizes the second order correlation between the entangled photons and the triangle geometry of SFF.We demonstrates that the ranging accuracy of this novel scheme can theoretically be increased and simulation results is in accord with it.Furthermore,the Extended Kalman Filter (EKF) is employed to estimate the relative state,which selects the quantum ranging scheme and a sensor onboard the chief satellite as the measurement model.Simulation results show that the estimation errors in position and velocity can reach 1 mm and 0.1 mm/s respectively.