Using form invariance under special infinitesimal transformations in which time is not variable, the non-Noether conserved quantity of the relativistic nonholonomic system with variable mass is studied. The differential equations of motion of the system are established. The definition and criterion of the form invariance of the system under infinitesimal transformations are studied. The necessaxy and sufficient condition under which the form invariance is a Lie symmetry is given. The condition under which the form invariance can be led to a non-Noether conserved quantity and the form of the conserved quantity are obtained. Finally, an example is given to illustrate the application of the result.