The paper was numerically focused on investigation of deformation, failure and instability of shear band-surrounding elastic rock system in plane strain direct shear test considering shear dilatancy according to fast lagrangian analysis of continua (FLAC). The adopted failure criterion was a composite Mohr-Coulomb criterion with tension cut-off and post-peak constitutive relation of rock, i.e. linear strain-softening. Numerical results show that dilation angle affects the responses of elements, the number and the position of yielded elements. Increasing dilation angle results in higher load-carrying capability of elements, higher deformation or strain corresponding to peak stress, less brittle post-peak stress-deformation curve. Strain-hardening behavior can occur if dilation angle is high.Therefore, shear band-elastic rock body system tends to be stable and rock burst does not occur as dilation angle is increased. Moreover, the number of yielded elements is increased with dilation angle increase and two parallel plastic zones initially generated in the middle of two loading ends of specimen no longer remain collinear, reflecting increase in deformation resistant of the system. Evolution of volumetric strain rate was investigated based on three-dimensional curved surface diagrams. Approximately, volumetric strain rate concentration regions coincide with plastic zones. Volumetric strain rate in yielded elements is very evident. However, in elastic zones volumetric strain has not been nearly changed throughout the numerical tests.