A novel vortex identification method for the visualization of the flow field is used for the study of the stall process of a transonic compressor. The parameter η4, which is one of the five invariants formed by the stain rate and vorticity tensors from the theory of modern rational mechanics, is found to have good ability to identify vortex stretching and vortex relaxation/breakdown processes, is introduced here to identify the tip leakage vortices. Compare with former generally used DPH(dynamic pressure head) contour, the new method reveals much more flow details which may advance our understanding of the compressor behaviors. The Vortices details are revealed in both peak efficiency and near stall condition. A possible stall process is also suggested based on the vortices analysis. The tip leakage flow from mid-chord, besides leading edge leakage flow, is also considered to play an important role in the stall process.