The transient critical boundary of dynamic security region (DSR) can be approximated by a few hyper planes correlated with instability separation modes. A method to fast predict instability separation modes is proposed for DSR calculation in power injection space. The method identifies coherent generation groups by the developed K-medoids algorithm, taking a similarity matrix derived from the reachability Grammian as the index. As an experimental result, reachability Grammian matrices under local injections are approximately invariant. It indicates that the generator coherency identifications are nearly consistent for different injections. Then instability separation modes can be predicted at the normal operating point, while average initial acceleration is considered as the measure of the critical generator group to amend the error. Moreover, based on these predicted instability separation modes, a critical point search strategy for DSR calculation is illustrated in the reduced injection space of the critical generators. The proposed method was evaluated using New England Test System, and the computation accuracy and speed in determining the practical DSR were improve.