We consider rotational motion of an interacting atomic Bose-Einstein condensate (BEC) with both two- and threebody interactions in a quadratic-plus-quartic and harmonic-plus-Gaussian trap. By using the variational method, the influence of the three-body interaction and the anharmonicity of the trap on the lowest energy surface mode excitation and the spontaneous shape deformation (responsible for the vortex formation) in a rotating BEC is discussed in detail. It is found that the repulsive three-body interaction helps the formation of the vortex and reduces the lowest energy surface mode frequency and the critical rotational frequency of the system. Moreover, the critical rotational frequency for the vortex formation in the harmonic-plus-Gaussian potential is lower than that in the quadratic-plus-quartic potential.