We study the rotation of the polarization plane for a ray of electromagnetic radiation propagating in the grav-itoelectromagnetic field caused by a rotating massive object with the quadrupole moment. The effect of thequadrupole moment on the gravitational Faraday rotation is investigated. It is found that the gravitational Fara-day effect of the quadrupole moment is negligible for Kerr black holes, but this effect is important for rapidlyrotating neutron stars.