Hard photon emitted from energetic heavy ion collisions is of very interesting since it does not experience the late-stage nuclear interaction, therefore it is useful to explore the early-stage information of matter phase. In this work, we have calculated the impact parameter and beam energy dependence for azimuthal asymmetry, characterized by directed transverse flow parameter F and elliptic asymmetry coeffcient v2, of direct photons and the corresponding free protons in intermediate energy heavy-ion collisions. It is further shown the anti-correlated azimuthal asymmetry between direct photons and free protons. Hard photon emitted from energetic heavy ion collisions is of very interesting since it does not experience the late-stage nuclear interaction, therefore it is useful to explore the early-stage information of matter phase. In this work, we have calculated the impact parameter and beam energy dependence for azimuthal asymmetry, characterized by directed transverse flow parameter F and elliptic asymmetry coeffcient v2, of direct photons and the corresponding free protons in intermediate energy heavy-ion collisions. It is further shown the anti-correlated azimuthal asymmetry between direct photons and free protons.