Polarization properties of single photons emitted by optical pumping from a single quantum dot (QD) are studied by using a four-level system model. The model is capable of explaining the polarization uncertainty observed in single photon emission experiments. It is found that the dependence of photon emission efficiency and polarization visibility on pump power are opposite in general cases. By employing QDs with small size and strong carrier confinement, the photon polarization visibility under high pump power can be improved. In addition, embedding a QD into a well designed microcavity is also found to be favourable, whereas the trade-off between high polarization visibility and multi-photon emission is noted.