Vector beams with spatially varying polarization distribution in the wavefront plane have received increasing attention in recent years. The manipulation of vector beams both in intensity and polarization distributions is highly desired and under development. In this work, we study the transmission property of vector beams through warm rubidium vapor and realize controllable transmission of vector beams based on atomic dichroism. By utilizing the linearly polarized beam and vector beams as the pump and probe beams in a pump–probe configuration, a spatially-dependent dichroism can be obtained, which leads to spatially varied absorption of the probe beam. The controllable intensity distribution of the probe beam, as a two-petal patt, can rotate with the variation of the pump beam’s polarization states. We experimentally demonstrate the mechanism of dichroism with linear polarization light and provide an explanation based on the optical pumping effect. Altatively, the varying trend of the probe beam’s intensity is also interpreted by utilizing the Jones matrix. Our results are thus beneficial for providing potential applications in optical manipulation in atomic ensembles.