Surface-enhanced Raman scattering (SERS) and photoluminescence (PL) are important photoexcitation spectroscopy techniques;however, understanding how to analyze and modulate the relationship between SERS and PL is rather important for enhancing SERS, having a great effect on practical applications. In this work, a charge-transfer (CT) mechanism is proposed to investigate the change and relationships between SERS and PL. Analyzing the change in PL and SERS before and after the adsorption of the probe molecules on Nd-doped ZnO indicates that the unique optical characteristics of Nd3+ions increase the SERS signal. On the other hand, the observed SERS can be used to explain the cause of PL background reduction. This study demonstrates that modulating the interaction between the probe molecules and the substrate can not only enhance Raman scattering but also reduce the SERS background. Our work also provides a guideline for the investigation of CT as well as a new method for exploring fluorescence quenching.