We proposed a hybrid transmission system consist of 1Tbit/s Nyquist WDM channels,100Gbit/s PDM-QPSK WDM channels and 10Gbit/s NRZ WDM channels to study whether Terabit Nyquist WDM signal could transmit in the existing hybrid systems with DCM(dispersion compensation module). The simulation results demonstrated the influence of all these three kinds of signals. The 1Tbit/s Nyquist WDM channels introduced 1.5d B Q penalty to the 100Gbit/s PDM-QPSK signals and their influence on the 10Gbit/s NRZ signals was extremely slight. In the hybrid optical system,1Tbit/s Nyquist channel has 5d B Q penalty due to the inline DCM and 2.5d B Q penalty due to the channel interference from existing channels. Therefore,the total Q penalty is 7.5d B caused by strong nonlinear effects at the launch power of 0d Bm. The results have the guiding significance for the design of future backbone system and also support the feasibility of future dynamic optical system. We propose a hybrid transmission system consisting of 1Tbit / s Nyquist WDM channels, 100Gbit / s PDM-QPSK WDM channels and 10Gbit / s NRZ WDM channels to study whether Terabit Nyquist WDM signal could transmit in the existing hybrid systems with DCM (dispersion compensation module The simulation results demonstrated the influence of all these three kinds of signals. The 1Tbit / s Nyquist WDM channels introduced 1.5d BQ penalty to the 100Gbit / s PDM-QPSK signals and their influence on the 10Gbit / s NRZ signals was extremely slight . The total Q penalty is 7.5d B caused by strong nonlinear effects at the launch power of 0d Bm. The results have the guiding significance for the design of future backbone system and also support the feasibility of future dynamic optical system.