To measure the radiation properties of relativistic diffraction generator(RDG) in Ka-band,a TM0n modal excitation model is established,which consists of an overmoded circular waveguide and a coaxial line feeding probe.Using the transverse E-field mode matching and the conservation of complex power technique(CCPT),we deduce the scattering matrix at coaxial line to coaxial line and coaxial line to circular waveguide junctions.Then using the overall cascaded junction scattering matrix,the numerical results for the reflection coefficient of the coaxial line and the power distribution of TM0n multi-modal are presented.The numerical results are in agreement with HFSS simulation results and experimental results.The analysis shows that by choosing the appropriate position of coaxial line probe,the power proportion of the device operating mode excited in circular waveguide could be the largest. To measure the radiation properties of the relativistic diffraction generator (RDG) in Ka-band, a TM0 modal excitation model is established, which consists of an overmoded circular waveguide and a coaxial line feeding probe. Using the transverse E-field mode matching and the conservation of complex power technique (CCPT), we deduce the scattering matrix at coaxial line to coaxial line and coaxial line to circular waveguide junctions. Using the overall cascaded junction scattering matrix, the numerical results for the reflection coefficient of the coaxial line and the power distribution of TM0n multi-modal are presented.The numerical results are in agreement with HFSS simulation results and experimental results. The analysis shows that by choosing the appropriate position of coaxial line probe, the power proportion of the device operating mode excited in circular channel could be the largest.