Mode matching method is used to analyze the scattering characteristics of the rectangular-to-circular groove waveguide junction. Firstly, the scattering matrix equation is obtained by matching the electromagnetic fields at the boundary of the junction. The scattering coefficients can be obtained from the equation. Secondly the scattering characteristics of the iris with rectangular window positioned in circular groove waveguide are briefly analyzed. Thirdly, the convergent problem is discussed and the numerical results are given. At last experiment is made and good agreement is found between the calculated results and the measured results. Firstly, the scattering matrix equation is obtained by matching the electromagnetic fields at the boundary of the junction. The scattering coefficients can be obtained from the equation . Twice the scattering characteristics of the iris with rectangular window positioned in circular groove waveguide are briefly analyzed. Thirdly, the convergent problem is discussed and the numerical results are given. At last experiment is made and good agreement is found between the calculated results and the measured results.