Mastering the influence laws of parameters on the solution structure of nonlinear systems is the basis of carrying out vibration isolation and control. Many researches on solution structure and bifurcation phenomenon in parameter spaces are carried out broadly in many fields, and the research on nonlinear gear systems has attracted the attention of many scholars. But there is little study on the solution domain boundary of nonlinear gear systems. For a periodic non-autonomous nonlinear dynamic system with several control parameters, a solution domain boundary analysis method of nonlinear systems in parameter spaces is proposed, which combines the cell mapping method based on Poincare point mapping in phase spaces with the domain decomposition technique of parameter spaces. The cell mapping is known as a global analysis method to analyze the global behavior of a nonlinear dynamic system with finite dimensions, and the basic idea of domain decomposition techniques is to divide and rule. The method is applied to analyze the solution domain boundaries in parameter spaces of a nonlinear gear system. The distribution of different period domains, chaos domain and the domain boundaries between different period domains and chaotic domain are obtained in control parameter spaces constituted by meshing damping ratio with excitation frequency, fluctuation coefficient of meshing stiffness and average exciting force respectively by calculation. The calculation results show that as the meshing damping increases, the responses of the system change towards a single motion, while the variations of the excitation frequency, meshing stiffness and exciting force make the solution domain presenting diversity. The proposed research contribution provides evidence for vibration control and parameter design of the gear system, and confirms the validity of the solution domain boundary analysis method.