UAV-assisted D2D networks can provide auxiliary communication for areas with poor com-munication facilities by using the characteristics of easy deployment of unmanned aerial vehicle (UAV), then it becomes a promising technology. However, the coexistence of UAV and D2D aggravates the con-flict of spectrum resources. In addition, when the UAV performs the communication service, it will in-evitably cause the location change, which will make the original channel allocation no longer applicable. Inspired by the influence of frequent channel switch-ing on channel allocation, we define the communica-tion utility as a tradeoff between the throughput and channel switching cost. In the considered model, we investigate the multi-stage hierarchical spectrum ac-cess problem with maximizing aggregate communica-tion utilities in UAV-assisted D2D networks. In partic-ular, due to the hierarchical feature of the considered network, we adopt Stackelberg game to formulate this spectrum access problem where both the throughput and channel switching cost are considered. We prove that the proposed game has a stable Stackelberg equi-librium (SE), and the heterogeneous network based channel allocation (HN-CA) algorithm is proposed to achieve the desired solution. Simulation results verify the validity of the proposed game and show the effec-tiveness of the HN-CA algorithm.