Unmanned Aerial Vehicles (UAVs) are useful in dangerous and dynamic tasks such as search-and-rescue, forest surveillance, and anti-terrorist operations. These tasks can be solved bet-ter through the collaboration of multiple UAVs under human supervision. However, it is still dif-ficult for human to monitor, understand, predict and control the behaviors of the UAVs due to the task complexity as well as the black-box machine learning and planning algorithms being used. In this paper, the coactive design method is adopted to analyze the cognitive capabilities required for the tasks and design the interdependencies among the heterogeneous teammates of UAVs or human for coherent collaboration. Then, an agent-based task planner is proposed to automatically decom-pose a complex task into a sequence of explainable subtasks under constrains of resources, execu-tion time, social rules and costs. Besides, a deep reinforcement learning approach is designed for the UAVs to learn optimal policies of a flocking behavior and a path planner that are easy for the human operator to understand and control. Finally, a mixed-initiative action selection mechanism is used to evaluate the learned policies as well as the human's decisions. Experimental results demonstrate the effectiveness of the proposed methods.