We use the advanced proximal policy optimization (PPO) reinforcement leaing algorithm to optimize the stochastic control strategy to achieve speed control of the "model-free" quadrotor. The model is controlled by four leaed neural networks, which directly map the system states to control commands in an end-to-end style. By introducing an integral compensator into the actor-critic framework, the speed tracking accuracy and robustness have been greatly enhanced. In addition, a two-phase leaing scheme which includes both offline- and online-leaing is developed for practical use. A model with strong generalization ability is leaed in the offline phase. Then, the flight policy of the model is continuously optimized in the online leaing phase. Finally, the performances of our proposed algorithm are compared with those of the traditional PID algorithm.