With the popularization of smartphones and high-speed networks, a larger num-ber of users are getting used to watching videos online and have increasing requirements of video quality. Therefore, the video content delivery has become a progressively challenging task, especially for ultra-high-definition (UHD) videos and heterogonous networks. Recent-ly, quality of experience (QoE), which represents the true visual experience of users, has shown its advantages in management of video delivery and thus attracted increasing atten-tion. In a video delivery system, the user QoE can be greatly influenced by numerous ef-fects from video sources to display terminals. In this paper, we first investigate the signifi-cant differences between quality of service (QoS) and QoE. In addition, we summarize the end-to-end QoE effects in video delivery and present their classification based on the de-ployment. We also specifically analyze the impacts of different kinds of factors on QoE in video transmission systems.