论文部分内容阅读
通过实验对FIFO、CAP和FAIR三种调度算法的特点与性能进行了对比分析,得出了它们各自优点和存在的问题.针对公平调度算法Fair Scheduler不适用于内存密集型作业调度的缺点,提出一种基于内存平衡的公平调度算法FMScheduler,在整个调度的过程中考虑作业的内存使用和计算节点的内存情况,通过加入内存比较机制、调整作业公平权重计算方法以及引入作业预留机制,对原有Hadoop公平调度算法进行改进与优化.最后,通过仿真实验对FMScheduler进行测试分析,实验结果表明,FMScheduler在高内存作业调度环境下的独立响应时间和作业整体的平均响应时间都比Fair Scheduler有所减少;并且在多用户多作业且包含内存密集型作业的环境中,FMScheduler与Hadoop原有的三种调度算法相比,在处理数据密集型作业和内存密集型作业的混合场景时,能够更合理公平地调度作业.
The characteristics and performance of the three scheduling algorithms of FIFO, CAP and FAIR are analyzed experimentally, and their respective advantages and disadvantages are obtained.Aiming at the disadvantage that fair scheduling algorithm is not suitable for memory-intensive job scheduling, A fair scheduling algorithm based on memory balance, FMScheduler, considers the memory usage of the job and the memory of the computing node in the whole scheduling process. By adding the memory comparison mechanism, adjusting the calculation method of the job fairness weight and introducing the job reservation mechanism, Hadoop fair scheduling algorithm is improved and optimized.Finally, FMScheduler is tested and analyzed through simulation experiments. The experimental results show that FMScheduler’s independent response time and overall job response time are higher than that of Fair Scheduler in high memory job scheduling And in multi-user multi-job and memory-intensive jobs, FMScheduler can be more reasonable when dealing with mixed scenarios of data-intensive jobs and memory-intensive jobs compared to Hadoop’s original three scheduling algorithms Dispatch homework fairly.