论文部分内容阅读
为了使Hadoop集群系统能够应对多变的任务及系统本身节点差异对集群性能带来的影响,采用TaskConfigure服务器构建Hadoop集群参数信息库系统实现对集群参数的自动调优配置.通过对集群节点及任务的分类,提出集群按类分配配置参数及采用节点资源利用效率生成集群系统参数的优化配置值.实验结果表明,参数信息库系统的自动调优保证了集群工作性能的充分发挥,有效地缩短了集群执行任务的工作时间,使集群具有良好的稳定性和扩展性.
In order to make Hadoop cluster system able to cope with changing tasks and the impact of node differences on cluster performance, the TaskConfigure server is used to construct Hadoop cluster parameter information database system to automatically tune the cluster parameters.Through cluster nodes and tasks , And puts forward the optimal allocation of cluster system parameters according to the class distribution configuration parameters and node resource utilization efficiency.Experimental results show that the automatic tuning of the parameter database system ensures the full performance of the cluster and effectively shortens Working hours of the cluster to perform tasks, so that the cluster has good stability and scalability.