[1]秦军[],冯亮亮[],孙蒙[]. 基于异构Hadoop集群的负载均衡策略研究[J].计算机技术与发展,2017,27(06):110-113.
　QIN Jun[],FENG Liang-liang[],SUN Meng[]. Research on Load Balancing Strategy with Heterogeneous Hadoop Clustering[J].,2017,27(06):110-113.
点击复制

 基于异构Hadoop集群的负载均衡策略研究()
分享到：

《计算机技术与发展》[ISSN:1006-6977/CN:61-1281/TN]

卷:
27
期数:
2017年06期
页码:
110-113
栏目:
智能、算法、系统工程
出版日期:
2017-06-10

文章信息/Info

Title:
 Research on Load Balancing Strategy with Heterogeneous Hadoop Clustering
文章编号:
1673-629X（2017)06-0110-04
作者:
 秦军[1] ; 冯亮亮[2] ; 孙蒙[2]
 1.南京邮电大学 教育科学与技术学院；2.南京邮电大学 计算机学院
Author(s):
 QIN Jun[1] ; FENG Liang-liang[2] ; SUN Meng[2]
关键词:
 ; Hadoop集群; MapReduce; 节点性能; 任务调度; 负载均衡
Keywords:
 ; Hadoop clustering; MapReduce; node performance; task scheduling; load balancing
分类号:
TP301.6
文献标志码:
A
摘要:
 异构Hadoop环境中,每个节点的处理能力各不相同,且集群中的节点会不断增加和删除,随着作业量的增大,负载倾斜会越来越明显.显然,负载均衡也成为影响Hadoop集群性能的重要因素之一.针对异构Hadoop环境中MapReduce任务调度,提出了一种新的负载均衡算法.该算法充分利用节点性能和当前的计算资源,根据集群负载平衡度量值进行任务分配,将任务分配给适合的节点,使集群负载逐渐趋于平衡,以提高集群节点利用率.由于Hadoop集群中各节点通过网络连接,以节省网络传输代价,因此在负载均衡调度时,根据数据分布特点,优先考虑数据的本地性,以缩短任务执行时间.仿真实验结果表明,所提出的负载均衡算法能明显改善系统性能,有效缩短MapReduce作业执行时间.
Abstract:
 In the heterogeneous Hadoop environment,processing capabilities of the nodes are diverse and various,among which each node may be continuously added or removed in the clustering and its load slope may increase obviously with tasks.Apparently,load balancing is one of the most important factors that affect the performance of Hadoop clustering.Thus a new load balancing algorithm has been proposed and employed in MapReduce task scheduling of the heterogeneous environment,which makes full use of the node performance and current computation resources according to the cluster load balancing measurement value to allocate task to suitable node for balancing the cluster load gradually and promoting coefficient of utilization of cluster nodes.Since the nodes in the Hadoop clustering are connected with network to save the costs of network transmission and the data locality should be considered with priority to decrease execution time for each task according to the characteristics of data distribution during load balancing scheduling.Simulation results show that the proposed load balancing algorithm has improved performances of whole system significantly and shorten the execution time of the MapReduce task.

相似文献/References:

[1]张志宏,吴庆波,邵立松,等.基于飞腾平台TOE协议栈的设计与实现[J].计算机技术与发展,2014,24(07):1.
　ZHANG Zhi-hong,WU Qing-bo,SHAO Li-song,et al. Design and Implementation of TCP/IP Offload Engine Protocol Stack Based on FT Platform[J].,2014,24(06):1.
[2]梁文快,李毅. 改进的基因表达算法对航班优化排序问题研究[J].计算机技术与发展,2014,24(07):5.
　LIANG Wen-kuai,LI Yi. Research on Optimization of Flight Scheduling Problem Based on Improved Gene Expression Algorithm[J].,2014,24(06):5.
[3]黄静,王枫,谢志新,等. EAST文档管理系统的设计与实现[J].计算机技术与发展,2014,24(07):13.
　HUANG Jing,WANG Feng,XIE Zhi-xin,et al. Design and Implementation of EAST Document Management System[J].,2014,24(06):13.
[4]侯善江[],张代远[][][]. 基于样条权函数神经网络P2P流量识别方法[J].计算机技术与发展,2014,24(07):21.
　HOU Shan-jiang[],ZHANG Dai-yuan[][][]. P2P Traffic Identification Based on Spline Weight Function Neural Network[J].,2014,24(06):21.
[5]李璨,耿国华,李康,等. 一种基于三维模型的文物碎片线图生成方法[J].计算机技术与发展,2014,24(07):25.
　LI Can,GENG Guo-hua,LI Kang,et al. A Method of Obtaining Cultural Debris’ s Line Chart Based on Three-dimensional Model[J].,2014,24(06):25.
[6]翁鹤,皮德常. 混沌RBF神经网络异常检测算法[J].计算机技术与发展,2014,24(07):29.
　WENG He,PI De-chang. Chaotic RBF Neural Network Anomaly Detection Algorithm[J].,2014,24(06):29.
[7]刘茜[],荆晓远[],李文倩[],等. 基于流形学习的正交稀疏保留投影[J].计算机技术与发展,2014,24(07):34.
　LIU Qian[],JING Xiao-yuan[,LI Wen-qian[],et al. Orthogonal Sparsity Preserving Projections Based on Manifold Learning[J].,2014,24(06):34.
[8]尚福华,李想,巩淼. 基于模糊框架-产生式知识表示及推理研究[J].计算机技术与发展,2014,24(07):38.
　SHANG Fu-hua,LI Xiang,GONG Miao. Research on Knowledge Representation and Inference Based on Fuzzy Framework-production[J].,2014,24(06):38.
[9]叶偲,李良福,肖樟树. 一种去除运动目标重影的图像镶嵌方法研究[J].计算机技术与发展,2014,24(07):43.
　YE Si,LI Liang-fu,XIAO Zhang-shu. Research of an Image Mosaic Method for Removing Ghost of Moving Targets[J].,2014,24(06):43.
[10]余松平[][],蔡志平[],吴建进[],等. GSM-R信令监测选择录音系统设计与实现[J].计算机技术与发展,2014,24(07):47.
　YU Song-ping[][],CAI Zhi-ping[] WU Jian-jin[],GU Feng-zhi[]. Design and Implementation of an Optional Voice Recording System Based on GSM-R Signaling Monitoring[J].,2014,24(06):47.
[11]李聪颖[],王瑞刚[],梁小江[]. 基于Hadoop的交互式大数据分析查询处理方法[J].计算机技术与发展,2016,26(08):134.
　LI Cong-ying[],WANG Rui-gang[],LIANG Xiao-jiang[]. An Interactive Processing Method of Analysis and Query for Big Data Based on Hadoop[J].,2016,26(06):134.

更新日期/Last Update: 2017-07-26