[1]姚经纬[],杨福军[]. Redis分布式缓存技术在Hadoop平台上的应用[J].计算机技术与发展,2017,27(06):146-150.
　YAO Jing-wei[],YANG Fu-jun[]. Application of Redis Distributed Caching Technology in Hadoop Framework[J].,2017,27(06):146-150.
点击复制

 Redis分布式缓存技术在Hadoop平台上的应用()
分享到：

《计算机技术与发展》[ISSN:1006-6977/CN:61-1281/TN]

卷:
27
期数:
2017年06期
页码:
146-150
栏目:
应用开发研究
出版日期:
2017-06-10

文章信息/Info

Title:
 Application of Redis Distributed Caching Technology in Hadoop Framework
文章编号:
1673-629X（2017)06-0146-05
作者:
 姚经纬[1]; 杨福军[2]
 1.江南大学 物联网工程学院；2.中国空气动力研究与发展中心 计算空气动力研究所
Author(s):
 YAO Jing-wei[1]; YANG Fu-jun[2]
关键词:
 ; Redis; 分布式缓存; Hadoop; MapReduce
Keywords:
 ; Redis; distributed caching; Hadoop; MapReduce
分类号:
TP311.5
文献标志码:
A
摘要:
 在使用Hadoop进行大规模数据分析时,经常会遇到的一个较为典型的问题就是共享数据的快速访问问题.该类问题存在的场景很多,如网页排名算法、最小错误率训练算法、最大期望算法等.虽然已有关于此类问题的解决方案,但实际取得的效果却不尽如人意.为此,提出了使用Redis内存数据库作为分布式缓存,以解决Hadoop中共享数据访问的问题.验证实验结果表明,Redis分布式缓存的吞吐率与集群规模有较好的线性关系,所提出的方法能够较好地解决Hadoop任务对共享数据的访问问题,同时也为其他大规模共享数据访问的问题提供了简便的解决思路.Redis作为开源的商业化工具,使得所提出的方法具有较好的适用性,可为科研以及生产实践中遇到的同类问题提供一种较为通用的解决方案.
Abstract:
 In the scene of large scale data analysis with Hadoop,rapid accessing to shared resources is a typical problem that has not been satisfactorily solved so far.Examples of such problem include page rank algorithm,minimum error-rate training algorithm,expectation maximization algorithm and so on.Although solutions to such problems have existed,the actual effect is not satisfactory.Thus,an open-source distributed in-memory database,Redis,has been explored to provide high-throughput access to shared resources in Hadoop.Experimental results illustrate that Redis has the characteristic of linear increase in throughput with respect to cluster size so that it can provide a general-purpose solution for rapid accessing to shared resources in Hadoop cluster,and that it has provided an easier implementation of algorithms that has not been satisfactorily solved at large scale with Hadoop.Meanwhile,the use of Redis,the commercial-grade open-source tool,implies that the proposed solution has been easily adapted in both research and production environments.

相似文献/References:

[1]张志宏,吴庆波,邵立松,等.基于飞腾平台TOE协议栈的设计与实现[J].计算机技术与发展,2014,24(07):1.
　ZHANG Zhi-hong,WU Qing-bo,SHAO Li-song,et al. Design and Implementation of TCP/IP Offload Engine Protocol Stack Based on FT Platform[J].,2014,24(06):1.
[2]梁文快,李毅. 改进的基因表达算法对航班优化排序问题研究[J].计算机技术与发展,2014,24(07):5.
　LIANG Wen-kuai,LI Yi. Research on Optimization of Flight Scheduling Problem Based on Improved Gene Expression Algorithm[J].,2014,24(06):5.
[3]黄静,王枫,谢志新,等. EAST文档管理系统的设计与实现[J].计算机技术与发展,2014,24(07):13.
　HUANG Jing,WANG Feng,XIE Zhi-xin,et al. Design and Implementation of EAST Document Management System[J].,2014,24(06):13.
[4]侯善江[],张代远[][][]. 基于样条权函数神经网络P2P流量识别方法[J].计算机技术与发展,2014,24(07):21.
　HOU Shan-jiang[],ZHANG Dai-yuan[][][]. P2P Traffic Identification Based on Spline Weight Function Neural Network[J].,2014,24(06):21.
[5]李璨,耿国华,李康,等. 一种基于三维模型的文物碎片线图生成方法[J].计算机技术与发展,2014,24(07):25.
　LI Can,GENG Guo-hua,LI Kang,et al. A Method of Obtaining Cultural Debris’ s Line Chart Based on Three-dimensional Model[J].,2014,24(06):25.
[6]翁鹤,皮德常. 混沌RBF神经网络异常检测算法[J].计算机技术与发展,2014,24(07):29.
　WENG He,PI De-chang. Chaotic RBF Neural Network Anomaly Detection Algorithm[J].,2014,24(06):29.
[7]刘茜[],荆晓远[],李文倩[],等. 基于流形学习的正交稀疏保留投影[J].计算机技术与发展,2014,24(07):34.
　LIU Qian[],JING Xiao-yuan[,LI Wen-qian[],et al. Orthogonal Sparsity Preserving Projections Based on Manifold Learning[J].,2014,24(06):34.
[8]尚福华,李想,巩淼. 基于模糊框架-产生式知识表示及推理研究[J].计算机技术与发展,2014,24(07):38.
　SHANG Fu-hua,LI Xiang,GONG Miao. Research on Knowledge Representation and Inference Based on Fuzzy Framework-production[J].,2014,24(06):38.
[9]叶偲,李良福,肖樟树. 一种去除运动目标重影的图像镶嵌方法研究[J].计算机技术与发展,2014,24(07):43.
　YE Si,LI Liang-fu,XIAO Zhang-shu. Research of an Image Mosaic Method for Removing Ghost of Moving Targets[J].,2014,24(06):43.
[10]余松平[][],蔡志平[],吴建进[],等. GSM-R信令监测选择录音系统设计与实现[J].计算机技术与发展,2014,24(07):47.
　YU Song-ping[][],CAI Zhi-ping[] WU Jian-jin[],GU Feng-zhi[]. Design and Implementation of an Optional Voice Recording System Based on GSM-R Signaling Monitoring[J].,2014,24(06):47.
[11]王康[],李东静[],陈海光[]. 分布式存储系统中改进的一致性哈希算法[J].计算机技术与发展,2016,26(07):24.
　WANG Kang[],LI Dong-jing[],CHEN Hai-guang[]. An Improved Consistent Hashing Algorithm in Distributed Storage System[J].,2016,26(06):24.
[12]孙杜靖,李玲娟. 面向Redis的数据序列化算法研究[J].计算机技术与发展,2017,27(05):77.
　SUN Du-jing,LI Ling-juan. Investigation on Data Serialization Algorithm for Redis[J].,2017,27(06):77.
[13]孙杜靖,李玲娟,马可. 面向流数据的DPFP-Stream算法的设计与实现[J].计算机技术与发展,2017,27(07):29.
　SUN Du-jing,LI Ling-juan,MA Ke. Realization and Implementation of Distributed Parallel Mining of Frequent Patterns for Data Streams[J].,2017,27(06):29.

更新日期/Last Update: 2017-07-28