[1]孙杜靖,李玲娟,马可. 面向流数据的DPFP-Stream算法的设计与实现[J].计算机技术与发展,2017,27(07):29-33.
　SUN Du-jing,LI Ling-juan,MA Ke. Realization and Implementation of Distributed Parallel Mining of Frequent Patterns for Data Streams[J].,2017,27(07):29-33.
点击复制

 面向流数据的DPFP-Stream算法的设计与实现()
分享到：

《计算机技术与发展》[ISSN:1006-6977/CN:61-1281/TN]

卷:
27
期数:
2017年07期
页码:
29-33
栏目:
智能、算法、系统工程
出版日期:
2017-07-10

文章信息/Info

Title:
 Realization and Implementation of Distributed Parallel Mining of Frequent Patterns for Data Streams
文章编号:
1673-629X（2017)07-0029-05
作者:
 孙杜靖; 李玲娟; 马可
 南京邮电大学 计算机学院
Author(s):
 SUN Du-jing；LI Ling-juan；MA Ke
关键词:
 ; DPFP-stream ; MapReduce; Storm; Redis
Keywords:
 ; DPFP-stream; MapReduce; Storm ; Redis
分类号:
TP301.6
文献标志码:
A
摘要:
 从海量数据中发现频繁模式一直是数据挖掘研究的热点,在零售市场数据分析、网络监控、网络使用挖掘和股票市场的预测等领域中也有着广泛的应用.尽管在过去的十年里,很多学者提出了许多基于静态数据集的频繁模式挖掘算法,而由于流数据持续、无限、有序而高速产生的特性,在流数据中隐藏的数据知识很可能随着时间的推移而产生变化,因而基于流数据的频繁模式挖掘应不同于以往基于静态数据集的频繁模式挖掘算法.为了更好地分析在线流数据,基于不同的时间粒度从流数据中抽取频繁模式并且监控频繁模式的变化,基于高效的FP-tree结构,借助倾斜时间窗口和MapReduce的思想,提出了针对数据流的频繁模式挖掘算法DPFP-stream.并将该算法在Storm平台上实现,算法数据源采用Kafka,并将中间结果存入内存数据库Redis中.通过大量的实验表明,该算法从高速的数据流中发现频繁模式的效率很高且性能稳定.在海量数据实时计算中,采用该算法,不仅能应对高速的数据流,而且能监控不同时间粒度的频繁模式的变化过程.
Abstract:
 Finding frequent patterns in a continuous stream of transactions is critical for many applications such as retail market data analysis,network monitoring,web usage mining and stock market prediction.Even though numerous frequent pattern mining algorithms have been developed over the past decade,new solutions for handling stream data are still required due to the continuous,unbounded and ordered sequence of data elements generated at a rapid rate in a data stream.As a result,the knowledge embedded in a data stream is more likely to be changed as time goes by.Therefore,extracting frequent patterns from data at multiple time granularities and monitoring the gradual changes of frequent patterns can enhance the analysis of online data streams.Based on efficient FP-tree structure,according to the ideas of tilted-time windows and MapReduce,the DPFP-stream is proposed and implemented in Storm.The data resource of it uses Kafka and stores middle result into Redis.Extensive experiment shows that the algorithm proposed is highly efficient in terms of time complexity when finding recent frequent patterns from a high-speed data stream.With the application of the algorithm in real-time computing,it can not only process high speed stream,but also monitor the change of frequent patterns with tilted-time windows.

相似文献/References:

[1]张志宏,吴庆波,邵立松,等.基于飞腾平台TOE协议栈的设计与实现[J].计算机技术与发展,2014,24(07):1.
　ZHANG Zhi-hong,WU Qing-bo,SHAO Li-song,et al. Design and Implementation of TCP/IP Offload Engine Protocol Stack Based on FT Platform[J].,2014,24(07):1.
[2]梁文快,李毅. 改进的基因表达算法对航班优化排序问题研究[J].计算机技术与发展,2014,24(07):5.
　LIANG Wen-kuai,LI Yi. Research on Optimization of Flight Scheduling Problem Based on Improved Gene Expression Algorithm[J].,2014,24(07):5.
[3]黄静,王枫,谢志新,等. EAST文档管理系统的设计与实现[J].计算机技术与发展,2014,24(07):13.
　HUANG Jing,WANG Feng,XIE Zhi-xin,et al. Design and Implementation of EAST Document Management System[J].,2014,24(07):13.
[4]侯善江[],张代远[][][]. 基于样条权函数神经网络P2P流量识别方法[J].计算机技术与发展,2014,24(07):21.
　HOU Shan-jiang[],ZHANG Dai-yuan[][][]. P2P Traffic Identification Based on Spline Weight Function Neural Network[J].,2014,24(07):21.
[5]李璨,耿国华,李康,等. 一种基于三维模型的文物碎片线图生成方法[J].计算机技术与发展,2014,24(07):25.
　LI Can,GENG Guo-hua,LI Kang,et al. A Method of Obtaining Cultural Debris’ s Line Chart Based on Three-dimensional Model[J].,2014,24(07):25.
[6]翁鹤,皮德常. 混沌RBF神经网络异常检测算法[J].计算机技术与发展,2014,24(07):29.
　WENG He,PI De-chang. Chaotic RBF Neural Network Anomaly Detection Algorithm[J].,2014,24(07):29.
[7]刘茜[],荆晓远[],李文倩[],等. 基于流形学习的正交稀疏保留投影[J].计算机技术与发展,2014,24(07):34.
　LIU Qian[],JING Xiao-yuan[,LI Wen-qian[],et al. Orthogonal Sparsity Preserving Projections Based on Manifold Learning[J].,2014,24(07):34.
[8]尚福华,李想,巩淼. 基于模糊框架-产生式知识表示及推理研究[J].计算机技术与发展,2014,24(07):38.
　SHANG Fu-hua,LI Xiang,GONG Miao. Research on Knowledge Representation and Inference Based on Fuzzy Framework-production[J].,2014,24(07):38.
[9]叶偲,李良福,肖樟树. 一种去除运动目标重影的图像镶嵌方法研究[J].计算机技术与发展,2014,24(07):43.
　YE Si,LI Liang-fu,XIAO Zhang-shu. Research of an Image Mosaic Method for Removing Ghost of Moving Targets[J].,2014,24(07):43.
[10]余松平[][],蔡志平[],吴建进[],等. GSM-R信令监测选择录音系统设计与实现[J].计算机技术与发展,2014,24(07):47.
　YU Song-ping[][],CAI Zhi-ping[] WU Jian-jin[],GU Feng-zhi[]. Design and Implementation of an Optional Voice Recording System Based on GSM-R Signaling Monitoring[J].,2014,24(07):47.

更新日期/Last Update: 2017-08-22