[1]于金良[],朱志祥[],梁小江[]. 基于Flume的MySQL数据自动收集系统[J].计算机技术与发展,2016,26(12):137-141.
　YU Jin-liang[],ZHU Zhi-xiang[],LIANG Xiao-jiang[]. Automatic Collection System for MySQL Data Based on Flume[J].,2016,26(12):137-141.
点击复制

 基于Flume的MySQL数据自动收集系统()
分享到：

《计算机技术与发展》[ISSN:1006-6977/CN:61-1281/TN]

卷:
26
期数:
2016年12期
页码:
137-141
栏目:
应用开发研究
出版日期:
2016-12-10

文章信息/Info

Title:
 Automatic Collection System for MySQL Data Based on Flume
文章编号:
1673-629X（2016)12-0137-05
作者:
 于金良[1]; 朱志祥[1]; 梁小江[2]
1.西安邮电大学；2.陕西省信息化工程研究院
Author(s):
 YU Jin-liang[1]; ZHU Zhi-xiang[1]; LIANG Xiao-jiang[2]
关键词:
 ; Flume; MySQL数据库; 数据收集; HBase; JDBC
Keywords:
 ; Flume; MySQL database; data collecting; HBase; JDBC
分类号:
TP274.2
文献标志码:
A
摘要:
 针对分布式系统中、不同数据中心之间的数据收集,同时解决将数据由关系型数据库交换到非关系型数据库的问题,提出一种基于Flume的MySQL数据库数据自动收集系统。为了符合现实中的生产环境,该系统采用的是一种星型拓扑结构。系统可以自动查询给定的MySQL数据库表,自动检测表中的数据更新,实现自动增量传输,并对原始数据进行封装、解析,最终将数据存储到非关系型数据库HBase中。在测试中,系统中每台机器的平均传输速度可达到1111 kb/s,系统总的平均传输速度可以达到3333 kb/s,并且保证了数据的完整性,实现了可靠高效传输的目标。
Abstract:
 For data collecting in distributed systems or between different data centers,while addressing the issue of exchange data from a relational database to non-relational databases,an automatic collecting system for MySQL data based on Flume is put forword. In order to meet the real-world production environment,this system uses a star topology. It can automatically query a given MySQL database table, automatic delection of the data updating of the table,automatic incremental transfer,packaging and parsing to the original data,finally sto-ring data into a database of HBase. In test,the average speed of transmission of every machine in the system can reach 1 111 kb/s,and the total speed of transmission can reach 3 333 kb/s,which ensure data integrity and achieve the goal of reliable and efficient transport.

相似文献/References:

[1]张志宏,吴庆波,邵立松,等.基于飞腾平台TOE协议栈的设计与实现[J].计算机技术与发展,2014,24(07):1.
　ZHANG Zhi-hong,WU Qing-bo,SHAO Li-song,et al. Design and Implementation of TCP/IP Offload Engine Protocol Stack Based on FT Platform[J].,2014,24(12):1.
[2]梁文快,李毅. 改进的基因表达算法对航班优化排序问题研究[J].计算机技术与发展,2014,24(07):5.
　LIANG Wen-kuai,LI Yi. Research on Optimization of Flight Scheduling Problem Based on Improved Gene Expression Algorithm[J].,2014,24(12):5.
[3]黄静,王枫,谢志新,等. EAST文档管理系统的设计与实现[J].计算机技术与发展,2014,24(07):13.
　HUANG Jing,WANG Feng,XIE Zhi-xin,et al. Design and Implementation of EAST Document Management System[J].,2014,24(12):13.
[4]侯善江[],张代远[][][]. 基于样条权函数神经网络P2P流量识别方法[J].计算机技术与发展,2014,24(07):21.
　HOU Shan-jiang[],ZHANG Dai-yuan[][][]. P2P Traffic Identification Based on Spline Weight Function Neural Network[J].,2014,24(12):21.
[5]李璨,耿国华,李康,等. 一种基于三维模型的文物碎片线图生成方法[J].计算机技术与发展,2014,24(07):25.
　LI Can,GENG Guo-hua,LI Kang,et al. A Method of Obtaining Cultural Debris’ s Line Chart Based on Three-dimensional Model[J].,2014,24(12):25.
[6]翁鹤,皮德常. 混沌RBF神经网络异常检测算法[J].计算机技术与发展,2014,24(07):29.
　WENG He,PI De-chang. Chaotic RBF Neural Network Anomaly Detection Algorithm[J].,2014,24(12):29.
[7]刘茜[],荆晓远[],李文倩[],等. 基于流形学习的正交稀疏保留投影[J].计算机技术与发展,2014,24(07):34.
　LIU Qian[],JING Xiao-yuan[,LI Wen-qian[],et al. Orthogonal Sparsity Preserving Projections Based on Manifold Learning[J].,2014,24(12):34.
[8]尚福华,李想,巩淼. 基于模糊框架-产生式知识表示及推理研究[J].计算机技术与发展,2014,24(07):38.
　SHANG Fu-hua,LI Xiang,GONG Miao. Research on Knowledge Representation and Inference Based on Fuzzy Framework-production[J].,2014,24(12):38.
[9]叶偲,李良福,肖樟树. 一种去除运动目标重影的图像镶嵌方法研究[J].计算机技术与发展,2014,24(07):43.
　YE Si,LI Liang-fu,XIAO Zhang-shu. Research of an Image Mosaic Method for Removing Ghost of Moving Targets[J].,2014,24(12):43.
[10]余松平[][],蔡志平[],吴建进[],等. GSM-R信令监测选择录音系统设计与实现[J].计算机技术与发展,2014,24(07):47.
　YU Song-ping[][],CAI Zhi-ping[] WU Jian-jin[],GU Feng-zhi[]. Design and Implementation of an Optional Voice Recording System Based on GSM-R Signaling Monitoring[J].,2014,24(12):47.

更新日期/Last Update: 2017-02-03