[1]樊春美,朱建生,单杏花,等.基于 Flink 实时计算的自动化流控制算法[J].计算机技术与发展,2020,30(08):66-72.[doi:10. 3969 / j. issn. 1673-629X. 2020. 08. 011]
　FAN Chun-mei,ZHU Jian-sheng,SHAN Xing-hua,et al.Automatic Flow Control Algorithm Based on Flink Real-time Computation[J].,2020,30(08):66-72.[doi:10. 3969 / j. issn. 1673-629X. 2020. 08. 011]
点击复制

基于 Flink 实时计算的自动化流控制算法()
分享到：

《计算机技术与发展》[ISSN:1006-6977/CN:61-1281/TN]

卷:
30
期数:
2020年08期
页码:
66-72
栏目:
智能、算法、系统工程
出版日期:
2020-08-10

文章信息/Info

Title:
Automatic Flow Control Algorithm Based on Flink Real-time Computation
文章编号:
1673-629X(2020)08-0066-07
作者:
樊春美; 朱建生; 单杏花; 杨立鹏; 李 雯
中国铁道科学研究院,北京 100081
Author(s):
FAN Chun-mei; ZHU Jian-sheng; SHAN Xing-hua; YANG Li-peng; LI Wen
China Academy of Railway Sciences,Beijing 100081,China
关键词:
Flink; 流处理; Spark; 大数据; 分布式
Keywords:
Flink; stream processing; Spark; big data; distributed
分类号:
TP301
DOI:
10. 3969 / j. issn. 1673-629X. 2020. 08. 011
摘要:
随着现在各种业务系统的复杂多样化,数据分析的实效性要求也变得越来越高,过去的离线分析很多已经不适用于当前的生产需要,针对于大数据的实时分析变得越来越重要。 以当前热门的 Flink 流处理架构为解析平台,构建了分布式实时采集解析流数据处理架构,针对于不同的数据流,只需要更新配置就可以实现业务数据的解析,极大地减少了代码开发量。 为了实现该架构的配置文件更新,重点分析了当前分布式解析架构实时更新配置文件时存在的问题,提出了通过流控制的方法来更改 Flink 的解析逻辑。 流控制的方法能够灵活地实时改变代码的解析逻辑,减少程序重启更新的次数,提高了应用效率。 通过完成同样的日志解析入库对比了是否使用流控制算法的效果,实验结果表明加入自动流控制算法的解析架构用更少的时间完成解析结构逻辑的开发和程序部署,并且可以大大地减少延迟入库的日志量,从而最大程度地保证了流的实时性。
Abstract:
With the complexity and diversification of various business systems, the requirement of effectiveness of data analysis is becoming higher and higher.? Offline analysis in the past is no longer suitable for current production needs,and real-time analysis for big data is becoming more and? ?more important. Taking the current popular Flink stream processing architecture as the parsing platform,a distributed and real-time processing architecture of collecting and parsing data stream is constructed. For different data streams,business data is analyzed by updating the configur-ation file for the architecture,which will reduce the amount of code development. In order to realize the configuration file update of this architecture, the problems existing in the current distributed parsing architecture when updating the configuration file in real time are analyzed emphatically,and then the parsing logic of Flink is proposed by flow control method. The method of flow control can flexibly change the parsing logic of the code in real time, reduce the number of program restarting and updating,and improve the efficiency of the application. By completing the same log parsing and storing,the effect of whether to use flow control algorithm is compared. The experiment shows that the analytical framework with automatic flow control algorithm takes less time to complete parsing logic structure of the development and application deployment and greatly reduces the delay of log volume,thus ensuring the real-time performance of the flow to the greatest extent.

相似文献/References:

[1]姜红玉,汪 朋,封 雷.基于流式计算的实时用户画像系统研究[J].计算机技术与发展,2020,30(07):186.[doi:10. 3969 / j. issn. 1673-629X. 2020. 07. 039]
　JIANG Hong-yu,WANG Peng,FENG Lei.Research on Real-time User Profile System Based on Stream Computing[J].,2020,30(08):186.[doi:10. 3969 / j. issn. 1673-629X. 2020. 07. 039]
[2]贾淑娟,王 菁.以用户为中心的流数据处理应用构造方法[J].计算机技术与发展,2020,30(12):45.[doi:10. 3969 / j. issn. 1673-629X. 2020. 12. 008]
　JIA Shu-juan,WANG Jing.Construction Method of User-centered Stream Data Processing Application[J].,2020,30(08):45.[doi:10. 3969 / j. issn. 1673-629X. 2020. 12. 008]

更新日期/Last Update: 2020-08-10