[1]颜晓莲,章 刚,邱晓红.Kafka 中改进型 Partition 过载优化算法[J].计算机技术与发展,2020,30(12):88-91.[doi:10. 3969 / j. issn. 1673-629X. 2020. 12. 016]
　YAN Xiao-lian,ZHANG Gang,QIU Xiao-hong.Improved Partition Overload Optimization Algorithm in Kafka[J].,2020,30(12):88-91.[doi:10. 3969 / j. issn. 1673-629X. 2020. 12. 016]
点击复制

Kafka 中改进型 Partition 过载优化算法()
分享到：

《计算机技术与发展》[ISSN:1006-6977/CN:61-1281/TN]

卷:
30
期数:
2020年12期
页码:
88-91
栏目:
智能、算法、系统工程
出版日期:
2020-12-10

文章信息/Info

Title:
Improved Partition Overload Optimization Algorithm in Kafka
文章编号:
1673-629X(2020)12-0088-04
作者:
颜晓莲1; 章 刚2; 邱晓红1
1. 江西理工大学 软件工程学院(南昌),江西 南昌 330013; 2. 江西北大科技园,江西 南昌 330013
Author(s):
YAN Xiao-lian1; ZHANG Gang2; QIU Xiao-hong1
1. School of Software Engineering (Nanchang),Jiangxi University of Science and Technology,Nanchang 330013,China; 2. Jiangxi Peking University Science Park,Nanchang 330013,China
关键词:
分布式消息系统; Kafka; Partition 过载问题; 协同管理; Broker 性能
Keywords:
distributed message system; Kafka; Partition overload problem; collaborative management; Broker performance
分类号:
TP393
DOI:
10. 3969 / j. issn. 1673-629X. 2020. 12. 016
摘要:
Kafka 作为一种发布-订阅机制的高吞吐量分布式消息系统,广受业界关注。 随着应用场景垂直化、多样化,Kafka 现有的技术体系面临挑战。Partition 过载问题(POP)指消息分发、消息订阅等操作引起 Partition 过度服务,并影响到物理载体 Broker 的性能。 该问题是由 Kafka 中 Partition 文件配置管理的被动、僵化及孤立等不足所导致。 针对此,提出一种改进型 Partition 过载优化算法(IPOOA)。 该算法通过即时服务耗量、Partition 相似度和配置文件自动修改相结合,实现消息分发预测以及消息分发与文件配置管理协同,从而可有效缓解 Partition 过载问题出现。实验从 Kafka 集群 CPU 使用率、Kafka 服务延时率、Kafka 系统收敛延时比等几个方面验证了算法的有效性及合理性。
Abstract:
Kafka,as a high-throughput distributed message system with publish subscribe mechanism,is widely concerned by the industry. Existing technology system of Kafka is facing challenges by the vertical and diversified application scenarios.Partition overload problem(POP) refers to the problem that operations such as message distribution and message subscription cause excessive service of partition which affects the performance of physical carrier broker. This problem is caused by the passivity, rigidity and isolation of partition file configuration management in Kafka. Therefore,we propose an imp-roved optimization algorithm of partition overload (IPOOA). This algorithm combines the instant service consumption, partition similarity and automatic modification of configuration file to realize the prediction of message distribution and the coordination of message distribution and file configuration management,which can effectively alleviate the problem of partition overload. The experiment verifies the validity and rationality of the algorithm from several aspects,such as the CPU utilization rate of Kafka cluster,Kafka service delay rate,Kafka system convergence delay rate,etc

相似文献/References:

[1]王建荣,华连生,唐怀瓯,等.数值预报产品分布式处理与存储系统设计[J].计算机技术与发展,2018,28(02):167.[doi:10．3969/j．issn．1673－629X．2018．02．036]
　WANG Jian-rong,HUA Lian-sheng,TANG Huai-ou,et al.Design of Distributed NWP Data Processing and Storage System[J].,2018,28(12):167.[doi:10．3969/j．issn．1673－629X．2018．02．036]
[2]王梓,梁正和,吴莹莹.基于 Kafka、Disruptor 技术对传统 ETL 的改进[J].计算机技术与发展,2018,28(11):26.[doi:10.3969/ j. issn.1673-629X.2018.11.006]
　WANG Zi,LIANG Zheng-he,WU Ying-ying.Improvement of Traditional ETL Based on Kafka and Disruptor Technology[J].,2018,28(12):26.[doi:10.3969/ j. issn.1673-629X.2018.11.006]
[3]李恩洲,况立群*,张 元,等.智慧供热大数据监测平台研究及应用[J].计算机技术与发展,2021,31(11):176.[doi:10. 3969 / j. issn. 1673-629X. 2021. 11. 029]
　LI En-zhou,KUANG Li-qun*,ZHANG Yuan,et al.Research and Application of Big Data Monitoring Platform for Intelligent Heating[J].,2021,31(12):176.[doi:10. 3969 / j. issn. 1673-629X. 2021. 11. 029]

更新日期/Last Update: 2020-12-10