[1]冯 冼,方 昆,文立恒,等.气象数据分区处理算法与策略研究[J].计算机技术与发展,2023,33(08):214-220.[doi:10. 3969 / j. issn. 1673-629X. 2023. 08. 031]
　FENG Xian,FANG Kun,WEN Li-heng,et al.Research on Algorithm and Strategy of Meteorological Data Partition Processing[J].,2023,33(08):214-220.[doi:10. 3969 / j. issn. 1673-629X. 2023. 08. 031]

点击复制

气象数据分区处理算法与策略研究()

分享到：

《计算机技术与发展》[ISSN:1006-6977/CN:61-1281/TN]

卷:

33

期数:

2023年08期

页码:

214-220

栏目:

新型计算应用系统

出版日期:

2023-08-10

文章信息/Info

Title:

Research on Algorithm and Strategy of Meteorological Data Partition Processing

文章编号:

1673-629X(2023)08-0214-07

作者:

冯 冼¹; 2 ; 方 昆¹ ; 文立恒¹ ; 朱宏武¹

1. 湖南省气象信息中心,湖南 长沙 410118;

2. 气象防灾减灾湖南省重点实验室,湖南 长沙 410118

Author(s):

FENG Xian1; 2 ; FANG Kun1 ; WEN Li-heng1 ; ZHU Hong-wu1

1. Hunan Meteorological Information Center,Changsha 410118,China;

2. Hunan Key Laboratory of Meteorological Disaster Prevention and Reduction,Changsha 410118,China

关键词:

气象数据; 关键特征值; 权重算法; 并发处理; 分区策略

Keywords:

meteorological data; key eigenvalue; weight algorithm; concurrent processing; partition policy

分类号:

TP311

DOI:

10. 3969 / j. issn. 1673-629X. 2023. 08. 031

摘要:

为解决海量气象数据并发处理的技术难点,提出了气象数据综合权重算法以及 Kafka 并发处理性能最优策略。 选取湖南省 2020 年 6 月气象数据作为实验数

据集,提取对数据处理系统性能影响最显著的关键特征值,基于熵权法计算关键特征值在流转和处理时消耗基础资源的综合权重,并将其作为气象数据分区处理

的客观依据。 参照气象大数据云平台架构,设计以 Kafka 为核心的数据处理模型,通过实验分别得出气象数据在 Producer 和 Consumer 端最优 Partition、Thread 配置策略,从而提升并发处理能力。 实验结果表明:对实验数据集进行分区并配置最优策略后,在有限基础资源支撑条件下,消息写入速度从 0. 69 MB / s 提升至 37. 44 MB / s,消息读取速度从 15. 65 MB / s 提升至 67. 34 MB / s。 该算法和策略已应用在气象卫星遥感数据处理业务,有效解决了海量卫星遥感数据传输处理过程出现消息阻塞的现象,在各类数据处理系统设计中具有较强的参考价值。

Abstract:

To solve the key challenges in the concurrent processing of massive meteorological data,a new comprehensive weight algorithmof meteorological?

data and the optimal strategy of Kafka concurrent processing performance are developed. The meteorological data ofHunan Province in June 2020?

was selected as the experimental data set,and the key eigenvalue that had the most significant influence onthe performance of the data processing?

system were extracted. Based on the entropy weight method,the comprehensive weight of thebasic resources consumed by the key eigenvalue during?

the flow and processing was calculated,which was used as the objective basis forthe partition processing of meteorological data. With reference to the meteorological big data cloud platform architecture, a dataprocessing model with Kafka as the core is designed,and the optimal Partition and Thread configuration strategies for meteorological datain the Producer and Consumer are obtained through experiments,so as to improve the concurrent processing capability. The experimentalresults show that the message writing speed is improved from 0. 69 MB / s to 37. 44 MB / s,and the reading speed is improved from 15. 65MB / s to 67. 34 MB / s after the optimal strategy is configured for the processing of the experimental data set on limited basic resources.The algorithm and strategy have been applied to the meteorological satellite remote sensing data processing business,effectively solvingthe message blocking phenomenon in the transmission and processing of massive satellite remote sensing data,which have strong referencevalue in the design of various data processing systems.

常用功能