[1]徐 达,曾 乐,王英杰.“天镜”全流程指标计算功能优化[J].计算机技术与发展,2023,33(07):20-26.[doi:10. 3969 / j. issn. 1673-629X. 2023. 07. 003]
　XU Da,ZENG Le,WANG Ying-jie.Optimization of Calculation Function of “ The Mirror” Whole Process Index[J].,2023,33(07):20-26.[doi:10. 3969 / j. issn. 1673-629X. 2023. 07. 003]
点击复制

“天镜”全流程指标计算功能优化()
分享到：

《计算机技术与发展》[ISSN:1006-6977/CN:61-1281/TN]

卷:
33
期数:
2023年07期
页码:
20-26
栏目:
大数据与云计算
出版日期:
2023-07-10

文章信息/Info

Title:
Optimization of Calculation Function of “ The Mirror” Whole Process Index
文章编号:
1673-629X(2023)07-0020-07
作者:
徐 达; 曾 乐; 王英杰
国家气象信息中心,北京 100081
Author(s):
XU Da; ZENG Le; WANG Ying-jie
National Meteorological Information Center,Beijing 100081,China
关键词:
气象; “ 天镜” ; Spark 计算; 全流程; 大数据
Keywords:
meteorology; “ The Mirror” ; Spark calculation; whole process; big data
分类号:
TP319
DOI:
10. 3969 / j. issn. 1673-629X. 2023. 07. 003
摘要:
气象综合业务实时监控系统-“天镜”为全国气象部门针对基础观测数据和产品在收集、分发、入库、同步各个环节提供数据全流程监视服务。 “ 天镜”系统中气象资料全流程的到报率和及时率指标作为对全国气象站上行资料的考核依据,是一线值班人员重点关注对象。 为保障“ 天镜”系统的可靠性和高效性,解决目前“ 天镜” 系统全流程指标计算慢的问题,基于 Spark 大数据计算引擎优化了全流程计算策略。 文中将原本复杂的计算任务按气象资料类型和是否考核进行拆解,并重构了 Spark 任务生成的模板。 结果表明,在相同的物理条件下,将单个庞大的计算任务拆解成多个子任务并行处理可以有效地提升 Spark 集群的计算效率,优化“ 天镜冶 系统中地面区域站气象资料的实时展示效果。
Abstract:
The meteorological integrated business real - time monitoring system," The Mirror" , provides the whole process monitoringservice for the national meteorological departments in the collection,distribution,storage,and synchronization of basic observation dataand products. The indicators of the arrival rate and timeliness rate are served as the assessment basis for the national meteorologicalstations’ uplink data,which are focused by operators on duty. To ensure the reliability and efficiency of the " The Mirror" system andsolve the problem of the calculation speed,the whole process calculation strategy is optimized based on Spark big data calculation engine.In this paper, the original complex calculation task is disassembled according to the type of meteorological data and whether it isevaluated. What’ s more,we reconstruct the Spark task template. The result shows that under the same physical conditions,disassemblinga single huge computing task into multiple subtasks for parallel processing can improve the computing efficiency of Spark cluster andoptimize the real-time display effect of meteorological data of ground regional stations in " The Mirror" system.

相似文献/References:

[1]严卿 苗放 叶成明.GIS实时分析及展示功能在网络访问中的实现[J].计算机技术与发展,2010,(12):247.
　YAN Qing,MIAO Fang,YE Cheng-ming.Implementation of Real-Time Analysis and Visualization Based on GIS in Network Access[J].,2010,(07):247.
[2]聂元丁,孙婧.基于LDM的气象TIGGE系统接收时效研究优化[J].计算机技术与发展,2014,24(11):91.
　NIE Yuan-ding,SUN Jing. Optimization of Receiving Efficiency of Meteorological TIGGE System Based on LDM[J].,2014,24(07):91.
[3]梁宝龙,崔学林,谢寒生,等.多源气象数据实时推送系统的设计与实现[J].计算机技术与发展,2018,28(08):139.[doi:10.3969/ j. issn.1673-629X.2018.08.029]
　LIANG Bao-long,CUI Xue-lin,XIE Han-sheng,et al.Design and Implementation of a Real-time Push System for Multi-source Meteorological Data[J].,2018,28(07):139.[doi:10.3969/ j. issn.1673-629X.2018.08.029]
[4]聂元丁.基于云计算的国家级气象资源池设计与建设[J].计算机技术与发展,2018,28(12):132.[doi:10.3969/j.issn.1673-629X.2018.12.028]
　NIEYuan-ding.Design and Construction of National Meteorological Resource Pool Based on Cloud Computing[J].,2018,28(07):132.[doi:10.3969/j.issn.1673-629X.2018.12.028]
[5]莫云音,李 勋,叶彩荣,等.海南省公共气象服务产品库研究与实现[J].计算机技术与发展,2019,29(06):176.[doi:10. 3969 / j. issn. 1673-629X. 2019. 06. 036]
　MO Yun-yin,LI Xun,YE Cai-rong,et al.Research and Implementation of Hainan Public Meteorological Service Product Library[J].,2019,29(07):176.[doi:10. 3969 / j. issn. 1673-629X. 2019. 06. 036]
[6]莫云音,李 勋,陈 亮,等.基于中间件的系统对接技术的研究与实现[J].计算机技术与发展,2019,29(07):170.[doi:10. 3969 / j. issn. 1673-629X. 2019. 07. 034]
　MO Yun-yin,LI Xun,CHEN Liang,et al.Research and Implementation of Docking Technology Based on Middleware[J].,2019,29(07):170.[doi:10. 3969 / j. issn. 1673-629X. 2019. 07. 034]
[7]郭 聪,马 强,张子剑.基于隐写技术的气象数据安全保护研究[J].计算机技术与发展,2019,29(08):119.[doi:10. 3969 / j. issn. 1673-629X. 2019. 08. 023]
　GUO Cong,MA Qiang,ZHANG Zi-jian.Research on Invisible Encryption Algorithm of Meteorological Data Based on Digital Watermarking[J].,2019,29(07):119.[doi:10. 3969 / j. issn. 1673-629X. 2019. 08. 023]
[8]莫云音,吴盛洪,陈 亮,等.气象信息发布监控系统应用研究与实现[J].计算机技术与发展,2019,29(08):147.[doi:10. 3969 / j. issn. 1673-629X. 2019. 08. 028]
　MO Yun-yin,WU Sheng-hong,CHEN Liang,et al.Application Research and Implementation of Information Dissemination and Monitoring System[J].,2019,29(07):147.[doi:10. 3969 / j. issn. 1673-629X. 2019. 08. 028]
[9]卫建国,李新庆,马 宁,等.宁夏气象综合数据库应用技术框架[J].计算机技术与发展,2019,29(08):167.[doi:10. 3969 / j. issn. 1673-629X. 2019. 08. 032]
　WEI Jian-guo,LI Xin-qing,MA Ning,et al.Ningxia Meteorological Comprehensive Database Application Technology Framework[J].,2019,29(07):167.[doi:10. 3969 / j. issn. 1673-629X. 2019. 08. 032]
[10]莫云音,吴盛洪,李超宁,等.气象产品制作发布与监控平台的设计与实现[J].计算机技术与发展,2020,30(06):186.[doi:10. 3969 / j. issn. 1673-629X. 2020. 06. 036]
　MO Yun-yin,WU Sheng-hong,LI Chao-ning,et al.Design and Implementation of Meteorological Products Production,Publication and Monitoring Platform[J].,2020,30(07):186.[doi:10. 3969 / j. issn. 1673-629X. 2020. 06. 036]

更新日期/Last Update: 2023-07-10