基于Hadoop的海量车牌图像处理优化技术-《计算机技术与发展》

文章信息/Info

Title:: A Processing Optimization Technique for Massive License Plate Images Based on Hadoop

Author(s):: HOU Xiang-ning; Engineering &Technical College of Chengdu University of Technology,Leshan 614007,China

摘要:: Hadoop 集群下每个小文件均占据一个 Block,一方面存储海量元数据信息消耗了大量的 NameNode 内存,另一方面,Hadoop 为每个小文件单独启动一个 Map 任务,大量的时间花费在启动和关闭 Map 任务上,从而严重降低了 MapReduce 的执行速率。对此,在详细分析已有解决方案的基础上,采用 CFIF 将多个小文件分片打包到大分片中,给每个大分片只启动一个 Map 任务来执行,通过减少启动 Map 任务的数量,提高了处理海量小文件时的效率。通过设计 Hadoop 图像接口类,继承并实现 CFIF 抽象类,最终完成了对海量图像小文件的处理。与常规 HDFS、HAR 和 MapFile 方案在 NameNode 内存空间和运行效率方面进行了对比,结果表明,CFIF 在 NameNode 内存占用率和运行效率方面,都有很好的表现。

Abstract:: Under the Hadoop cluster,each small file occupies a block. On the one hand,to store massive metadata information consumes a lot of NameNode memory;on the other hand,Hadoop starts a Map task for each small file,spending a lot of time on startup and shutdown Map tasks,which severely reduces the execution speed of the MapReduce. In view of this,on the basis of analysis of several existing solutions,we use CFIF abstract class to package multiple small files into a big split,for each big split only start a Map task to perform. By reducing the number of Map tasks,we improve the efficiency when dealing with massive small files. Through designing the Hadoop image interface class,we inherit and implement CFIF abstract class for final completion of the processing of large image small files. The comparison between CFIF and conventional HDFS,HAR and MapFile solutions in the NameNode memory usage rate and operating efficiency shows that the CFIF performs well.