[1]王义武,杨余旺,于天鹏,等.基于 Spark 平台的 K-means 算法的设计与优化[J].计算机技术与发展,2019,29(03):72-76.[doi:10.3969/ j. issn.1673-629X.2019.03.015]
　WANG Yi-wu,YANG Yu-wang,YU Tian-peng,et al.Design and Optimization of K-means Algorithm Based on Spark Platform[J].,2019,29(03):72-76.[doi:10.3969/ j. issn.1673-629X.2019.03.015]
点击复制

基于 Spark 平台的 K-means 算法的设计与优化()
分享到：

《计算机技术与发展》[ISSN:1006-6977/CN:61-1281/TN]

卷:
29
期数:
2019年03期
页码:
72-76
栏目:
智能、算法、系统工程
出版日期:
2019-03-10

文章信息/Info

Title:
Design and Optimization of K-means Algorithm Based on Spark Platform
文章编号:
1673-629X(2019)03-0072-05
作者:
王义武1 ; 杨余旺1 ; 于天鹏2 ; 沈兴鑫1 ; 李猛坤3
1. 南京理工大学 计算机科学与工程学院,江苏 南京 210000;2. 304 兵器厂,山西 长治 046000;3. 清华大学 经管学院,北京 100000
Author(s):
WANG Yi-wu1 ; YANG Yu-wang1 ; YU Tian-peng2 ; SHEN Xing-xin1 ; LI Meng-kun3
1. School of Computer Science and Engineering,Nanjing University of Science and Technology, Nanjing 210000,China;2. 304Weapon Factory,Changzhi 046000,China;3. School of Economics and Management,Tsinghua University,Beijing 100000,China
关键词:
聚类; 聚类中心; K-means; 最大最小距离算法; 非加权组平均法
Keywords:
clustering; clustering center; K-means; maximum and minimum distance algorithm; unweighted pair group method with arithmetic mean
分类号:
TP301
DOI:
10.3969/ j. issn.1673-629X.2019.03.015
摘要:
聚类中心需要手动设置是 K-means 算法最大的问题,而通常情况是并不能确定现实中数据的分类情况。 为了解决这一问题,提出了一种新的 OCC K-means 算法。 不同于传统算法以随机选择的方式产生聚类中心,该算法进行必要的预处理,利用 UPGMA 和最大最小距离算法对数据点进行筛选,得到可以反映数据分布特征的点,并作为初始的聚类中心,以提高聚类的精度。 从两次的实验结果可以对比出,在不同的数据集上,改进算法在衡量聚类效果的准确率、召回率、 F -测量值上的表现要优于传统 K-means 算法。 这是因为 OCC 算法选择的中心点来自于不同的且数据密集的区域,并在筛选的过程中排除了噪声数据、边缘数据对实验的干扰;同时为了契合大数据发展潮流,使用 Scala 语言在 Spark 平台进行了并行化实现,提高了算法处理海量数据的能力,并通过实验指标验证了算法具有良好的并行化能力。
Abstract:
The clustering center needs to be set manually is the biggest problem of K-means algorithm,and it is usually impossible to determine the classification of data in reality. In order to solve the problem,we propose a new OCC K-means algorithm. Different from the traditional algorithm,which generates the clustering center in the way of random selection,this algorithm carries out necessary preprocessing,and uses UPGMA and maximum and minimum distance algorithm to screen data points for the ones that can reflect data distribution characteristics as the initial clustering center,so as to improve the accuracy of clustering. From the two experimental results,it can be found that in different data sets,the improved algorithm is better in the measurement of clustering accuracy,recall, F -measurement than the traditional K-means algorithm. This is because the center point selected by OCC algorithm comes from different and data-intensive areas,and noise data and edge data interference to the experiment are excluded in the process of screening. At the same time,in order to conform to the trend of big data development,the parallelization implementation is carried out on Spark platform with Scala language, which improves the ability of the algorithm to deal with massive data,and the better parallelization of the algorithm is verified by experimental indexes.

相似文献/References:

[1]蒋璐璐 王适 王宝成 李慧敏 李鑫慧.一种改进的标记分水岭遥感图像分割方法[J].计算机技术与发展,2010,(01):36.
　JIANG Lu-lu,WANG Shi,WANG Bao-cheng,et al.Segmentation of Remote Sensing Image Based on an Improved Labeling Watershed Algorithm[J].,2010,(03):36.
[2]张甜 罗眉 孟晓红 赵宗涛.一种基于状态特征的航天发射故障诊断技术[J].计算机技术与发展,2010,(01):93.
　ZHANG Tian,LUO Mei,MENG Xiao-hong,et al.A Technology in Fault Diagnosis of Spaceflight Launch Based on State Character[J].,2010,(03):93.
[3]王会颖 章义刚.求解聚类问题的改进人工鱼群算法[J].计算机技术与发展,2010,(03):84.
　WANG Hui-ying,ZHANG Yi-gang.An Improved Artificial Fish- Swarm Algorithm of Solving Clustering Analysis Problem[J].,2010,(03):84.
[4]赵敏 倪志伟 刘斌.K—means与朴素贝叶斯在商务智能中的应用[J].计算机技术与发展,2010,(04):179.
　ZHAO Min,NI Zhi-wei,LIU Bin.Application Research of K - Means Clustering and Naive Bayesian Algorithm in Business Intelligence[J].,2010,(03):179.
[5]吴楠 胡学钢.基于聚类分区的序列模式挖掘算法研究[J].计算机技术与发展,2010,(06):109.
　WU Nan,HU Xue-gang.Research on Clustering Partition-Based Approach of Sequential Pattern Mining[J].,2010,(03):109.
[6]耿波 仲红 徐杰 闫娜娜.用关联分析法对负荷预测结果进行二次处理[J].计算机技术与发展,2008,(04):171.
　GENG Bo,ZHONG Hong,XU Jie,et al.Using Correlation Analysis to Treat Load Forecasting Results[J].,2008,(03):171.
[7]游芳 姜建国 张坤.基于二维属性的高维数据聚类算法研究[J].计算机技术与发展,2009,(05):111.
　YOU Fang,JIANG Jian-guo,ZHANG Kun.Cluster- Algorithm Studies Based on Two- Dimensional Attribute Higher - Dimension Data[J].,2009,(03):111.
[8]刘淑英 程国建 彭方.人工神经生长细胞结构网络在医疗诊断的应用[J].计算机技术与发展,2009,(05):231.
　LIU Shu-ying,CHENG Guo-jian,PENG Fang.Applications of Growing Cell Structures of Artificial Neural Network for Medical Diagnosis[J].,2009,(03):231.
[9]范新 沈闻 丁泉勋 沈洁.基于正例和未标文档的半监督分类研究[J].计算机技术与发展,2009,(06):58.
　FAN Xin,SHEN Wen,DING Quan-xun,et al.Research on Semi- Supervised Classification Based on Positive and Unlabeled Text Document[J].,2009,(03):58.
[10]王园园 倪志伟 赵裕啸 伍章俊.基于决策树的模糊聚类评价算法及其应用[J].计算机技术与发展,2009,(09):232.
　WANG Yuan-yuan,NI Zhi-wei,ZHAO Yu-xiao,et al.Fuzzy Clustering Evaluation Algorithm Based on Decision Tree and Application[J].,2009,(03):232.
[11]朱云贺 张春海 张博.基于数据分段的K-means的优化研究[J].计算机技术与发展,2010,(11):130.
　ZHU Yun-he,ZHANG Chun-hai,ZHANG Bo.Optimizing Research on K-means Based on Data Partition[J].,2010,(03):130.

更新日期/Last Update: 2019-03-10