Spark成长之路(6)-Correlation

spark ml
源码

spark准备彻底支持DataFrame特性，所以重新了ml的api，原先的以RDD为基础的api都放在了mllib中，但是都是维护阶段，推荐使用ml下的api。

相关性

有2种相关性，皮尔森积矩相关系数和斯皮尔曼等级相关，具体原理请自行搜索，主要是判断两个向量的关联性。

样例

import org.apache.spark.ml.linalg.{Matrix, Vectors}import org.apache.spark.ml.stat.Correlationimport org.apache.spark.sql.{Row, SparkSession}object CorrelationExample {  def main(args: Array[String]): Unit = {    val spark = SparkSession.builder.appName("CorrelationExample").getOrCreate()    spark.sparkContext.setLogLevel("WARN")    val data = spark.sparkContext.makeRDD(Seq(      Vectors.sparse(4, Seq((0, 1.0), (3, -2.0))),      Vectors.dense(4.0, 5.0, 0.0, 3.0),      Vectors.dense(6.0, 7.0, 0.0, 8.0),      Vectors.sparse(4, Seq((0, 9.0), (3, 1.0)))    ))    import spark.implicits._    val df = data.map(Tuple1.apply).toDF("features")    val Row(coeff1: Matrix) = Correlation.corr(df, "features").head    println("Pearson correlation matrix:\n" + coeff1.toString)    val Row(coeff2: Matrix) = Correlation.corr(df, "features", "spearman").head    println("Spearman correlation matrix:\n" + coeff2.toString)  }}

执行结果

Pearson correlation matrix:1.0                   0.055641488407465814  NaN  0.4004714203168137  0.055641488407465814  1.0                   NaN  0.9135958615342522  NaN                   NaN                   1.0  NaN                 0.4004714203168137    0.9135958615342522    NaN  1.0                 Spearman correlation matrix:1.0                  0.10540925533894532  NaN  0.40000000000000174  0.10540925533894532  1.0                  NaN  0.9486832980505141   NaN                  NaN                  1.0  NaN                  0.40000000000000174  0.9486832980505141   NaN  1.0  

每一行都有四个数，代表当前第几个向量与Seq中的4个向量的相关性，比如皮尔森的第一行结果1.0 0.055641488407465814 NaN 0.4004714203168137与自己的相关性是1.0，与第二个相关性为0.055641488407465814,与第三个无法计算相关性，与第四个相关性0.055641488407465814。