Spark MLlib 入门学习笔记

FPGrowth频繁项集算法

关联规则(AssociationRule)是研究不同类型的物品相互直接关联关系的规则。Apriori算法是关联规则挖掘的一个经典算法。FPGrowth算法克服了Apriori算法需要对数据集进行多次读取的弊端，只需要读取两次数据集。

支持度表示X和Y中的项在同一情况下出现的概率，公式： Support(A->B)=P(A U B)。置信度表示A出现了，B出现的概率，公式： Confidence(A->B)=P(A | B)。

运行spark自带的列子。

程序

package assimport org.apache.spark.{SparkConf, SparkContext}import org.apache.spark.mllib.fpm.FPGrowthimport org.apache.spark.rdd.RDDobject fpgrowth {  def main(args: Array[String]) {    val conf = new SparkConf().setMaster(args(0)).setAppName("fpgrowth")    val sc = new SparkContext(conf)    val data = sc.textFile(args(1))    val transactions: RDD[Array[String]] = data.map(s => s.trim.split(' '))    val fpg = new FPGrowth()      .setMinSupport(0.3)      .setNumPartitions(10)    val model = fpg.run(transactions)    model.freqItemsets.collect().foreach { itemset =>      println(itemset.items.mkString("[", ",", "]") + ", " + itemset.freq)    }    val minConfidence = 0.8    model.generateAssociationRules(minConfidence).collect().foreach { rule =>      println(        rule.antecedent.mkString("[", ",", "]")          + " => " + rule.consequent.mkString("[", ",", "]")          + ", " + rule.confidence)    }  }}

数据

r z h k pz y x w v u t ss x o n rx z y m t s q ezx z y r q t p

运行结果（太多了，省略大部分）

[z], 5[x], 4[x,z], 3[y], 3[y,x], 3[y,x,z], 3[y,z], 3[r], 3......[t,s,y] => [x], 1.0[t,s,y] => [z], 1.0[y,x,z] => [t], 1.0[y] => [x], 1.0[y] => [z], 1.0[y] => [t], 1.0[p] => [r], 1.0[p] => [z], 1.0[q,t,z] => [y], 1.0[q,t,z] => [x], 1.0[q,y] => [x], 1.0[q,y] => [z], 1.0[q,y] => [t], 1.0[t,s,x] => [y], 1.0[t,s,x] => [z], 1.0[q,t,y,z] => [x], 1.0[q,t,x,z] => [y], 1.0......