sparl ML使用TFIDF

来源：互联网发布：linux中vi保存退出编辑：程序博客网时间：2024/05/21 11:28

TFIDF将文本向量化,用于后期的特征使用

1.创建DF

// $example on$val sentenceData = sqlContext.createDataFrame(Seq(  (0, "Hi I heard about Spark"),  (0, "I wish Java could use case classes"),  (1, "Logistic regression models are neat"))).toDF("label", "sentence")

JSON格式:

{"label":0,"sentence":"I wish Java could use case classes"}
{"label":1,"sentence":"Logistic regression models are neat"}
{"label":0,"sentence":"Hi I heard about Spark"}

2.由于我们需要处理的是term词条,所以先tokenizer

val tokenizer = new Tokenizer().setInputCol("sentence").setOutputCol("words")val wordsData = tokenizer.transform(sentenceData)

3.接着求TF即词条的频率

val hashingTF = new HashingTF()  .setInputCol("words").setOutputCol("rawFeatures").setNumFeatures(20)val featurizedData = hashingTF.transform(wordsData)

格式:

scala> featurizedData.select("rawFeatures").show
+--------------------+
| rawFeatures|
+--------------------+
|(20,[5,6,9],[2.0,...|
|(20,[3,5,12,14,18...|
|(20,[5,12,14,18],...|
+--------------------+

格式2:
scala> res36.take(5).foreach(println)
[(20,[5,6,9],[2.0,1.0,2.0])]
[(20,[3,5,12,14,18],[2.0,2.0,1.0,1.0,1.0])]
[(20,[5,12,14,18],[1.0,2.0,1.0,1.0])]

4.获取iDF

val idf = new IDF().setInputCol("rawFeatures").setOutputCol("features")val idfModel = idf.fit(featurizedData)

5.得到TF-iDF

val rescaledData = idfModel.transform(featurizedData)rescaledData.select("features", "label").take(3).foreach(println)
 
scala>rescaledData.select("features", "label").take(3).foreach(println)[(20,[5,6,9],[0.0,0.6931471805599453,1.3862943611198906]),0][(20,[3,5,12,14,18],[1.3862943611198906,0.0,0.28768207245178085,0.28768207245178085,0.28768207245178085]),0][(20,[5,12,14,18],[0.0,0.5753641449035617,0.28768207245178085,0.28768207245178085]),1]

0 0