协同过滤算法+相似度度量+交替最小二乘法

来源：互联网发布：数据打点是什么意思编辑：程序博客网时间：2024/05/01 15:42

一.协同过滤算法（Collaborative Filtering）

1.简介

协同过滤算法：是一种基于群体用户或者物品的经典推荐算法。分两种：

(1).通过考察具有相同爱好的用户对相同物品的评分标准进行计算。

(2).考察具有相同特质的物品从而推荐给选择了某件物品的用户。

A和B是“志同道合”的基友（相似度很高），将A喜欢的物品推荐给B是合理的

在无先验知识的前提下，根据A所喜欢物品的相似性，将相似物品推荐给A，是合理的

2.问题

基于用户，数据量大，对于通用物品往往优先推荐，对于热点物品不够准

基于物品，数据量相对小，而推荐同类物品存在用户已持有不再需要的问题

二.相似度度量

1.基于欧几里得距离的相似度计算

(1)欧几里得距离(Euclidean distance)：表示三维空间中两个点的真实距离。是一种基于用户之间直线距离的计算方式。

不同的物品或用户对应不同的坐标，特定目标定位为坐标原点。

欧几里得距离公式：

相似度：1/(d+1)

2.基于余弦角度的相似度计算

特定目标作为坐标上的点，但不是原点。夹角的大小来反映目标之间的相似度。

3.欧几里得相似度与余弦相似度的比较

distance and similarity

欧几里得相似度注重目标之间的差异。

余弦相似度更多是对目标从方向趋势上区分。

4.余弦相似度实战

/**  * Created by Administrator on 2017/4/17.  */import org.apache.spark.{SparkConf, SparkContext}import scala.collection.mutable.Mapobject testVector {  val conf = new SparkConf()    .setMaster("local")    .setAppName("testVector");  val sc = new SparkContext(conf);  //将内存数据读入Spark系统中  val users = sc.parallelize(Array("aaa","bbb","ccc","ddd","eee"));  val films = sc.parallelize(Array("smzdm","ylxb","znh","nhsc","fcwr"));  //使用一个source嵌套map作为姓名，电影名和分值的评价  var source = Map[String,Map[String,Int]]();  //设置一个用于存放电影分的map  val filmSource = Map[String,Int]();  //设置电影评分  def getSource(): Map[String,Map[String,Int]] = {    val user1FilmSource = Map("smzdm" -> 2,"ylxb" -> 3,"znh" -> 1,"nhsc" -> 0,"fcwr" -> 1);    val user2FilmSource = Map("smzdm" -> 1,"ylxb" -> 2,"znh" -> 2,"nhsc" -> 1,"fcwr" -> 4);    val user3FilmSource = Map("smzdm" -> 2,"ylxb" -> 1,"znh" -> 0,"nhsc" -> 1,"fcwr" -> 4);    val user4FilmSource = Map("smzdm" -> 3,"ylxb" -> 2,"znh" -> 0,"nhsc" -> 5,"fcwr" -> 3);    val user5FilmSource = Map("smzdm" -> 5,"ylxb" -> 3,"znh" -> 1,"nhsc" -> 1,"fcwr" -> 2);    source += ("aaa" -> user1FilmSource);    source += ("bbb" -> user2FilmSource);    source += ("ccc" -> user3FilmSource);    source += ("ddd" -> user4FilmSource);    source += ("eee" -> user5FilmSource);    return source;  }  //两两计算分值，采用余弦相似性  def getCollaborateSource(user1:String,user2:String):Double = {    val user1FilmSource = source.get(user1).get.values.toVector;//获得第一个用户的评分    val user2FilmSource = source.get(user2).get.values.toVector;//获得第二个用户的评分    val member = user1FilmSource.zip(user2FilmSource).map(d => d._1 * d._2).reduce(_ + _).toDouble;//对分子部分进行计算    //求出分母第一个变量值    val temp1 = math.sqrt(user1FilmSource.map(num => {      math.pow(num,2);    }).reduce(_ + _));//进行叠加    //求出分母第二个变量值    val temp2 = math.sqrt(user2FilmSource.map(num => {      math.pow(num,2);    }).reduce(_ + _));//进行叠加    val denominator = temp1 * temp2;    return member / denominator;  }  def main(args: Array[String]): Unit = {    getSource();//初始化分数    val name = "bbb";    users.foreach(user =>{      println(name + "相对于" + user + "的相似性分数是：" + getCollaborateSource(name,user));    })  }}

bbb相对于aaa的相似性分数是：0.7089175569585667
bbb相对于bbb的相似性分数是：1.0000000000000002
bbb相对于ccc的相似性分数是：0.8780541105074453
bbb相对于ddd的相似性分数是：0.6865554812287477
bbb相对于eee的相似性分数是：0.6821910402406466

三.交替最小二乘法（ALS）--最常用的逼近计算的一种算法。

1.最小二乘法

(1)概念：最小二乘法多项式曲线拟合，根据给定的m个点,并不要求这条曲线精确地经过这些点，而是曲线y=f(x)的近似曲线y= φ(x)。

(2)常见的曲线拟合方法：

1.使偏差绝对值之和最小

2.使偏差绝对值最大的最小

3.使偏差平方和最小

按偏差平方和最小的原则选取拟合曲线，并且采取二项式方程为拟合曲线的方法,称为最小二乘法。

(3)推导过程

1. 设拟合多项式为：

2. 各点到这条曲线的距离之和，即偏差平方和如下：

3. 为了求得符合条件的a值，对等式右边求ai偏导数，因而我们得到了：

.......

4. 将等式左边进行一下化简，然后应该可以得到下面的等式：

.......

5. 把这些等式表示成矩阵的形式，就可以得到下面的矩阵：

6. 将这个范德蒙得矩阵化简后可得到:

7. 也就是说X*B=Y，那么

，便得到了系数矩阵B，同时，我们也就得到了拟合曲线。

(4)java实现

    public class TheleastSquareMethod {
   //定义系数
    private double[] x;
    private double[] y;
    private double[] weight;
    private int n;
    private double[] coefficient;

    /**
     * Constructor method.
     *
     * @param x
     *            Array of x
     * @param y
     *            Array of y
     * @param n
     *            The order of polynomial
     */
    public TheleastSquareMethod(double[] x, double[] y, int n) {
        if (x == null || y == null || x.length < 2 || x.length != y.length
                || n < 2) {
            throw new IllegalArgumentException(
                    "IllegalArgumentException occurred.");
        }
        this.x = x;
        this.y = y;
        this.n = n;
        weight = new double[x.length];
        for (int i = 0; i < x.length; i++) {
            weight[i] = 1;
        }
        compute();
    }

    /**
     * Constructor method.
     *
     * @param x
     *            Array of x
     * @param y
     *            Array of y
     * @param weight
     *            Array of weight
     * @param n
     *            The order of polynomial
     */
    public TheleastSquareMethod(double[] x, double[] y, double[] weight, int n) {
        if (x == null || y == null || weight == null || x.length < 2
                || x.length != y.length || x.length != weight.length || n < 2) {
            throw new IllegalArgumentException(
                    "IllegalArgumentException occurred.");
        }
        this.x = x;
        this.y = y;
        this.n = n;
        this.weight = weight;
        compute();
    }

    /**
     * Get coefficient of polynomial.
     *
     * @return coefficient of polynomial
     */
    public double[] getCoefficient() {
        return coefficient;
    }

    /**
     * Used to calculate value by given x.
     *
     * @param x
     *            x
     * @return y
     */
    public double fit(double x) {
        if (coefficient == null) {
            return 0;
        }
        double sum = 0;
        for (int i = 0; i < coefficient.length; i++) {
            sum += Math.pow(x, i) * coefficient[i];
        }
        return sum;
    }

    /**
     * Use Newton's method to solve equation.
     *
     * @param y
     *            y
     * @return x
     */
    public double solve(double y) {
        return solve(y, 1.0d);
    }

    /**
     * Use Newton's method to solve equation.
     *
     * @param y
     *            y
     * @param startX
     *            The start point of x
     * @return x
     */
    public double solve(double y, double startX) {
        final double EPS = 0.0000001d;
        if (coefficient == null) {
            return 0;
        }
        double x1 = 0.0d;
        double x2 = startX;
        do {
            x1 = x2;
            x2 = x1 - (fit(x1) - y) / calcReciprocal(x1);
        } while (Math.abs((x1 - x2)) > EPS);
        return x2;
    }

    /*
     * Calculate the reciprocal of x.
     *
     * @param x x
     *
     * @return the reciprocal of x
     */
    private double calcReciprocal(double x) {
        if (coefficient == null) {
            return 0;
        }
        double sum = 0;
        for (int i = 1; i < coefficient.length; i++) {
            sum += i * Math.pow(x, i - 1) * coefficient[i];
        }
        return sum;
    }

    /*
     * This method is used to calculate each elements of augmented matrix.
     */
    private void compute() {
        if (x == null || y == null || x.length <= 1 || x.length != y.length
                || x.length < n || n < 2) {
            return;
        }
        double[] s = new double[(n - 1) * 2 + 1];
        for (int i = 0; i < s.length; i++) {
            for (int j = 0; j < x.length; j++) {
                s[i] += Math.pow(x[j], i) * weight[j];
            }
        }
        double[] b = new double[n];
        for (int i = 0; i < b.length; i++) {
            for (int j = 0; j < x.length; j++) {
                b[i] += Math.pow(x[j], i) * y[j] * weight[j];
            }
        }
        double[][] a = new double[n][n];
        for (int i = 0; i < n; i++) {
            for (int j = 0; j < n; j++) {
                a[i][j] = s[i + j];
            }
        }

        // Now we need to calculate each coefficients of augmented matrix
        coefficient = calcLinearEquation(a, b);
    }

    /*
     * Calculate linear equation.
     *
     * The matrix equation is like this: Ax=B
     *
     * @param a two-dimensional array
     *
     * @param b one-dimensional array
     *
     * @return x, one-dimensional array
     */
    private double[] calcLinearEquation(double[][] a, double[] b) {
        if (a == null || b == null || a.length == 0 || a.length != b.length) {
            return null;
        }
        for (double[] x : a) {
            if (x == null || x.length != a.length)
                return null;
        }

        int len = a.length - 1;
        double[] result = new double[a.length];

        if (len == 0) {
            result[0] = b[0] / a[0][0];
            return result;
        }

        double[][] aa = new double[len][len];
        double[] bb = new double[len];
        int posx = -1, posy = -1;
        for (int i = 0; i <= len; i++) {
            for (int j = 0; j <= len; j++)
                if (a[i][j] != 0.0d) {
                    posy = j;
                    break;
                }
            if (posy != -1) {
                posx = i;
                break;
            }
        }
        if (posx == -1) {
            return null;
        }

        int count = 0;
        for (int i = 0; i <= len; i++) {
            if (i == posx) {
                continue;
            }
            bb[count] = b[i] * a[posx][posy] - b[posx] * a[i][posy];
            int count2 = 0;
            for (int j = 0; j <= len; j++) {
                if (j == posy) {
                    continue;
                }
                aa[count][count2] = a[i][j] * a[posx][posy] - a[posx][j]
                        * a[i][posy];
                count2++;
            }
            count++;
        }

        // Calculate sub linear equation
        double[] result2 = calcLinearEquation(aa, bb);

        // After sub linear calculation, calculate the current coefficient
        double sum = b[posx];
        count = 0;
        for (int i = 0; i <= len; i++) {
            if (i == posy) {
                continue;
            }
            sum -= a[posx][i] * result2[count];
            result[i] = result2[count];
            count++;
        }
        result[posy] = sum / a[posx][posy];
        return result;
    }

    public static void main(String[] args) {
       TheleastSquareMethod eastSquareMethod = new TheleastSquareMethod(
                new double[] { 0.5, 1.0, 1.5, 2.0, 2.5, 3.0 }, new double[] {
                        1.75, 2.45, 3.81, 4.8, 7.0, 8.6 }, 3);
        /*double[] coefficients = eastSquareMethod.getCoefficient();
        for (double c : coefficients) {
            System.out.println(c);
        }*/

        System.out.println(eastSquareMethod.fit(4));

        TheleastSquareMethod eastSquareMethod2 = new TheleastSquareMethod(
                new double[] { 0.5, 1.0, 1.5, 2.0, 2.5, 3.0 }, new double[] {
                        1.75, 2.45, 3.81, 4.8, 7.0, 8.6 }, 2);
        System.out.println(eastSquareMethod2.solve(100));

    }
}
三.基于ALS算法的协同过滤

(1)数据集

1 11 21 12 31 13 11 14 02 11 12 12 22 13 22 14 12 15 43 11 23 12 33 13 13 14 03 15 14 11 14 12 24 13 24 14 14 15 45 11 15 12 25 13 25 14 15 15 4

(2)建立ALS数据模型

MLlib中ALS算法固定格式：case class Rating(user: Int,product: Int,rating: Double)

ALS.tran码源：

def train(           ratings: RDD[Rating];           rank: Int;           iterations: Int;           lambda: Double;           blocks: Int;           seed: Long;         ): MatrixFactorizationModel = {  new ALS(blocks,rank,iterations,lambda,false,1.0,seed).run(ratings)

(3)基于ALS算法的协同过滤推荐

import org.apache.spark.mllib.recommendation.{ALS, Rating}import org.apache.spark.{SparkConf, SparkContext}object testVector {  def main(args: Array[String]): Unit = {    var conf = new SparkConf().setMaster("local").setAppName("testVector");//设置环境变量    val sc = new SparkContext(conf);//实例化环境    val data = sc.textFile("kimi.txt");//设置数据集    val ratings = data.map(_.split(' ')match{//处理数据      case Array(user,item,rate) => //将数据集转化        Rating(user.toInt,item.toInt,rate.toDouble);//将数据集转化成专用rating    })    val rank = 2;//设置隐藏因子    val numIterations = 2;//设置迭代次数    val model = ALS.train(ratings,rank,numIterations,0.01);//进行模型训练    var rs = model.recommendProducts(2,1);//为用户2推荐一个商品    rs.foreach(println);//打印结果  }}

/Rating(2,15,3.9713808775549495)，为用户 2 推荐一个编号 15 的商品，预测评分 3.97 与实际的 4 接近．

0 0