Cost Function的原理及实现(Python, matlab)

成本函数(Cost Function)

J(θ0,θ1)=12m∑i=1m(hθ(x(i))−y(i))2

m : Number of training examples.
x : input.
y : output.
Parameters: θ0, θ1.

以下为MATLAB实现方式：

function J = computeCost(X, y, theta)    %COMPUTECOST Compute cost for linear regression    %   J = COMPUTECOST(X, y, theta) computes the cost of using theta as the parameter for linear regression to fit the data points in X and y    m = length(y); % number of training examples    J = 0;    J = sum((X * theta - y) .^2) / 2 / m;end

以下为Python实现方式：

def computeCost(X, y, theta):    """Cost Functions    Parameters    ----------    X : np.ndarray, like (49 * 2)    y : np.ndarray, like (49 * 1)    theta : np.ndarray, like (2 * 1)    Returns    -------    J : float, cost    """    y = np.transpose(y)    J = sum((np.dot(X, theta) - y.reshape(len(y),1)) **2) / 2.0 / len(y)    # np.dot(A, B) 矩阵乘积，A * B 矩阵点乘    # pow(A, 2) 多次乘积，**n 点次方    return J

注意：np中点乘和矩阵乘积，点次方与多次乘积的区别，与MATLAB不同。

梯度下降法(Gradient Descent)

repeatuntilconvergence{θj:=θj−α∂J(θ0,θ1)∂θj(j=0,1)}

α : learning rate.
∂ : 偏微分

以下为MATLAB实现：

function [theta, J_history] = gradientDescent(X, y, theta, alpha, num_iters)    %GRADIENTDESCENT Performs gradient descent to learn theta    %   theta = GRADIENTDESCENT(X, y, theta, alpha, num_iters) updates theta by     %   taking num_iters gradient steps with learning rate alpha    m = length(y); % number of training examples    J_history = zeros(num_iters, 1);    for iter = 1:num_iters        theta = theta -  X' * (X * theta - y) * (alpha / m);          J_history(iter) = computeCost(X, y, theta);    endend

以下为Python实现：

def gradientDescent(X, y, theta, alpha, num_iters):    """Gradient Descent for (Multivariate) Linear Regression    Parameters    ----------    X : np.ndarray, like (49 * 2)    y : np.ndarray, like (49 * 1)    theta : np.ndarray, like (49 * 1)    alpha : learning rate    num_iters : number of iter    Returns    -------    tuple(J_history, theta)    J_history : np.ndarray, like (num_iters, 1)    theta : theta of convergence, like (2 * 1)    """    J_history = np.zeros((num_iters, 1))    for n_iter in range(num_iters):        theta = theta - np.dot(X.T, np.dot(X, theta) - y.reshape(len(y),1)) *alpha / len(y)        J_history[n_iter, 0] = computeCost(X, y, theta)    return J_history, theta

线性回归中的梯度下降公式

θ0:=θ0−α1m∑i=1m(hθ(x(i))−y(i))

θ1:=θ1−α1m∑i=1m(hθ(x(i))−y(i))x(i)1

同理可得：

θj:=θj−α1m∑i=1m(hθ(x(i))−y(i))x(i)j

在（多元）线性回归中，由于在代码中直接使用矩阵进行运算，因此代码同上。