softmax 回归原理及python实现

来源：互联网发布：兰蔻小黑瓶眼膜霜知乎编辑：程序博客网时间：2024/04/29 00:15

softmax回归python实现程序

1、Logistic回归：

[(x (1), y (1)), (x (2), y (2)), . . ., (x (m), y (m))]

y \in {0, 1}

h θ (x) = 1 1 + e x p ( - θ \cdot x )

似然函数
$L = \prod h θ (x (i)) y (i) (1 - h θ (x (i))) 1 - y (i)$
对数似然函数为：
$l o g L = \sum i = 1 m (y (i) l o g (h θ (x (i)) + (1 - y (i)) l o g (1 - h θ (x (i))))$

代价函数
$J (θ) = - 1 m \sum i = 1 m (y (i) l o g (h θ (x (i)) + (1 - y (i)) l o g (1 - h θ (x (i))))$
代价函数的偏导数：
$\partial J ( θ ) \partial θ j = - 1 m (\sum i = 1 m (y (i) - h θ (x (i))) x (i))$

θ j : = : = θ j - α \partial J ( θ ) \partial θ j θ j + α m (\sum i = 1 m (y (i) - h θ (x (i))) x (i) j)

对于逻辑回归，L2正则化之后，损失函数为：
$J (θ) = - 1 m (\sum i = 1 m (y (i) l o g (h θ (x (i))) + (1 - y (i)) l o g (1 - h θ (x (i)))) + λ 2 m \sum j = 1 n θ 2$
θ迭代更新公式为
$θ j : = θ j + α m (\sum i = 1 m (y (i) - h θ (x (i)) x (i) j) - α λ m θ j (j = 1, . . ., n)$
θj为每列x对应的参数，为一个数

[(x (1), y (1)), (x (2), y (2)), . . ., (x (m), y (m))] y \in {0, 1, . . ., k}

x(m)表示x的一行数据，可能包含多列，是一个向量

h θ (x (i)) = ⎡ ⎣ ⎢ ⎢ ⎢ ⎢ ⎢ p (y (1) = 1 | x (i); θ) p (y (1) = 2 | x (i); θ) ⋮ p (y (1) = k | x (i); θ) ⎤ ⎦ ⎥ ⎥ ⎥ ⎥ ⎥ = 1 \sum k j = 1 e x p ( θ T j x ( i ) ) ⎡ ⎣ ⎢ ⎢ ⎢ ⎢ ⎢ e x p (θ T 1 x (i)) e x p (θ T 2 x (i)) ⋮ e x p (θ T k x (i)) ⎤ ⎦ ⎥ ⎥ ⎥ ⎥ ⎥

p 1 l = e x p ( θ T l x ( 1 ) ) \sum k j = 1 e x p ( θ T j x ( 1 ) ), l = 1, 2, \dots, k

P = ⎡ ⎣ ⎢ ⎢ ⎢ ⎢ ⎢ p 11 p 21 ⋮ p m 1 p 12 p 22 ⋮ p m 2 \dots \dots ⋮ \dots p 1 k p 2 k ⋮ p m k ⎤ ⎦ ⎥ ⎥ ⎥ ⎥ ⎥

θ为一个k×(n+1)的矩阵

$θ = ⎡ ⎣ ⎢ ⎢ ⎢ ⎢ ⎢ θ T 1 θ T 2 ⋮ θ T k ⎤ ⎦ ⎥ ⎥ ⎥ ⎥ ⎥$

J (θ) = - 1 m ⎡ ⎣ \sum i = 1 m \sum j = 1 k 1 {y (i) = j} l o g e x p ( θ T j x ( i ) ) \sum k l = 1 e x p ( θ T l x ( i ) ) ⎤ ⎦

损失函数的矩阵形式为：
$J θ = - 1 m s u m (Y l o g P)$
对Y相应的解释
$y = ⎡ ⎣ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ 1100 k ⎤ ⎦ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥$
对应Y为
$Y = ⎡ ⎣ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ 0011011000 \dots \dots \dots \dots \dots 00001 ⎤ ⎦ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥$

y=k 则第k列数字对应1，其余列为0.

▽ θ j J (θ) = = = = - 1 m ⎡ ⎣ \sum i = 1 m \sum j = 1 k 1 {y (i) = j} \sum k l = 1 e x p ( θ T l x ( i ) ) e x p ( θ T j x ( i ) ) (e x p ( θ T j x ( i ) ) \sum k l = 1 e x p ( θ T l x ( i ) ))' ⎤ ⎦ - 1 m ⎡ ⎣ \sum i = 1 m \sum k l = 1 e x p ( θ T l x ( i ) ) e x p ( θ T j x ( i ) ) 1 { y ( i ) = j } e x p ( θ T j x ( i ) ) \sum k l = 1 e x p ( θ T l x ( i ) ) - e x p ( θ T j x ( i ) ) e x p ( θ T j x ( i ) ) ( \sum k l = 1 e x p ( θ T l x ( i ) ) ) 2 x (i) ⎤ ⎦ - 1 m ⎡ ⎣ \sum i = 1 m ⎛ ⎝ 1 {y (i) = j} - e x p ( θ T j x ( i ) ) \sum k l = 1 e x p ( θ T l x ( i ) ) ⎞ ⎠ x (i) ⎤ ⎦ - 1 m [\sum i = 1 m (1 {y (i) = j} - p (y (i) = j | x (i); θ)) x (i)]

(Y−P)维度为m×k,X为m×(n+1)

损失函数：
$J (θ) = - 1 m ⎡ ⎣ \sum i = 1 m \sum j = 1 k 1 {y (i) = j} l o g e x p ( θ T j x ( i ) ) \sum k l = 1 e x p ( θ T l x ( i ) ) ⎤ ⎦ + λ 2 \sum j = 1 k \sum b = 0 n θ 2 j b$
损失函数关于θj的偏导为：

▽ θ j J (θ) = - 1 m [\sum i = 1 m (1 {y (i) = j} - p (y (i) = j | x (i); θ)) x (i)] + λ θ j

θ j : = θ j - α ▽ θ j J (θ) j = 1, 2, . . ., k

θj为一个向量

矩阵更新：

θ : = θ - ▽ θ J (θ)

θ为一个矩阵

0 0