Why and When Perceptron Halts?

来源：互联网发布：wer机器人编程编辑：程序博客网时间：2024/04/28 03:17

Pereceptron Learning Algorithm (PLA) is a binary classifier which can partition the linear separable points into two classes.

Based on the Perceptron Convergence Theorem, we have:

For any finite set of linearly separable labeled examples, the PLA will halt after a finite number of iterations.

But why and when perceptron halts?

Next, we will prove the Perceptron Convergence Theorem step by step.

Notations:

$w^{n}$ : the weight of $n^{th}$ step

$\left (x^{n}, y_{0}^{n} \right )$ : the example point used at $n^{th}$ step

$w^{*}$ : the perfect weight corresponding to the target function, which means $y_{0}^{\mu}w^{*T}x^{\mu}>0,\forall \mu$

$\theta (n)$ : the angle between $w^{*} \ and \ w^{nT}$

$c(n) \equiv cos(\theta) = \frac{w^{nT}w^{*}}{\left \| w^{n} \right \| \left \| w^{*} \right \|}$ : the cos value of angle between $w^{*} \ and \ w^{nT}$

$\delta^{\mu} = \frac{y_0^{\mu}w^{*T}x^{\mu}}{\left \| w^{*} \right \|}$ : margin, i.e. the Euclidean distance of the point $x^{\mu}$ from the plane $w^{*}$ , where $\delta^{\mu}$ is strictly positive since all points are classified correctly.

$\delta^{*}=\underset{\mu}{min}\ \delta^{\mu}$ : the minimal margin relative to the separation hyperplane $w^{*}$ .

Assume at the $n^{th}$ step, $y_0^{n}(w^{n-1T}x^{n})<0$ , then the weight $w^{n}$ is updated by $w^{n}=w^{n-1} + \eta y_0^{n}x^{n}$ .

So we have $w^{n} - w^{n-1} \equiv \Delta w^{n} = \eta y_0^{n}x^{n}$ , and $w^{n-1T}\Delta w^{n} <0$ .

Then the numerator of $c(n)$ is:

$w^{nT}w^{*} = w^{n-1T}w^{*} + \Delta w^{nT}w^{*}= w^{n-1T}w^{*} + \eta y_0^{n}x^{nT}w^{*} = w^{n-1T}w^{*} + \eta y_0^{n}w^{*T}x^{n} = w^{n-1T}w^{*} + \eta \delta ^{n}\left \| w^{*} \right \|\geq w^{n-1T}w^{*} + \eta \delta^{*}\left \| w^{*} \right \|$

After applying the above inequality above n times, starting from $w^{0}$ , to get $w^{nT}w^{*}\geq w^{0T}w^{*} + n\eta\delta^{*}\left \|w^{*} \right \|$ (here we get the numerator of $c(n)$ )

If n is large enough, then we have $W^{nT}W^{*}\geq n\eta\delta^{*}\left \|W^{*} \right \|$

Consider the denominator of $c(n)$ ,

$\left \| w^{n} \right \|^{2} = \left | w^{n-1} + \Delta w^{n} \right \|^{2} = \left \| w^{n-1} \right \|^{2} + \eta^{2}\left \| x^{n} \right \|^{2} + 2\left ( w^{n-1T}\Delta w^{n} \right ) < \left \| w^{n-1} \right \|^{2} + \eta^{2}\left \| x^{n} \right \|^{2} \leq \left \| w^{n-1} \right \|^{2} + \eta^{2}D^{2}$

where $D\equiv max_{\mu}\left \| x^{\mu} \right \|$

Apply the above inequality n times, we get

$\left \| w^{n} \right \|^{2} \leq \left \| w^{0} \right \|^{2} + n\eta^{2}D^{2}$

if n is large enough, then we get $\left \| w^{n} \right \| \leq \sqrt{n}\eta D$ (here we get the denominator of $c(n)$ )

Based on the inequality of both numerator and denominator of $c(n)$ , we get

$c(n)\geq \frac{\eta n \delta^{*}\left \| w^{*} \right \|}{\sqrt{n}\eta D\left \| w^{*} \right \|} = \frac{\delta^{*}}{D}\sqrt{n}$

We also know $c(n) \equiv cos(\theta) = \frac{w^{nT}w^{*}}{\left \| w^{n} \right \| \left \| w^{*} \right \|} \leq 1$ , so $\frac{\delta^{*}}{D}\sqrt{n}\leq 1$ and $n \leq \frac{D^{2}}{\delta^{*2}}$

Now we get the maximum step is less than $\frac{D^{2}}{\delta^{*2}}$

0 0