Backpropagation neural network

One conviction underlying the book is that it’s better to obtain a solid understanding of the core principles of neural networks and deep learning rather than a hazy understanding of a long laundry list of ideas.

ωljk to denote kth neuron in the (l-1)th to jth neuron in the l layer.

zlj=∑k(ωljkal−1k+blj)

namely,
Zl=WlAl−1+Bl

and append with a activation function

Al=σ(Zl)=σ(WlAl−1+Bl)

we need to establish a loss function and then optimize it.

C=12n∑x(y(x)−aL(x))2

and we write the quadratic cost for matrix form.

C=0.5∗||y−aL||2

the Hadamard product

s⊙t

Optimize

δlj the error on jth neuron on layer l.

δlj=∂C∂zlj

and for the last layer L:

• BP1
  
  δLj=∂C∂zLj=∂C∂aLjσ′(zLj)

namely

δL=▿C⊙σ′(zL)

• BP2
  
  δl=(ωl+1)Tδl+1⊙σ′(zl)

Proof:
1.

δl=∂C∂zl

2.

δl+1=∂C∂zl+1

so we get

δlj=∂C∂zl+1k∂zl+1k∂zlj=δl+1k∂(∑iωl+1kiali)∂zlj

and

∂(∑iωlkiali)∂zlj=∂(∑iωl+1kiσ(zli))∂zlj=ωl+1kjσ,(zlk)

we get

δlj=δl+1kωl+1kjσ′(zlk)

we write it in matrix form:

δl=1K(ωl+1)Tδl+1⊙σ′(zl)

after we get the ωl, we can use it to update the ∂C∂ω and ∂C∂b :

∂C∂ωljk=∂C∂zlj∂zlj∂ωljk=δljalk

and

∂C∂blj=∂C∂zlk∂zlk∂blk=δlk

the formula to update (ω) in n+1 times :

ω′=ω−λ∂C∂ω

b′=b−λ∂C∂b

thus we get everything.