[toc]

前面没细看，这只是个 mathjax 测试 $\delta^4$

Perceptron

感知器：step function

参数的数学表示：weights和bias

矩阵

能实现各种逻辑运算

用神经网络可以实现各种逻辑计算

可以通过算法自学习

最重要的特性，通过输入更新参数(hyper arg)

Sigmoid/Logistic neurons

重要特性： 因为该函数小的输入变化产生小的输出变化。

: Δoutput is a linear function of the changes Δwj and Δb in the weights and bias.

代数上方便：指数的微分性质

The Architecture of neural network

MLP(Multiple Perceptron)

选取隐藏层没有统一的经验法则。

feedforward： 输入不受输出影响

Recurrent Neural Network：一种有限时间激发的，输入受输出影响的神经网络。

梯度下降

Cost Function: Loss function, Objective function.

quadratic cost function(mean squared error or just MSE.):

$$
\begin{eqnarray} C(w,b) \equiv
\frac{1}{2n} \sum_x | y(x) - a|^2.
\end{eqnarray}
$$

变量很多时，通过微分计算极值很麻烦的

Summing up, the way the gradient descent algorithm works is to repeatedly compute the gradient ∇C, and then to move in the opposite direction,

$$
\begin{eqnarray}
\Delta C \approx \nabla C \cdot \Delta v.
\tag{9}\end{eqnarray}
$$

$$
\begin{eqnarray}
\Delta v = -\eta \nabla C,
\tag{10}\end{eqnarray}
$$

The rule doesn't always work - several things can go wrong and prevent gradient descent from finding the global minimum of C, a point we'll return to explore in later chapters. But, in practice gradient descent often works extremely well, and in neural networks we'll find that it's a powerful way of minimizing the cost function, and so helping the net learn.

rule

$$
\begin{eqnarray}
w_k & \rightarrow & w_k' = w_k-\eta \frac{\partial C}{\partial w_k} \tag{16}\end{eqnarray}
$$

$$
\begin{eqnarray}
b_l & \rightarrow & b_l' = b_l-\eta \frac{\partial C}{\partial b_l}.
\tag{17}\end{eqnarray}
$$

stochastic gradient descent,mini-batch

$$
\begin{eqnarray}
\nabla C \approx \frac{1}{m} \sum_{j=1}^m \nabla C_{X_{j}},
\tag{19}\end{eqnarray}
$$

$$
\begin{eqnarray}
w_k & \rightarrow & w_k' = w_k-\frac{\eta}{m}
\sum_j \frac{\partial C_{X_j}}{\partial w_k} \tag{20}
\end{eqnarray}
$$

$$
\begin{eqnarray}
b_l & \rightarrow & b_l' = b_l-\frac{\eta}{m}
\sum_j \frac{\partial C_{X_j}}{\partial b_l},
\tag{21}\end{eqnarray}
$$

online learning?

In general, debugging a neural network can be challenging. This is especially true when the initial choice of hyper-parameters produces results no better than random noise.

More generally, we need to develop heuristics for choosing good hyper-parameters and a good architecture

sophisticated algorithm ≤ simple learning algorithm + good training data. 

backpropagation

At the heart of backpropagation is an expression for the partial derivative ∂C/∂w of the cost function C with respect to any weight w (or bias b) in the network.

feedforward matrix-based computation

$$
\begin{eqnarray}
a^{l} = \sigma(w^l a^{l-1}+bl).
\tag{25}\end{eqnarray}
$$

The goal of backpropagation is to compute the partial derivatives ∂C/∂w and ∂C/∂b of the cost function C with respect to any weight w or bias b in the network.

two assumptions

1. This is the case for the quadratic cost function, where the cost for a single training example is $C_x=\frac{1}{2}∥y−a^L∥2. $

   backpropagation actually lets us do is compute the partial derivatives ∂Cx/∂w and ∂Cx/∂b for a single training example.

2. The second assumption we make about the cost is that it can be written as a function of the outputs from the neural network:

Hadamard product or Schur product: element-wise multiply

反向传播的四个基础公式

定义z的变化率 $\delta^l_j$

$$
\begin{eqnarray}
\delta^l_j \equiv \frac{\partial C}{\partial z^l_j}.
\tag{29}\end{eqnarray}
$$

之所以不用 $\frac{\partial{C}}{\partial{a^l_j}}$ 只是因为计算方便。

An equation for the error in the output layer

$$
\begin{eqnarray}
\delta^L_j = \frac{\partial C}{\partial a^L_j} \sigma'(z^L_j).
\tag{BP1}\end{eqnarray}
$$

$$
\begin{eqnarray}
\delta^L = \nabla_a C \odot \sigma'(z^L).
\tag{BP1a}\end{eqnarray}
$$

if cost function is quardratic function:

$$
\begin{eqnarray}
\delta^L = (a^L-y) \odot \sigma'(z^L).
\tag{30}\end{eqnarray}
$$

An equation for the error $ \delta_l $ ** in terms of the error in the next layer** $ δ_{l+1} $

$$
\begin{eqnarray}
\delta^l = ((w^{l+1})T \delta^{l+1}) \odot \sigma'(z^l),
\tag{BP2}\end{eqnarray}
$$

By combining (BP2) with (BP1) we can compute the error δl for any layer in the network. We start by using (BP1) to compute δL, then apply Equation (BP2) to compute δL−1, then Equation (BP2) again to compute δL−2, and so on, all the way back through the network.

An equation for the rate of change of the cost with respect to any bias in the network

$$
\begin{eqnarray} \frac{\partial C}{\partial b^l_j} =
\delta^l_j.
\tag{BP3}\end{eqnarray}
$$

$$
\begin{eqnarray}
\frac{\partial C}{\partial b} = \delta,
\tag{31}\end{eqnarray}
$$

An equation for the rate of change of the cost with respect to any weight in the network

$$
\begin{eqnarray}
\frac{\partial C}{\partial w^l_{jk}} = a^{l-1}_k \delta^l_j.
\tag{BP4}\end{eqnarray}
$$

$$
\begin{eqnarray} \frac{\partial
C}{\partial w} = a_{\rm in} \delta_{\rm out},
\tag{32}\end{eqnarray}
$$

satureted: learn slowly , ∂C/∂w will also tend to be small

Summing up, we've learnt that a weight will learn slowly if either the input neuron is low-activation, or if the output neuron has saturated, i.e., is either high- or low-activation.

Summary: equations of backpropagation

Input a set of training examples

结合向量化的minibatch来更新权重和偏差。