反向传播算法简单推导笔记

1.全连接神经网络

该结构的前向传播可以写成:

$z^{(1)}=W^{(1)}x+b^{(1)}$
$a^{(1)}=\sigma (z^{(1)})$
$z^{(2)}=W^{(2)}{a}^{(1)}+b^{(2)}$
$a^{(2)}=\sigma (z^{(2)})$

2.符号约定

1. ${\delta }_i^{(x)}$:$\frac{\partial Loss}{\partial z_i^{(x)}}$ 标量.

2. ${\delta }^{(x)}$:向量,由${\delta }_1^{(x)},{\delta }_2^{(x)}...,{\delta }_n^{(x)}$组成.

3. $\odot$:表示按元素级别的乘法操作.
   $\left[\begin{array}{c}a\\ b\end{array}\right]\odot \left[\begin{array}{c}c\\ d\end{array}\right]=\left[\begin{array}{c}ac\\ bd\end{array}\right]$

   $\left[\begin{array}{cc}a& b\\ c& d\end{array}\right]\odot \left[\begin{array}{c}e\\ f\end{array}\right]=\left[\begin{array}{cc}ae& be\\ cf& df\end{array}\right]$

3.全连接反向传播公式推导

针对本博客给出来的2层,每层2个神经元的全连接神经网络模型,给出其反向传播推导过程.

现在欲求$\frac{\partial Loss}{\partial W^{(1)}},\frac{\partial Loss}{\partial W^{(2)}},\frac{\partial Loss}{\partial b^{(1)}},\frac{\partial Loss}{\partial b^{(2)}},{\delta }^{(1)},{\delta }^{(2)}$.

依据模型图并根据链式求导法则可以得到下面的等式:
欲求${\delta }^{(2)}$先考虑下面式子

${\delta }_1^{(2)}=\frac{\partial Loss}{\partial a_1^{(2)}}\ast \frac{\partial a_1^{(2)}}{\partial z_1^{(2)}}=\frac{\partial Loss}{\partial a_1^{(2)}}\ast {\sigma }^{\prime }(z_1^{(2)})$

${\delta }_2^{(2)}=\frac{\partial Loss}{\partial a_2^{(2)}}\ast \frac{\partial a_2^{(2)}}{\partial z_2^{(2)}}=\frac{\partial Loss}{\partial a_2^{(2)}}\ast {\sigma }^{\prime }(z_2^{(2)})$
那么
${\delta }^{(2)}=\left[\begin{array}{c}{\delta }_1^{(2)}\\ \\ {\delta }_2^{(2)}\end{array}\right]=\left[\begin{array}{c}\frac{\partial Loss}{\partial a_1^{(2)}}\\ \\ \frac{\partial Loss}{\partial a_2^{(2)}}\end{array}\right]\odot \left[\begin{array}{c}{\sigma }^{\prime }(z_1^{(2)})\\ \\ {\sigma }^{\prime }(z_2^{(2)})\end{array}\right]$

---

欲求$\frac{\partial Loss}{\partial W^{(2)}}$先考虑下面式子
$\frac{\partial Loss}{\partial w_{11}^{(2)}}={\delta }_1^{(2)}\ast \frac{\partial z_1^{(2)}}{\partial w_{11}^{(2)}}={\delta }_1^{(2)}\ast a_1^{(1)}$

$\frac{\partial Loss}{\partial w_{12}^{(2)}}={\delta }_1^{(2)}\ast \frac{\partial z_1^{(2)}}{\partial w_{12}^{(2)}}={\delta }_1^{(2)}\ast a_2^{(1)}$

$\frac{\partial Loss}{\partial w_{21}^{(2)}}={\delta }_2^{(2)}\ast \frac{\partial z_2^{(2)}}{\partial w_{21}^{(2)}}={\delta }_2^{(2)}\ast a_1^{(1)}$

$\frac{\partial Loss}{\partial w_{22}^{(2)}}={\delta }_2^{(2)}\ast \frac{\partial z_2^{(2)}}{\partial w_{22}^{(2)}}={\delta }_2^{(2)}\ast a_2^{(1)}$

那么
$\frac{\partial Loss}{\partial W^{(2)}}=\left[\begin{array}{cc}{\delta }_1^{(2)}\cdot a_1^{(1)}& {\delta }_1^{(2)}\cdot a_2^{(1)}\\ & \\ {\delta }_2^{(2)}\cdot a_1^{(1)}& {\delta }_2^{(2)}\cdot a_2^{(1)}\end{array}\right]=\left[\begin{array}{c}{\delta }_1^{(2)}\\ \\ {\delta }_2^{(2)}\end{array}\right]\cdot \left[\begin{array}{cc}{a}_1^{(1)}& a_2^{(1)}\end{array}\right]={\delta }^{(2)}\cdot a^{(1)T}$

---

欲求$\frac{\partial Loss}{\partial b^{(2)}}$,先考虑下面式子

$\frac{\partial Loss}{\partial b_1^{(2)}}={\delta }_1^{(2)}\ast 1$

$\frac{\partial Loss}{\partial b_2^{(2)}}={\delta }_2^{(2)}\ast 1$

那么
$\frac{\partial Loss}{\partial b^{(2)}}=\left[\begin{array}{c}{\delta }_1^{(2)}\\ \\ {\delta }_2^{(2)}\end{array}\right]\odot \left[\begin{array}{c}1\end{array}\right]={\delta }^{(2)}$

---

欲求${\delta }^{(1)}$,先考虑下面式子

${\delta }_1^{(1)}={\delta }_1^{(2)}\cdot \frac{\partial z_1^{(2)}}{\partial a_1^{(1)}}\cdot \frac{\partial a_1^{(1)}}{\partial z_1^{(1)}}+{\delta }_2^{(2)}\cdot \frac{\partial z_2^{(2)}}{\partial a_1^{(1)}}\cdot \frac{\partial a_1^{(1)}}{\partial z_1^{(1)}}={\delta }_1^{(2)}\cdot w_{11}^{(2)}\cdot {\sigma }^{\prime }(z_1^{(1)})+{\delta }_2^{(2)}\cdot w_{21}^{(2)}\cdot {\sigma }^{\prime }(z_1^{(1)})$

${\delta }_2^{(1)}={\delta }_1^{(2)}\cdot \frac{\partial z_1^{(2)}}{\partial a_2^{(1)}}\cdot \frac{\partial a_2^{(1)}}{\partial z_2^{(1)}}+{\delta }_2^{(2)}\cdot \frac{\partial z_2^{(2)}}{\partial a_2^{(1)}}\cdot \frac{\partial a_2^{(1)}}{\partial z_2^{(1)}}={\delta }_1^{(2)}\cdot w_{12}^{(2)}\cdot {\sigma }^{\prime }(z_2^{(1)})+{\delta }_2^{(2)}\cdot w_{22}^{(2)}\cdot {\sigma }^{\prime }(z_2^{(1)})$

那么

${\delta }^{(1)}=\left[\begin{array}{cc}{w}_{11}^{(2)}& w_{21}^{(2)}\\ & \\ w_{12}^{(2)}& w_{22}^{(2)}\end{array}\right]\cdot \left[\begin{array}{c}{\delta }_1^{(2)}\\ \\ {\delta }_2^{(2)}\end{array}\right]\odot \left[\begin{array}{c}{\sigma }^{\prime }(z_1^{(1)})\\ \\ {\sigma }^{\prime }(z_2^{(1)})\end{array}\right]=W^{(2)T}\cdot {\delta }^{(2)}\odot {\sigma }^{\prime }(z^{(1)})$

---

求$\frac{\partial Loss}{\partial W^{(1)}},\frac{\partial Loss}{\partial b^{(1)}}$的方法与之前求$\frac{\partial Loss}{\partial W^{(2)}},\frac{\partial Loss}{\partial b^{(2)}}$完全相同,这里就不赘述了

利用向量微积分简化推导过程

上面的推到方法我们专注于拆成标量用链式求导法则算,最后拼成矩阵相乘的形式,这样从数学上来说比较严谨,但是较为麻烦.

而实际上我们有一种更为简单(玄学 )的做法,那就是直接对向量运用链式求导法则,并且根据矩阵的维数来调整项目的位置和对项目进行转置.

欲求${\delta }^{(2)}=\frac{\partial Loss}{\partial z^{(2)}}=\frac{\partial Loss}{\partial a^{(2)}}\cdot \frac{\partial a^{(2)}}{\partial z^{(2)}}$

由于dim{δ(2)}=2∗1,dim{∂Loss∂a(2)}=2∗1dim\{\delta^{(2)}\} = 2*1,dim\{\ \frac{\partial Loss}{\partial a^{(2)}} \} = 2*1dim{δ(2)}=2∗1,dim{ ∂a(2)∂Loss​}=2∗1,根据矩阵乘法的法则,理论上应该有dim{∂a(2)∂z(2)}=1∗1dim\{\ \frac{\partial a^{(2)}}{\partial z^{(2)}} \} = 1*1dim{ ∂z(2)∂a(2)​}=1∗1.

而实际上如果是列向量($a^{(2)}$)对列向量($z^{(2)}$)进行求导,大多都是对应元素进行求导,比如$\frac{\partial a^{(2)}}{\partial z^{(2)}}=\left[\begin{array}{c}{\sigma }^{\prime }(z_1^{(2)})\\ \\ {\sigma }^{\prime }(z_2^{(2)})\end{array}\right]={\sigma }^{\prime }(z^{(2)})$,也就是说dim{∂a(2)∂z(2)}=2∗1dim\{\frac{\partial a^{(2)}}{\partial z^{(2)}} \} = 2*1dim{∂z(2)∂a(2)​}=2∗1.

那怎么办呢,$2\ast 1,2\ast 2$的矩阵不满足乘法规律,这样的话,使用$\odot$这个操作刚好就可以,所以:
${\delta }^{(2)}=\frac{\partial Loss}{\partial z^{(2)}}=\frac{\partial Loss}{\partial a^{(2)}}\cdot \frac{\partial a^{(2)}}{\partial z^{(2)}}=\frac{\partial Loss}{\partial a^{(2)}}\odot {\sigma }^{\prime }(z^{(2)})$

看到这里,明白人一定会说:你这不扯淡吗,毫无道理.

没错,你来打我呀.

我们再看一个例子:

欲求$\frac{\partial Loss}{\partial W^{(2)}}=(\frac{\partial Loss}{\partial a^{(2)}}\cdot \frac{\partial a^{(2)}}{\partial z^{(2)}})\cdot \frac{\partial z^{(2)}}{\partial W^{(2)}}={\delta }^{(2)}\cdot \frac{\partial z^{(2)}}{\partial W^{(2)}}$

注意到dim{∂Loss∂W(2)}=2∗2,dim{δ(2)}=2∗1dim\{\frac{\partial Loss}{\partial W^{(2)}}\}=2*2,dim\{\delta^{(2)}\}=2*1dim{∂W(2)∂Loss​}=2∗2,dim{δ(2)}=2∗1
那么理论上应该有dim{∂z(2)∂W(2)}=1∗2dim\{\frac{\partial z^{(2)}}{\partial W^{(2)}}\} = 1*2dim{∂W(2)∂z(2)​}=1∗2.

我们根据$z^{(2)}=W^{(2)}{a}^{(1)}+b^{(2)}$,知道dim{a(1)}=2∗1dim\{a^{(1)}\}=2*1dim{a(1)}=2∗1,所以令$\frac{\partial z^{(2)}}{\partial W^{(2)}}=a^{(1)T}$.

于是$\frac{\partial Loss}{\partial W^{(2)}}={\delta }^{(2)}\cdot a^{(1)T}$.

虽然很扯淡,但这样做跟前面用数学推导得到的公式是一样的.

我们再看一个例子:

${\delta }^{(1)}=\frac{\partial Loss}{\partial z^{(1)}}=\frac{\partial Loss}{\partial z^{(2)}}\cdot \frac{\partial z^{(2)}}{\partial a^{(1)}}\cdot \frac{\partial a^{(1)}}{\partial z^{(1)}}$

由于dim{∂Loss∂z(1)}=n∗1,dim{∂Loss∂z(2)}=m∗1,dim{∂z(2)∂a(1)}=?,dim{∂a(1)∂z(1)}=n∗1dim\{\frac{\partial Loss}{\partial z^{(1)}}\}=n*1,dim\{\frac{\partial Loss}{\partial z^{(2)}}\}=m*1,dim\{\frac{\partial z^{(2)}}{\partial a^{(1)}}\} = ?,dim\{\frac{\partial a^{(1)}}{\partial z^{(1)}}\}=n*1dim{∂z(1)∂Loss​}=n∗1,dim{∂z(2)∂Loss​}=m∗1,dim{∂a(1)∂z(2)​}=?,dim{∂z(1)∂a(1)​}=n∗1.

再根据$z^{(2)}=W^{(2)}{a}^{(1)}+b^{(2)}$,那么$\frac{\partial z^{(2)}}{\partial a^{(1)}}$一定是有$W^{(2)}$变化而来的,而dim{W(2)}=m∗ndim\{W^{(2)}\}=m*ndim{W(2)}=m∗n,所以玄学一波(调整一下顺序以及转置)应该得到:
(其中$n,m=2$)

${\delta }^{(1)}=W^{(2)T}\cdot {\delta }^{(2)}\odot {\sigma }^{\prime }(z^{(1)})$.这与上面数学推导得到的结果是一致的.

---

4.卷积层反向传播

输入层为$X$,卷积核为$K$,输出层为$Y$.

那么有 $X\otimes K=Y$.

如果$X$的宽度为$x$,$K$的宽度为$k$,$Y$的宽度为$y$.那么有$y=x-k+1$成立.

$\left[\begin{array}{ccc}{x}_{11}& x_{12}& x_{13}\\ x_{21}& x_{22}& x_{23}\\ x_{31}& x_{32}& x_{33}\end{array}\right]\otimes \left[\begin{array}{cc}{k}_{11}& k_{12}\\ x_{21}& k_{22}\end{array}\right]=\left[\begin{array}{cc}{y}_{11}& y_{12}\\ y_{21}& y_{22}\end{array}\right]$

展开之后我们可以写成:

$y_{11}=x_{11}{k}_{11}+x_{12}{k}_{12}+x_{21}{k}_{21}+x_{22}{k}_{22}$
$y_{12}=x_{12}{k}_{11}+x_{13}{k}_{12}+x_{22}{k}_{21}+x_{23}{k}_{22}$
$y_{21}=x_{21}{k}_{11}+x_{22}{k}_{12}+x_{31}{k}_{21}+x_{32}{k}_{22}$
$y_{22}=x_{22}{k}_{11}+x_{23}{k}_{12}+x_{32}{k}_{21}+x_{33}{k}_{22}$

记${\delta }_{ij}=\frac{\partial Loss}{\partial y_{ij}}=\nabla y_{ij}$

$\frac{\partial Loss}{\partial k_{11}}={\delta }_{11}\cdot x_{11}+{\delta }_{12}\cdot x_{12}+{\delta }_{21}\cdot x_{21}+{\delta }_{22}\cdot x_{22}$
$\frac{\partial Loss}{\partial k_{12}}={\delta }_{11}\cdot x_{12}+{\delta }_{12}\cdot x_{13}+{\delta }_{21}\cdot x_{22}+{\delta }_{22}\cdot x_{23}$
$\frac{\partial Loss}{\partial k_{21}}={\delta }_{11}\cdot x_{21}+{\delta }_{12}\cdot x_{22}+{\delta }_{21}\cdot x_{31}+{\delta }_{22}\cdot x_{32}$
$\frac{\partial Loss}{\partial k_{22}}={\delta }_{11}\cdot x_{22}+{\delta }_{12}\cdot x_{23}+{\delta }_{21}\cdot x_{32}+{\delta }_{22}\cdot x_{33}$

我们发现

$\left[\begin{array}{cc}\nabla k_{11}& \nabla k_{12}\\ & \\ \nabla k_{21}& \nabla k_{22}\end{array}\right]=\left[\begin{array}{ccc}{x}_{11}& x_{12}& x_{13}\\ x_{21}& x_{22}& x_{23}\\ x_{31}& x_{32}& x_{33}\end{array}\right]\otimes \left[\begin{array}{cc}\nabla y_{11}& \nabla y_{12}\\ & \\ \nabla y_{21}& \nabla y_{22}\end{array}\right]$

而这刚好也是一个卷积操作,可以写成

$\frac{\partial Loss}{\partial K}=\frac{\partial Loss}{\partial Y}\cdot \frac{\partial Y}{\partial K}=X\otimes \nabla Y$

我们写成 $\nabla K=X\otimes \nabla Y$

这说明了卷积层的梯度传播仍然是卷积操作.

那么对卷积核$K$的梯度$\nabla K$我们求出来了,下面我们省略对$X$的梯度推导步骤,直接得出公式$\nabla X=pad(\nabla Y)\otimes ro{t}_{180}(K)$

其中$ro{t}_{180}$操作表示将矩阵旋转$180$度,可以理解成先左右旋转,再上下旋转.

其中$pad$操作表示对矩阵周围进行补$0$操作,为什么需要补$0$呢,这是为了保证最后维数是匹配的.$\nabla X$的维度是$r$,$K$的唯独是$k$,那么我们前向传播时候得到的$Y$的维度是$r-k+1$.假设$pad(\nabla Y)$的维度是$?$,那么必须有$r=?-k+1$,所以$?=r+k-1$,因此$pad$操作将维度为$r-k+1$的矩阵补成维度$r+k-1$.

例如这个例子中的$pad(\nabla Y)$应该写成:

$\left[\begin{array}{cccc}0& 0& 0& 0\\ 0& \nabla y_{11}& \nabla y_{12}& 0\\ 0& \nabla y_{21}& \nabla y_{22}& 0\\ 0& 0& 0& 0\end{array}\right]$

这个例子中的$ro{t}_{180}(K)$应该写成:

$\left[\begin{array}{cc}{w}_{22}& w_{21}\\ & \\ w_{12}& w_{11}\end{array}\right]$

---

5.卷积操作的实现

卷积操作其实也可以写作矩阵乘法来做.

比如下面式子可以写成矩阵相乘的形式:

$\left[\begin{array}{ccc}{x}_{11}& x_{12}& x_{13}\\ x_{21}& x_{22}& x_{23}\\ x_{31}& x_{32}& x_{33}\end{array}\right]\otimes \left[\begin{array}{cc}{k}_{11}& k_{12}\\ x_{21}& k_{22}\end{array}\right]=\left[\begin{array}{cc}{y}_{11}& y_{12}\\ y_{21}& y_{22}\end{array}\right]$

即
$y_{11}=x_{11}{k}_{11}+x_{12}{k}_{12}+x_{21}{k}_{21}+x_{22}{k}_{22}$
$y_{12}=x_{12}{k}_{11}+x_{13}{k}_{12}+x_{22}{k}_{21}+x_{23}{k}_{22}$
$y_{21}=x_{21}{k}_{11}+x_{22}{k}_{12}+x_{31}{k}_{21}+x_{32}{k}_{22}$
$y_{22}=x_{22}{k}_{11}+x_{23}{k}_{12}+x_{32}{k}_{21}+x_{33}{k}_{22}$

即

$\left[\begin{array}{c}{y}_{11}\\ y_{12}\\ y_{21}\\ y_{22}\end{array}\right]=\left[\begin{array}{cccc}{x}_{11}& x_{12}& x_{21}& x_{22}\\ x_{12}& x_{13}& x_{22}& x_{23}\\ x_{21}& x_{22}& x_{31}& x_{32}\\ x_{22}& x_{23}& x_{32}& x_{33}\end{array}\right]\cdot \left[\begin{array}{c}{k}_{11}\\ k_{12}\\ k_{21}\\ k_{22}\end{array}\right]$

从上面的例子可以看出吗,要把卷积操作变成矩阵相乘需要把$Y$矩阵和$K$矩阵都展成一列向量,并且要对$X$矩阵做一个操作,就是把$X$按照卷积核的大小拆出一个大小为$k^2\ast k^2$的矩阵来.

最后矩阵乘法做完以后,将$y$向量$reshape$成矩阵回来就好了.