Deep Learning

Basic

• 神经网络：

• 监督学习：1个x对应1个y；

• Sigmoid : 激活函数
  $$sigmoid=\frac{1}{1+e^{-x}}$$

• ReLU : 线性整流函数；

Logistic Regression

–>binary classification / x–>y 0 1

some sign

$$\begin{gathered} (x,y),x\in {\mathbb{R}}^{n_x},y\in 0,1 \\ M=m_{train}{m}_{test}=test \\ M:(x^{(1)},y^{(1)}),(x^{(2)},y^{(2)})...,(x^{(m)},y^{(m)}) \\ X=\left[\begin{array}{cccc}{x}^{(1)}& x^{(2)}& \cdots & x^{(m)}\end{array}\right]\leftarrow n^x\times m \\ \hat{y}=P(y=1\mid x)\hat{y}=\sigma (w^{t}x+b)w\in {\mathbb{R}}^{n_x}b\in \mathbb{R} \\ \sigma (z)=\frac{1}{1+e^{-z}} \end{gathered}$$

Loss function

单个样本
$$\begin{gathered} Lossfunction:\mathcal{L}(\hat{y},y)=\frac{1}{2}(\hat{y}-y{)}^2 \\ p(y\mid x)={\hat{y}}^y(1-\hat{y}{)}^{(1-y)} \\ mincost\to max\log (y\mid x) \\ \mathcal{L}(\hat{y},y)=-(y\log (\hat{y})+(1-y)\log (1-\hat{y})) \\ y=1:\mathcal{L}(\hat{y},y)=-\log \hat{y}\log \hat{y}\leftarrow larger\hat{y}\leftarrow larger \\ y=0:\mathcal{L}(\hat{y},y)=-\log (1-\hat{y})\log (1-\hat{y})\leftarrow larger\hat{y}\leftarrow smaller \end{gathered}$$

cost function

$$\mathcal{J}(w,b)=\frac{1}{m}\sum _{i=1}^m\mathcal{L}({\hat{y}}^{(i)},y^{(i)})$$

Gradient Descent

find w,b that minimiaze J(w,b) ;

Repeat:
$$\begin{gathered} w:=w-\alpha \frac{\partial \mathcal{J}(w,b)}{\partial w}(dw) \\ b:=b-\alpha \frac{\partial \mathcal{J}(w,b)}{\partial b}(db) \end{gathered}$$

Computation Grapha

example:
$$J=3(a+bc)$$

one example gradient descent computer grapha:

recap:
$$\begin{gathered} z=w^{T}x+b \\ \hat{y}=a=\sigma (z)=\frac{1}{1+e^{-z}} \\ \mathcal{L}(a,y)=-(y\log (a)+(1-y)\log (1-a)) \end{gathered}$$
The grapha:

$$\begin{gathered} {}^{\prime }d{a}^{\prime }=\frac{d\mathcal{L}(a,y)}{da}=-\frac{y}{a}+\frac{1-y}{1-a}{ \\ }^{\prime }d{z}^{\prime }=\frac{d\mathcal{L}(a,y)}{dz}=\frac{d\mathcal{L}}{da}\cdot \frac{da}{dz}=a-y{ \\ }^{\prime }d{w}_1^{\prime }=x_1\cdot dz... \\ {w}_1:=w_1-\alpha d{w}_1... \end{gathered}$$
m example gradient descent computer grapha:

recap:
$$\mathcal{J}(w,b)=\frac{1}{m}\sum _{i=1}^m\mathcal{L}(a^{(i)},y^{(1)})$$
The grapha: (two iterate)
∂ ∂ w 1 J ( w , b ) = 1 m ∑ i = 1 m ∂ ∂ w 1 L ( a ( i ) , y ( 1 ) ) F o r i = 1 t o m : { a ( i ) = σ ( w T x ( i ) + b ) J + = − [ y ( i ) log ⁡ a i + ( 1 − y ( i ) log ⁡ ( 1 − a ( i ) ) ) ] d z ( i ) = a ( i ) − y ( i ) d w 1 + = x 1 ( i ) d z ( i ) d w 2 + = x 2 ( i ) d z ( i ) d b + = d z ( i ) } J / = m ; d w 1 / = m ; d w 2 / = m ; d b / = m d w 1 = ∂ J ∂ w 1 w 1 = w 1 − α d w 1 \frac{\partial}{\partial w_1}\mathcal{J}(w,b)=\frac{1}{m}\sum_{i=1}^m\frac{\partial}{\partial w_1}\mathcal{L}(a^{(i)},y^{(1)})\\\\ For \quad i=1 \quad to \quad m:\{\\ a^{(i)}=\sigma (w^Tx^{(i)}+b)\\ \mathcal{J}+=-[y^{(i)}\log a^{i}+(1-y^{(i)}\log(1-a^{(i)}))] \\ dz^{(i)}=a^{(i)}-y^{(i)}\\ dw_1+=x_1^{(i)}dz^{(i)}\\ dw_2+=x_2^{(i)}dz^{(i)}\\ db+=dz^{(i)}\}\\ \mathcal{J}/=m;dw_1/=m;dw_2/=m;db/=m\\ dw_1=\frac{\partial\mathcal{J}}{\partial w_1}\\ w_1=w_1-\alpha dw_1 ∂w1​∂​J(w,b)=m1​i=1∑m​∂w1​∂​L(a(i),y(1))Fori=1tom:{a(i)=σ(wTx(i)+b)J+=−[y(i)logai+(1−y(i)log(1−a(i)))]dz(i)=a(i)−y(i)dw1​+=x1(i)​dz(i)dw2​+=x2(i)​dz(i)db+=dz(i)}J/=m;dw1​/=m;dw2​/=m;db/=mdw1​=∂w1​∂J​w1​=w1​−αdw1​

Vectorization

vectorized

$$z=np.dot(w,x)+b$$
logistic regression derivatives:

change:
$$\begin{gathered} d{w}_1=0,d{w}_2=0\to dw=np.zeros((n_x,1)) \\ \{\begin{array}{l}d{w}_1+=x_1^{(i)}d{z}^{(i)}\\ d{w}_2+=x_2^{(i)}d{z}^{(i)}\end{array}\to dw+=x^{(i)}d{z}^{(i)} \\ Z=\left(\begin{array}{cccc}{z}^{(1)}& z^{(2)}& ...& z^{(m)}\end{array}\right)=w^{T}X+b \\ A=\sigma (Z) \\ dz=A-Y=\left(\begin{array}{cccc}{a}^{(1)}-y^{(1)}& z^{(2)}-y^{(2)}& ...& z^{(m)}-y^{(m)}\end{array}\right) \\ db=\frac{1}{m}\sum _{i=1}^{m}d{z}^{(i)}=\frac{1}{m}np.sum(dz) \\ dw=\frac{1}{m}Xd{z}^T=\frac{1}{m}\left(\begin{array}{cccc}{x}^{(1)}\cdot d{z}^{(1)}& x^{(2)}\cdot d{z}^{(2)}& ...& x^{(m)}\cdot d{z}^{(m)}\end{array}\right) \end{gathered}$$

Implementing:

$$\begin{gathered} Z=w^{T}X+b=np.dot(w^T,X)+b \\ A=\sigma (Z) \\ J=-\frac{1}{m}\sum _{i=1}^m(y^{(i)}\log (a^{(i)})+(1-y^{(i)})\log (1-a^{(i)})) \\ dZ=A-Y \\ dw=\frac{1}{m}Xd{Z}^T \\ db=\frac{1}{m}np.sum(dZ) \\ w:=w-\alpha dw \\ b:=b-\alpha db \end{gathered}$$

broadcasting

$$np.dot(w^T,X)+b$$
A note on numpy
$$\begin{gathered} a=np.random.randn(5)//wrong\to a=a.reshape(5,1) \\ assert(a.shape==(5,1)) \\ a=np.random.randn(5,1)\to columvector \end{gathered}$$

Shallow Neural Network

Representation

2 layer NN:
$$\begin{gathered} Inputlayer\to hidden\to layer\to outlayer \\ {a}^{[0]}\to a^{[1]}\to a^{[2]} \\ {z}^{[1]}=W^{[1]}{a}^{[0]}+b^{[1]} \\ {a}^{[1]}=\sigma (z^{[1]}) \\ {z}^{[2]}=W^{[2]}{a}^{[1]}+b^{[2]} \\ {a}^{[2]}=\sigma (z^{[2]})=\hat{y} \end{gathered}$$

computing:

$$\begin{gathered} z_i^{[1]}=w_i^{[1]T}x+b_i^{[1]} \\ {a}_i^{[1]}=\sigma (z_i^{[1]}) \\ \left[\begin{array}{c}{w}_1^{[1]T}\\ w_2^{[1]T}\\ w_3^{[1]T}\\ w_4^{[1]T}\end{array}\right]\cdot \left[\begin{array}{c}{x}_1\\ x_2\\ x_3\end{array}\right]+\left[\begin{array}{c}{b}_1^{[1]}\\ b_2^{[1]}\\ b_3^{[1]}\\ b_4^{[1]}\end{array}\right]=\left[\begin{array}{c}{z}_1^{[1]}\\ z_2^{[1]}\\ z_3^{[1]}\\ z_4^{[1]}\end{array}\right] \end{gathered}$$

Vectorize:

$$\begin{gathered} x^{(i)}\to a^{[2](i)}={\hat{y}}^{(i)} \\ {Z}^{[1]}=W^{[1]}X+b^{[1]} \\ {A}^{[1]}=\sigma (Z^{[1]}) \\ {Z}^{[2]}=W^{[2]}{A}^{[1]}+b^{[2]} \\ {A}^{[2]}=\sigma (Z^{[2]}) \\ {W}^{[1]}\cdot \left[\begin{array}{cccc}{x}^{(1)}& x^{(2)}& \cdots & x^{(m)}\end{array}\right]+b=\left[\begin{array}{cccc}{z}^{[1](1)}& z^{[1](2)}& \cdots & z^{[1](m)}\end{array}\right]=Z^{[1]} \end{gathered}$$

Activation functions

$$\begin{gathered} a=\frac{1}{1+e^{-z}},a^{\prime }=a(1-a) \\ a=\tanh (z)=\frac{e^z-e^{-z}}{e^z+e^{-z}},a\in (-1,1),a^{\prime }=1-a^2 \\ a=max(0,z) \\ a=max(0.01z,z) \end{gathered}$$

Gradient descent

computation

$$\begin{gathered} z^{[1]}=W^{[1]}x+b^{[1]}\to \\ {a}^{[1]}=\sigma (z^{[1]})\to \\ {z}^{[2]}=W^{[2]}{a}^{[1]}+b^{[2]}\to \\ {a}^{[2]}=\sigma (z^{[2]})\to \\ \mathcal{L}(a^{[2]},y) \\ d{z}^{[2]}=a^{[2]}-y \\ d{w}^{[2]}=d{z}^{[2]}{a}^{[1]T} \\ d{b}^{[2]}=d{z}^{[2]} \\ d{z}^{[1]}=w^{[2]T}d{z}^{[2]}\ast a^{{}^{\prime }[1]} \\ d{w}^{[1]}=d{z}^{[1]}\cdot x^T \\ d{b}^{[1]}=d{z}^{[1]} \end{gathered}$$

dz^[1]的推导涉及到了矩阵求导

the dimension

$$\begin{gathered} x:(n_0,m)W^{[1]}:(n_1,n_0)\to \\ {a}^{[1]}:(n_1,m)W^{[2]:}:(n_2,n_1)\to \\ {a}^{[2]}:(n_2,m) \end{gathered}$$

vectorize

$$\begin{gathered} d{Z}^{[2]}=A^{[2]}-Y \\ d{W}^{[2]}=\frac{1}{m}d{Z}^{[2]}{A}^{[1]T} \\ d{b}^{[2]}=np.sum(d{Z}^{[2]},axis=1,keepdims=True) \\ d{Z}^{[1]}=W^{[2]T}d{Z}^{[2]}\ast A^{{}^{\prime }[1]} \\ d{W}^{[1]}=\frac{1}{m}d{Z}^{[1]}{X}^T \\ d{b}^{[1]}=\frac{1}{m}np.sum(dZ[1],axis=1,keepdims=True) \end{gathered}$$

Random Initialization

$$\begin{gathered} w^{[1]}=np.random.randn((2,2))\ast 0.01 \\ {b}^{[1]}=np.zero((2,1)) \end{gathered}$$

Deep neural network

notation

$$\begin{gathered} example:LlayerNN \\ {a}^{[l]}\to activationfunction \\ {w}^{[l]}\to weightsfor{z}^{[l]} \\ \hat{y}=a^{[L]} \end{gathered}$$

Forward propagation

$$\begin{gathered} forl=1,2,\mathrm{3..} \\ {z}^{[l]}=w^{[l]}{a}^{[l-1]}+b^{[l]} \\ cache{z}^{[l]},w^{[l]},b^{[l]} \\ {a}^{[l]}=g^{[l]}(z^{[l]}) \end{gathered}$$

Backward propagation

$$\begin{gathered} d{a}^{[l]}\to d{a}^{[l-1]}(d{z}^{[l]},d{w}^{[l]},d{b}^{[l]}) \\ d{z}^{[l]}=d{a}^{[l]}\ast g^{[l{]}^{\prime }}(z^{[l]})=w^{[l+1]}d{z}^{[l+1]}\ast g^{[l{]}^{\prime }}(z^{[l]}) \\ d{w}^{[l]}=d{z}^{[l]}\cdot a^{[l-1]T} \\ d{b}^{[l]}=d{z}^{[l]} \\ d{a}^{[l-1]}=w^{[l]T}\cdot d{z}^{[l]} \end{gathered}$$

matrix dimensions

$$\begin{gathered} dw,w^{[l]}:(n^{[l]},n^{[l-1]}) \\ db,b^{[l]}:(n^{[l]},1) \\ {Z}^{[l]},A^{[l]}:(n^{[l]},m) \end{gathered}$$