Solving Linear Equations

Case 2

$Ax=b,A\in {\mathbb{R}}^{m\times n},m\le n,rankA=m,x\in {\mathbb{R}}^n,b\in {\mathbb{R}}^m$

$\Rightarrow$ infinite many solutions

$\Rightarrow min||x||$
s.t. $Ax=b$

注：在此情形下，由于有无穷多解，可以将$Ax=b$看成是优化问题的约束条件，而这个问题可以看成有限子条件的优化问题。

Theorem

Thm: The unique solution $x^{\ast }$ of $Ax=b$ that minimizes $||x||$ is given by $x^{\ast }=A^T(A{A}^T{)}^{-1}b$ .

Kaczmarz’s Algorithm

为了避免计算逆矩阵，我们介绍Kaczmarz算法。

1. Set $i=0,x^0$
2. For $j=1,\cdots ,m$ do
   $x^{im+j}=x^{im+j-1}+\mu (b_j-{a_j}^T{x}^{im+j-1})\frac{a_j}{{a_j}^T{a}_j}$
3. $i$++, goto 2

注：$0<\mu <2$。由于带限制的优化问题较为复杂，故在本门课程中对带限制的优化问题算法的收敛速度不进行研究。

Theorem

In Kaczmarz’s Algorithm, if $x^0=0$, then $x^k\to x^{\ast }=A^T(A{A}^T{)}^{-1}b$ as $k\to \infty$.

Example

$A=\left[\begin{array}{cc}1& -1\\ 0& 1\end{array}\right]$

$b=\left[\begin{array}{c}2\\ 3\end{array}\right]$

$\mu =1,x^0=\left[\begin{array}{c}0\\ 0\end{array}\right]$

$a_1=\left[\begin{array}{c}1\\ -1\end{array}\right]$

$a_2=\left[\begin{array}{c}0\\ 1\end{array}\right]$

$b_1=2,b_2=3$

$i=0,j=1:x^1=\left[\begin{array}{c}0\\ 0\end{array}\right]+\left(2-[1,-1]\cdot \left[\begin{array}{c}0\\ 0\end{array}\right]\right)\frac{\left[\begin{array}{c}1\\ -1\end{array}\right]}{[1,-1]\cdot \left[\begin{array}{c}1\\ -1\end{array}\right]}=\left[\begin{array}{c}1\\ -1\end{array}\right]$

$i=0,j=2:x^2=\left[\begin{array}{c}1\\ -1\end{array}\right]+\left(3-[0,-1]\cdot \left[\begin{array}{c}1\\ -1\end{array}\right]\right)\frac{\left[\begin{array}{c}0\\ 1\end{array}\right]}{[0,1]\cdot \left[\begin{array}{c}0\\ 1\end{array}\right]}=\left[\begin{array}{c}1\\ 3\end{array}\right]$

$i=1,j=1:x^3=\left[\begin{array}{c}1\\ 3\end{array}\right]+\left(2-[1,-1]\cdot \left[\begin{array}{c}1\\ 3\end{array}\right]\right)\frac{\left[\begin{array}{c}1\\ -1\end{array}\right]}{[1,-1]\cdot \left[\begin{array}{c}1\\ -1\end{array}\right]}=\left[\begin{array}{c}3\\ 1\end{array}\right]$

$\cdots$

$x^{\ast }=\left[\begin{array}{c}5\\ 3\end{array}\right]$

Pseudoinverse

Definition

$A^{+}\in {\mathbb{R}}^{m\times n}$ is a pseudoinverse of $A$, if $A{A}^{T}A=A$ and $\exists U\in {\mathbb{R}}^{n\times n},V\in {\mathbb{R}}^{m\times m}$ s.t. $A^{+}=U{A}^T,A^{+}=A^{T}V$
注：pseudoinverse表示伪逆，是矩阵逆的一种广义形式。

Special Case 1

$m\ge n,rankA=n$

$A^{+}=(A^{T}A{)}^{-1}{A}^T\to A{A}^{+}A=A$

$U=(A^{T}A{)}^{-1},V=A(A^{T}A{)}^{-1}(A^{T}A{)}^{-1}{A}^T$

$A^{+}=U{A}^T,A^{+}=A^{T}V$

Special Case 2

$m\le n,rankA=m$

$A^{+}=A^T(A{A}^T{)}^{-1}\to A{A}^{+}A=A$

$U=A^T(A{A}^T{)}^{-1}(A{A}^T{)}^{-1}A,V=(A{A}^T{)}^{-1}$

$A^{+}=U{A}^T,A^{+}=A^{T}V$

Properties of Pseudoinverse

Lemma 1: Unique pseudoinverse

pf: Assume: $A_1^{+},A_2^{+}$ pseudoinverse of $A$.
$A{A}_1^{+}A=A{A}_2^{+}A=A$
And $U_1,U_2\in {\mathbb{R}}^{n\times n},V_1,V_2\in {\mathbb{R}}^{m\times m}$

$A_1^{+}=U_1{A}^T=A^T{V}_1,A_2^{+}=U_2{A}^T=A^T{V}_2$

Let $D=A_2^{+}-A_1^{+},U=U_2-U_1,V=V_2-V_1$

Then $O=ADA,D=U{A}^T=A^{T}V$

$\Rightarrow (DA{)}^{T}DA=A^T{D}^{T}DA=A^T{V}^{T}ADA=O$

$\Rightarrow DA=O$

$\Rightarrow D{D}^T=DA{U}^T=O\Rightarrow D=O$

Lemma 2: Full Rank Factorization

Let $A\in {\mathbb{R}}^{m\times n},rankA=r\le min(m,n)$. Then exist $B\in {\mathbb{R}}^{m\times r}$ and $C\in {\mathbb{R}}^{r\times n}$, with $rankB=rankC=r$ and $A=BC$.

Lemma 3

Let $A\in {\mathbb{R}}^{m\times n}$ have full rank factorization $A=BC$, Then $A^{+}=C^{+}{B}^{+}$.

Example

$A=\left[\begin{array}{cccc}2& 1& -2& 5\\ 1& 0& -3& 2\\ 3& -1& -13& 5\end{array}\right]$

$rankA=2$

$B=\left[\begin{array}{cc}2& 1\\ 1& 0\\ 3& -1\end{array}\right]$

$C=\left[\begin{array}{cccc}0& 1& -3& 2\\ 1& 0& 4& 1\end{array}\right]$

$A=BC,A^{+}=C^{+}{B}^{+}$

Case 3

$Ax=b,A\subseteq {\mathbb{R}}^{m\times n},rankA\le min(m,n)$

①$x$ minimize $||Ax-b|{|}^2$

②$x$ minimize $||x||$

注：Case 1和Case 2的主要区别在于$m$和$n$的大小关系，此时$A$满秩。Case 3则表示了$A$不满秩的更为一般的情形。在$m=n$时，Case 1和Case 2两种情况均适用。在$A$满秩的时候，Case 3也适用。故在分类条件中，并没有严格分类讨论，而是把该结论适用的最大范围写了上去。

Theorem

Given $Ax=b$ with $rankA=r$, the unique vector $x^{\ast }=A^{+}b$ minimizes $||Ax-b|{|}^2$. Furthermore, among all vectors that minimize $||Ax-b|{|}^2$, $x^{\ast }$ is the unique one with minimal norm.

Neural Networks

Single neuron

Task: data fitting

Training data: $<x^d,y^d>,x^d\in {\mathbb{R}}^{s\times n},y^d\in {\mathbb{R}}^s$

$f(x)={\sum }_{i=1}^m{\omega }_i{x}_i={\omega }^{T}x$

Find $\omega \in {\mathbb{R}}^n$ minimizing $\frac{1}{2}{\sum }_{i=1}^n(y_i^d-{x_i^d}^T{\omega }_i{)}^2=\frac{1}{2}||y^d-{x^d}^T\omega |{|}^2$

Case 1

$rankX=s\le n$

$\Rightarrow \exists$ infinite $\omega :y^d={x^d}^T\omega$

$\Rightarrow min||\omega ||$
s.t. $y^d={x^d}^T\omega$

$\Rightarrow {\omega }^{\ast }=x^d({x^d}^T{x}^d{)}^{-1}{y}^d$

$\Rightarrow$Kaczmarz’s Algorithm

Case 2

$rankX=n\le s$

$\Rightarrow {\omega }^{\ast }=({x^d}^T{x}^d{)}^{-1}{x^d}^T{y}^d$

$\Rightarrow$ Gradient algorithm

${\omega }^{k+1}={\omega }^k-{\alpha }^k\nabla f({\omega }^k)$

$\Rightarrow {\omega }^{k+1}={\omega }^k+{\alpha }^k{x}^d{e}^k$, where $e^k=y^d-{x^d}^T{\omega }^k$

$y=f({\sum }_{i=1}^m{\omega }_i{x}_i)\Rightarrow min\frac{1}{2}||y^d-f({\sum }_{i=1}^m{\omega }_i{x}_i)|{|}^2$

${\omega }^{k+1}={\omega }^k+{\alpha }^k\frac{x^d{e}^k}{||x^d|{|}^2}$

[only one $<x^d,y^d>$]

$e^k=y^d-f({x^d}^T{\omega }^k)$

Multiple Layers

backpropagation algorithm(反向传播算法)
由于情况比较复杂，此处不作展开。

总结

上节课介绍了解线性方程组的第一种情况，这节课介绍了第二种和第三种情况。为了使结论更具一般性，还引入了矩阵的伪逆概念。接下来开始介绍神经网络。对神经网络做了一些数学上的简化，为了便于理论研究。主要介绍了最简单的单层神经网络。还提及了多层神经网络的反向传播算法，但由于过于复杂，于是没有具体展开计算。