模式识别作业

1.说明判别分类器(如logistic回归)与上述特定类别的高斯朴素贝叶斯分类器之间的关系正是logistic回归所采用的形式。

经过第2问更加普遍的推导过程：

对应参数为：

二次项：
$v=[\frac{{\sigma }_{11}^2-{\sigma }_{10}^2}{2{\sigma }_{11}^2{\sigma }_{10}^2},...,\frac{{\sigma }_{D1}^2-{\sigma }_{D0}^2}{2{\sigma }_{D1}^2{\sigma }_{D0}^2}]$

一次项：
$w=[\frac{{\sigma }_{10}^2{\mu }_{11}-{\sigma }_{11}^2{\mu }_{10}}{{\sigma }_{11}^2{\sigma }_{10}^2},...,\frac{{\sigma }_{D0}^2{\mu }_{D1}-{\sigma }_{D1}^2{\mu }_{D0}}{{\sigma }_{D1}^2{\sigma }_{D0}^2}]$

常数项：
$b=ln(\frac{\pi }{1-\pi })+\sum ln\frac{{\sigma }_{i0}}{\sigma i_1}+\sum \frac{{\sigma }_{11}^2{\mu }_{10}^2-{\sigma }_{10}^2{\mu }_{11}^2}{2{\sigma }_{11}^2{\sigma }_{10}^2}$

其中

$f(x)=\frac{1}{1+exp(\sum v_i{x}_i^2+w_i{x}_i+b)}$

由于${\sigma }_{i0}={\sigma }_{i1}={\sigma }_i$

发现$v=0.$

二次项消失，一次项和常数项如下：

一次项：
$w=[\frac{{\mu }_{11}-{\mu }_{10}}{{\sigma }_1^2},...,\frac{{\mu }_{D1}-{\mu }_{D0}}{{\sigma }_D^2}]$

常数项：
$b=ln(\frac{\pi }{1-\pi })+\sum \frac{{\mu }_{10}^2-{\mu }_{11}^2}{2{\sigma }_1^2}$

$f(x)=\frac{1}{1+exp(w_i{x}_i+b)}$

---

2.当换成更普遍的高斯函数，是否仍有Logistic Regression形式？

生成式高斯朴素贝叶斯分类器如下：

$P(y=1|X)=\frac{P(X|y=1)P(y=1)}{P(X)}=\frac{P(X|y=1)P(y=1)}{P(X|y=1)P(y=1)+P(X|y=0)P(y=0)}$

$=\frac{1}{1+\frac{P(X|y=1)P(y=1)}{P(X|y=0)P(y=0)}}=\frac{1}{1+exp(ln(\frac{P(X|y=1)P(y=1)}{P(X|y=0)P(y=0)}))}$

其中

$ln(\frac{P(X|y=1)P(y=1)}{P(X|y=0)P(y=0)})=ln(\frac{\pi }{1-\pi })+ln\frac{P(X|y=1)}{P(X|y=0)}$

$=ln(\frac{\pi }{1-\pi })+ln(\frac{(\sqrt{2\pi }{\sigma }_{i1}{)}^{-1}exp(-(X-{\mu }_1{)}^2/2{\sigma }_1^2)}{(\sqrt{2\pi }{\sigma }_{i0}{)}^{-1}exp(-(X-{\mu }_0{)}^2/2{\sigma }_0^2)})$

$=ln(\frac{\pi }{1-\pi })+\sum ln(\frac{(\sqrt{2\pi }{\sigma }_{i1}{)}^{-1}exp(-(X-{\mu }_{i1}{)}^2/2{\sigma }_{i1}^2)}{(\sqrt{2\pi }{\sigma }_{i0}{)}^{-1}exp(-(X-{\mu }_{i0}{)}^2/2{\sigma }_{i0}^2)})$

$=ln(\frac{\pi }{1-\pi })+\sum (ln\frac{{\sigma }_{i0}}{\sigma i_1}+x_i^2\frac{{\sigma }_{i1}^2-{\sigma }_{i0}^2}{2{\sigma }_{i1}^2{\sigma }_{i0}^2}+x_i\frac{{\sigma }_{i0}^2{\mu }_{i1}-{\sigma }_{i1}^2{\mu }_{i0}}{{\sigma }_{i1}^2{\sigma }_{i0}^2}+\frac{{\sigma }_{i1}^2{\mu }_{i0}^2-{\sigma }_{i0}^2{\mu }_{i1}^2}{2{\sigma }_{i1}^2{\sigma }_{i0}^2})$

所以

当${\sigma }_{i1}^2={\sigma }_{i0}^2$时，$x_i^2$项不复存在，其对应形式刚好为logistic regression。

对应参数为：

二次项：
$v=[\frac{{\sigma }_{11}^2-{\sigma }_{10}^2}{2{\sigma }_{11}^2{\sigma }_{10}^2},...,\frac{{\sigma }_{D1}^2-{\sigma }_{D0}^2}{2{\sigma }_{D1}^2{\sigma }_{D0}^2}]$

一次项：
$w=[\frac{{\sigma }_{10}^2{\mu }_{11}-{\sigma }_{11}^2{\mu }_{10}}{{\sigma }_{11}^2{\sigma }_{10}^2},...,\frac{{\sigma }_{D0}^2{\mu }_{D1}-{\sigma }_{D1}^2{\mu }_{D0}}{{\sigma }_{D1}^2{\sigma }_{D0}^2}]$

常数项：
$b=ln(\frac{\pi }{1-\pi })+\sum ln\frac{{\sigma }_{i0}}{\sigma i_1}+\sum \frac{{\sigma }_{11}^2{\mu }_{10}^2-{\sigma }_{10}^2{\mu }_{11}^2}{2{\sigma }_{11}^2{\sigma }_{10}^2}$

其中

$f(x)=\frac{1}{1+exp(\sum v_i{x}_i^2+w_i{x}_i+b)}$

---

3.非朴素高斯贝叶斯分类器是否仍具有Logistic Regress的性质？

$P(y|X)=\frac{P(x1,x2|y=1)P(y=1)}{P(X)}=\frac{P(x1,x2|y=1)P(y=1)}{P(x1,x2|y=1)P(y=1)+P(x1,x2|y=0)P(y=0)}$

$=\frac{1}{1+\frac{P(x1,x2|y=1)P(y=1)}{P(x1,x2|y=0)P(y=0)}}=\frac{1}{1+exp(e)}$

其中

$e=ln\frac{\pi }{1-\pi }+ln(\frac{P(x1,x2|y=1)}{P(x1,x2|y=0)})$

由于

$P(x1,x2|y=k)=\frac{1}{2\pi {\sigma }_1{\sigma }_2\sqrt{1-p^2}}exp(略去)$

将其带入式子$e$中，得到：

$e\ast 2(1-p^2){\sigma }_1^2{\sigma }_2^2=ln\frac{\pi }{1-\pi }+x_1^2({\sigma }_2^2-{\sigma }_2^2)+x_2^2({\sigma }_1^2-{\sigma }_1^2)+x_1(-2{\mu }_{10}{\sigma }_2^2+2p{\sigma }_1{\sigma }_2{\mu }_{20}+2{\mu }_{11}{\sigma }_2^2-2p{\sigma }_1{\sigma }_2{\mu }_{21})+x_2(-2{\mu }_{20}{\sigma }_1^2+2p{\sigma }_1{\sigma }_2{\mu }_{10}+2{\mu }_{21}{\sigma }_1^2-2p{\sigma }_1{\sigma }_2{\mu }_{11})+x_1{x}_2(-2p{\sigma }_1{\sigma }_2+2p{\sigma }_1{\sigma }_2)-2p{\sigma }_1{\sigma }_2{\mu }_{10}{\mu }_{20}+2p{\sigma }_1{\sigma }_2{\mu }_{11}{\mu }_{21}$

$=x_1(-2{\mu }_{10}{\sigma }_2^2+2p{\sigma }_1{\sigma }_2{\mu }_{20}+2{\mu }_{11}{\sigma }_2^2-2p{\sigma }_1{\sigma }_2{\mu }_{21})+x_2(-2{\mu }_{20}{\sigma }_1^2+2p{\sigma }_1{\sigma }_2{\mu }_{10}+2{\mu }_{21}{\sigma }_1^2-2p{\sigma }_1{\sigma }_2{\mu }_{11})+2p{\sigma }_1{\sigma }_2({\mu }_{11}{\mu }_{21}-{\mu }_{10}{\mu }_{20})$

故

$b=2p{\sigma }_1{\sigma }_2({\mu }_{11}{\mu }_{21}-{\mu }_{10}{\mu }_{20})/(2(1-p^2){\sigma }_1^2{\sigma }_2^2)$

$w_1=(-2{\mu }_{10}{\sigma }_2^2+2p{\sigma }_1{\sigma }_2{\mu }_{20}+2{\mu }_{11}{\sigma }_2^2-2p{\sigma }_1{\sigma }_2{\mu }_{21})/(2(1-p^2){\sigma }_1^2{\sigma }_2^2)$

$w_2=(-2{\mu }_{20}{\sigma }_1^2+2p{\sigma }_1{\sigma }_2{\mu }_{10}+2{\mu }_{21}{\sigma }_1^2-2p{\sigma }_1{\sigma }_2{\mu }_{11})/(2(1-p^2){\sigma }_1^2{\sigma }_2^2)$

则原式可以写成：

$P(y|X)=\frac{1}{1+exp(b+w_1{x}_1+w_2{x}_2)}$

因此，仍然满足logistic regression形式。

---