【Machine Learning】Generalization Theory

本笔记基于清华大学《机器学习》的课程讲义中泛化理论相关部分，基本为笔者在考试前一两天所作的Cheat Sheet。内容较多，并不详细，主要作为复习和记忆的资料。

For algroithm $A^{'}$ , exsits $f$ that is perfect answer of $D\in C\times\{0,1\}$ , such that $L_D(f)=0$ and

$E_{S\sim D^m}[L_D(A'(S))]\ge \frac{1}{4}$

Then $\Pr[L_D(A'(S))\ge \frac{1}{8}]\ge \frac{1}{7}$
Proof:
$\begin{align*} \max_{i}E_{S\sim D_i^m}[L_D(A'(S))]&=\max_{i}\frac{1}{k}\sum_{i=1}^k L_{D_i}(A'(S_i))\\ &\ge \frac{1}{T}\sum_{j=1}^T\frac{1}{k}\sum_{i=1}^k L_{D_j}(A'(S_i))\\ &\ge \frac{1}{k}\sum_{i=1}^k\frac{1}{T}\sum_{j=1}^T L_{D_j}(A'(S_i))\\ &\ge \min_S\frac{1}{T}\sum_{j=1}^T L_{D_j}(A'(S))\\ &\ge \min_S\frac{1}{T}\sum_{j=1}^T \frac{1}{2m}\sum_{i=1}^p1_{A'\text{ wrong at }v_i}\\ &\ge \min_S\frac{1}{T}\sum_{j=1}^T \frac{1}{2p}\sum_{i=1}^p1_{A'\text{ wrong at }v_i}\\ &\ge \frac{1}{2}\min_S\frac{1}{T}\sum_{j=1}^T \min_{i} 1_{A'\text{ wrong at }v_i}\\ &\ge \frac{1}{4} \end{align*}$
- The last inequality is beause divide $T$ into $2$ parts. One pair $f_i,f_{i'}$ only differs at $v_i$ .

With realizable assumption, the hypothesis class found by ERM is good enough with at least some samples
- Consider the probability of bad samples $L_S(h_S)=L_S(h^*)=0$ but $L_{D,f}(h_S)>\epsilon$ . Then we need $S$ to be the union(apply union bound) of misleading set $L_S(h_S)=0$ , each sample has probability $\le 1-\epsilon$ . Then probability is $|H_B|(1-\epsilon)^m$
PAC learnable: As sample number $m\ge m(\epsilon,\delta)$ , w.p. $1-\delta$ we can find a $h$ such that $L_{D,f}(h)\le \epsilon$ .
- Agnostic PAC learnable: $L_{D}(h)\le L_{D}(h^*)+\epsilon$
VC dimension

Generalization:
$L_D(h)-L_S(h)\le 2E_{S'\sim D^m}R(l\circ H\circ S')+c\sqrt{\frac{2\ln\frac{2}{\delta}}{m}}$
Massart Lemma:
$R(A)\le \max_{a\in A}|a-\bar{a}|\frac{\sqrt{2\log N}}{m}$
Contraction Lemma: If $\phi$ is $\rho$ -lipschitz, then
$R(\phi\circ A)\le \rho R(A)$