#! https://zhuanlan.zhihu.com/p/686235508

深度强化学习(二)(贝尔曼方程)

一.贝尔曼方程（将 $Q_{\pi }$ 表示成 $Q_{\pi }$ ）

Theorem :假设 $R_t$ 是 $S_t、A_t、S_{t+1}$ 的函数。那么
$$Q_{\pi }\left(s_t,a_t\right)={\mathbb{E}}_{S_{t+1},A_{t+1}}\left[R_t+\gamma \cdot Q_{\pi }\left(S_{t+1},A_{t+1}\right)\mid S_t=s_t,A_t=a_t\right].\text{\hspace{1em}(1.1)}$$

proof:令 S t + 1 : = { S t + 1 , S t + 2 , ⋯ } \mathcal{S}_{t+1:}=\left\{S_{t+1}, S_{t+2}, \cdots\right\} St+1:​={St+1​,St+2​,⋯}, A t + 1 : = { A t + 1 , A t + 2 , ⋯ } \mathcal{A}_{t+1:}=\left\{A_{t+1}, A_{t+2}, \cdots\right\} At+1:​={At+1​,At+2​,⋯},由$U_t$的定义知$U_t=R_t+\gamma \cdot U_{t+1}$

$$\begin{array}{rl}{Q}_{\pi }\left(s_t,a_t\right)& ={\mathbb{E}}_{{\mathcal{S}}_{t+1:},{\mathcal{A}}_{t+1:}}\left[U_t\mid S_t=s_t,A_t=a_t\right]\\ & ={\mathbb{E}}_{{\mathcal{S}}_{t+1:},{\mathcal{A}}_{t+1:}}\left[R_t+\gamma \cdot U_{t+1}\mid S_t=s_t,A_t=a_t\right]\\ & =\underset{(1)}{\underbrace{{\mathbb{E}}_{{\mathcal{S}}_{t+1},{\mathcal{A}}_{t+1}}\left[R_t|S_t=s_t,A_t=a_t\right]}}+\gamma \cdot \underset{(2)}{\underbrace{{\mathbb{E}}_{{\mathcal{S}}_{t+1:},{\mathcal{A}}_{t+1:}}\left[U_{t+1}\mid S_t=s_t,A_t=a_t\right]}}\end{array}$$
其中,$t$时刻的回报$R_t$只与$t+1$时刻的状态$S_{t+1}$有关，而$S_{t+1}$只与$S_t,A_t$有关,则
$$\begin{array}{rl}(1)& ={\mathbb{E}}_{{\mathcal{S}}_{t+1},{\mathcal{A}}_{t+1}}\left[R_t|S_t=s_t,A_t=a_t\right]\\ & ={\mathbb{E}}_{S_{t+1}}\left[R_t|S_t=s_t,A_t=a_t\right]\\ & ={\mathbb{E}}_{S_{t+1},A_{t+1}}\left[R_t|S_t=s_t,A_t=a_t\right]\end{array}$$
对$(2)$中的式子变形可得
$$\begin{array}{rl}(2)& ={\mathbb{E}}_{{\mathcal{S}}_{t+1:},{\mathcal{A}}_{t+1:}}\left[U_{t+1}\mid S_t=s_t,A_t=a_t\right]\\ & ={\mathbb{E}}_{S_{t+1},A_{t+1},{\mathcal{S}}_{t+2},{\mathcal{A}}_{t+2}}\left[U_{t+1}|S_t=s_t,A_t=a_t\right]\\ & ={\mathbb{E}}_{S_{t+1},A_{t+1}}\left[{\mathbb{E}}_{{\mathcal{S}}_{t+2},{\mathcal{A}}_{t+2}}\left[U_{t+1}|S_{t+1},A_{t+1},S_t=s_t,A_t=a_t\right]|S_t=s_t,A_t=a_t\right]利用马尔可夫性\\ & ={\mathbb{E}}_{S_{t+1},A_{t+1}}\left[{\mathbb{E}}_{{\mathcal{S}}_{t+2},{\mathcal{A}}_{t+2}}\left[U_{t+1}|S_{t+1},A_{t+1}\right]|S_t=s_t,A_t=a_t\right]\\ & ={\mathbb{E}}_{S_{t+1},A_{t+1}}\left[Q_{\pi }\left(S_{t+1},A_{t+1}\right)\mid S_t=s_t,A_t=a_t\right]\end{array}$$
由此证毕。

二.贝尔曼方程$\text{ (将 }{Q}_{\pi }\text{ 表示成 }{V}_{\pi }\text{ ) }$

Theorem :假设 $R_t$ 是 $S_t、A_t、S_{t+1}$ 的函数。那么
$$Q_{\pi }\left(s_t,a_t\right)={\mathbb{E}}_{S_{t+1}}\left[R_t+\gamma \cdot V_{\pi }\left(S_{t+1}\right)\mid S_t=s_t,A_t=a_t\right]\text{\hspace{1em}(1.2)}$$

proof:$\text{ 由于 }{V}_{\pi }\left(S_{t+1}\right)={\mathbb{E}}_{A_{t+1}\sim \pi \left(\cdot \mid S_{t+1}\right)}\left[Q\left(S_{t+1},A_{t+1}\right)\right]={\mathbb{E}}_{A_{t+1}}\left[Q_{\pi }(S_{t+1},A_{t+1})|S_{t+1}\right]$
$$\begin{array}{rl}(2)=& {\mathbb{E}}_{S_{t+1},A_{t+1}}\left[Q_{\pi }\left(S_{t+1},A_{t+1}\right)\mid S_t=s_t,A_t=a_t\right]\\ =& {\mathbb{E}}_{S_{t+1}}\left[{\mathbb{E}}_{A_{t+1}}\left[Q_{\pi }(S_{t+1},A_{t+1})|S_{t+1}\right]|S_t=s_t,A_t=a_t\right]\\ =& {\mathbb{E}}_{S_{t+1}}\left[V_{\pi }\left(S_{t+1}\right)|S_t=s_t,A_t=a_t\right]\end{array}$$
证毕

三.贝尔曼方程(将 $V_{\pi }$ 表示成 $V_{\pi }$ )

Theorem :假设 $R_t$ 是 $S_t、A_t、S_{t+1}$ 的函数。那么
$$V_{\pi }\left(s_t\right)={\mathbb{E}}_{A_t,S_{t+1}}\left[R_t+\gamma \cdot V_{\pi }\left(S_{t+1}\right)\mid S_t=s_t\right]\text{\hspace{1em}(1.3)}$$

proof:
$$\begin{array}{rl}{V}_{\pi }\left(s_t\right)& ={\mathbb{E}}_{A_t,{\mathcal{S}}_{t+1},{\mathcal{A}}_{t+1}}\left[U_t\mid S_t=s_t\right]\\ & ={\mathbb{E}}_{A_t,{\mathcal{S}}_{t+1},{\mathcal{A}}_{t+1}},\left[R_t+\gamma U_{t+1}|S_t=s_t\right]\\ & ={\mathbb{E}}_{A_t,{\mathcal{S}}_{t+1},{\mathcal{A}}_{t+1}}\left[R_t\mid S_t=s_t\right]+\gamma {\mathbb{E}}_{A_t,{\mathcal{S}}_{t+1},{\mathcal{A}}_{t+1}}\left[U_{t+1}\mid S_t=s_t\right]\\ & ={\mathbb{E}}_{A_t,S_{t+1}}\left[R_t\mid S_t=s_t\right]+\gamma {\mathbb{E}}_{S_{t+1}}\left[{\mathbb{E}}_{A_t{\mathcal{A}}_{t+1},{\mathcal{S}}_{t+2}}\left[U_{t+1}\mid S_{t+1},S_t=s_t\right]\mid S_t=s_t\right]\\ & ={\mathbb{E}}_{A_t,S_{t+1}}\left[R_t\mid S_t=s_t\right]+\gamma {\mathbb{E}}_{S_{t+1}}\left[E_{{\mathcal{A}}_{t+1},{\mathcal{S}}_{t+2}}\left[U_{t+1}\mid S_{t+1}\right]\mid S_t=s_t\right]马尔可夫性\\ & ={\mathbb{E}}_{A_t,S_{t+1}}\left[R_t\mid S_t=s_t\right]+\gamma {\mathbb{E}}_{S_{t+1}}\left[V_{\pi }(S_{t+1})\mid S_t=s_t\right]\\ & ={\mathbb{E}}_{A_t,S_{t+1}}\left[R_t\mid S_t=s_t\right]+\gamma {\mathbb{E}}_{A_t,S_{t+1}}\left[V_{\pi }(S_{t+1})\mid S_t=s_t\right]马尔可夫性\\ \text{证毕}& \end{array}$$
或者直接利用式$1.2$，两边同时对$A_t\sim \pi (\cdot |s_t)$求期望得
$$\begin{array}{rl}{\mathbb{E}}_{A_t\sim \pi (\cdot |s_t)}[Q_{\pi }\left(s_t,A_t\right)]& ={\mathbb{E}}_{A_t\sim \pi (\cdot |s_t)}[{\mathbb{E}}_{S_{t+1}}\left[R_t+\gamma \cdot V_{\pi }\left(S_{t+1}\right)\mid S_t=s_t,A_t\right]]\\ \Updownarrow & \\ {\mathbb{E}}_{A_t}[Q_{\pi }\left(S_t,A_t\right)\mid S_t=s_t]& ={\mathbb{E}}_{A_t}[{\mathbb{E}}_{S_{t+1}}\left[R_t+\gamma \cdot V_{\pi }\left(S_{t+1}\right)\mid S_t=s_t,A_t\right]\mid S_t=s_t]\\ & ={\mathbb{E}}_{S_{t+1},A_t}[R_t+\gamma \cdot V_{\pi }\left(S_{t+1}\right)\mid S_t=s_t]\end{array}$$

利用式$1.3$，进一步写出显示表达式可得
$$\begin{array}{rl}{V}_{\pi }(s_t)& ={\mathbb{E}}_{A_t,S_{t+1}}\left[R_t\mid S_t=s_t\right]+\gamma {\mathbb{E}}_{A_t,S_{t+1}}\left[V_{\pi }(S_{t+1})\mid S_t=s_t\right]\\ & ={\mathbb{E}}_{A_t}[{\mathbb{E}}_{S_{t+1}}[R_t\mid A_t,S_t=s_t]\mid S_t=s_t]+\gamma {\mathbb{E}}_{A_t}\left[{\mathbb{E}}_{S_{t+1}}\left[V_{\pi (S_{t+1})}\mid A_t,S_t=s_t\right]\mid S_t=s_t\right]\\ & =\sum _{A_t}\pi (a_t\mid s_t){\mathbb{E}}_{S_{t+1}}[R_t\mid A_t,S_t=s_t]+\gamma \sum _{A_t}\pi (a_t\mid s_t){\mathbb{E}}_{S_{t+1}}\left[V_{\pi (S_{t+1})}\mid A_t,S_t=s_t\right]\\ & =\sum _{A_t}\pi (a_t\mid s_t)\sum _{S_{t+1}}r\cdot p(s_{t+1}\mid s_t,a_t)+\gamma \sum _{A_t}\pi (a_t\mid s_t)\sum _{S_{t+1}}{V}_{\pi }(s_{t+1})\cdot p(s_{t+1}\mid s_t,a_t)\end{array}$$
其中$r=r(s_t,s_{t+1},a_t)$

四.最优贝尔曼方程

Theorem :假设 $R_t$ 是 $S_t、A_t、S_{t+1}$ 的函数。那么
$$Q_{\star }\left(s_t,a_t\right)={\mathbb{E}}_{S_{t+1}\sim p\left(\cdot \mid s_t,a_t\right)}\left[R_t+\gamma \cdot \underset{A\in \mathcal{A}}{\max }{Q}_{\star }\left(S_{t+1},A\right)\mid S_t=s_t,A_t=a_t\right]\text{\hspace{1em}(1.4)}$$

由贝尔曼方程可知
$$Q_{\star }\left(s_t,a_t\right)={\mathbb{E}}_{S_{t+1},A_{t+1}}\left[R_t+\gamma \cdot Q_{\star }\left(S_{t+1},A_{t+1}\right)\mid S_t=s_t,A_t=a_t\right]$$
因为动作 $A_{t+1}={\mathrm{argmax}}_A{Q}_{\star }\left(S_{t+1},A\right)$ 是状态 $S_{t+1}$ 的确定性函数, 所以
$$Q_{\star }\left(s_t,a_t\right)={\mathbb{E}}_{S_{t+1}}\left[R_t+\gamma \cdot \underset{A\in \mathcal{A}}{\max }{Q}_{\star }\left(S_{t+1},A\right)\mid S_t=s_t,A_t=a_t\right]$$