Reinforcement Learning with Code

This note records how the author begin to learn RL. Both theoretical understanding and code practice are presented. Many material are referenced such as ZhaoShiyu’s Mathematical Foundation of Reinforcement Learning, .

Chapter 7. Temporal-Difference Learning

​ Temporal-difference (TD) algorithms can be seen as special Robbins-Monro (RM) algorithm solving expectation form of Bellman or Bellman optimality equation.

7.1 TD leaning of state value

​ Recall the Bellman equation in section 2.2

$$\begin{gathered} v_{\pi }(s)=\sum _a\pi (a|s)(\sum _{r}p(r|s,a)r+\gamma \sum _{s^{\prime }}p(s^{\prime }|s,a)v_{\pi }(s^{\prime }))\text{(elementwise form)} \\ {v}_{\pi }=r_{\pi }+\gamma P_{\pi }{v}_{\pi }\text{(matrix-vector form)} \end{gathered}$$

where

$$[r_{\pi }{]}_s\triangleq \sum _a\pi (a|s)\sum _{r}p(r|s,a)r[P_{\pi }{]}_{s,s^{\prime }}=\sum _a\pi (a|s)\sum _{s^{\prime }}p(s^{\prime }|s,a)$$

​ Recall the definition of state value: state value is defined as the mean of all possible returns starting from a state, which is actually the expectation of return from a specific state. We can rewrite the above equation into

$$\begin{gathered} v_{\pi }=\mathbb{E}[R+\gamma v_{\pi }]\text{(matrix-vector form)} \\ {v}_{\pi }(s)=\mathbb{E}[R+\gamma v_{\pi }(S^{\prime })|S=s],s\in \mathcal{S}\text{(elementwise form)} \end{gathered}$$

where $S,S^{\prime }$ and $R$ are the random variables representing the current state, next state and immediate reward. This equation also called the Bellman expectation equation.

​ We can use Robbins-Monro algorithm introduced in chapter 6 to solve the Bellman expectation equation. Reformulate the problem that is to find the root $v_{\pi }(s)$ of equation $g(v_{\pi }(s))=v_{\pi }(s)-\mathbb{E}[R+\gamma v_{\pi }(S^{\prime })|S=s]=0$, where $S^{\prime }$ is iid with $s$. We can only get the measurement with noise ​

$$\begin{array}{rl}\tilde{g}(v_{\pi }(s),\eta )& =v_{\pi }(s)-(r+\gamma v_{\pi }(s^{\prime }))\\ & =\underset{g(v_{\pi }(s))}{\underbrace{v_{\pi }(s)-\mathbb{E}[R+\gamma v_{\pi }(S^{\prime })|S=s]}}+\underset{\eta }{\underbrace{(\mathbb{E}[R+\gamma v_{\pi }(S^{\prime })|S=s]-[r+\gamma v_{\pi }(s^{\prime })])}}\end{array}$$

Hence, according to the Robbins-Monro algorithm, we can get the TD learning algorithm as

$$v_{k+1}(s)=v_k(s)-{\alpha }_k(v_k(s)-[r_k+\gamma v_{\pi }(s_k^{\prime })])$$

We do some modification in order to remove some assumptions of TD learning. One modification is the sample data $\{(s,r_k,s_k^{\prime })\}$ is changed to $\{(s_t,r_{t+1},s_{t+1})\}$. Due to the modification the algorithm is called temporal-difference learning. Rewrite it in a more concise way:

$$\text{TD learning}:\{\begin{array}{rl}\underset{\text{new estimation}}{\underbrace{v_{t+1}(s_t)}}& =\underset{\text{current estimation}}{\underbrace{v_t(s_t)}}-{\alpha }_t(s_t)\stackrel{\text{TD error or Innovation }{\delta }_t}{\overbrace{[v_t(s_t)-\underset{\text{TD target }{\bar{v}}_t}{\underbrace{(r_{t+1}+\gamma v_t(s_{t+1}))}}]}}\\ v_{t+1}(s)& =v_t(s),\text{for all }s\ne s_t\end{array}$$

where $t=0,1,2,\dots$. Here, $v_t(s_t)$ is the estimated state value of $v_{\pi }(s_t)$ and $a_t(s_t)$ is the learning rate of $s_t$ at time $t$. And

$${\bar{v}}_t\triangleq r_{t+1}+\gamma v(s_{t+1})$$

is called the TD target and

$${\delta }_t\triangleq v(s_t)-[r_{t+1}+\gamma v(s_{t+1})]=v(s_t)-{\bar{v}}_t$$

is called the TD error. TD error reflects the deficiency between the current estimate $v_t$ and the true state value $v_{\pi }$.

7.2 TD learning of action value: Sarsa

​ Sarsa is an algorithm directly estimate action values. Estimating action value is important because the policy can be improved based on action values.

​ Recall the Bellman equation of action value in section 2.5

$$\begin{array}{rl}{q}_{\pi }(s,a)& =\sum _{r}p(r|s,a)r+\gamma \sum _{s^{\prime }}p(s^{\prime }|s,a)v_{\pi }(s^{\prime })\text{(elementwise form)}\\ & =\sum _{r}p(r|s,a)r+\gamma \sum _{s^{\prime }}p(s^{\prime }|s,a)\sum _{a^{\prime }\in \mathcal{A}(s^{\prime })}\pi (a^{\prime }|s^{\prime })q_{\pi }(s^{\prime },a^{\prime })\text{(elementwise form)}\end{array}$$

Due to the conditional probability $p(a,b)=p(b)p(a|b)$, we have

$$\begin{array}{rl}p(s^{\prime },a^{\prime }|s,a)& =p(s^{\prime }|s,a)p(a^{\prime }|s^{\prime },s,a)\text{(conditional probility)}\\ & =p(s^{\prime }|s,a)p(a^{\prime }|s^{\prime })\text{(due to conditional independence)}\\ & =p(s^{\prime }|s,a)\pi (a^{\prime }|s^{\prime })\end{array}$$

Due to the above equation, we have

$$q_{\pi }(s,a)=\sum _{r}p(r|s,a)r+\gamma \sum _{s^{\prime }}\sum _{a^{\prime }}p(s^{\prime },a^{\prime }|s,a)q_{\pi }(s^{\prime },a^{\prime })$$

Regard the probability $p(r|s,a)$ and $p(s^{\prime },a^{\prime }|s,a)$ as the distribution of random variable $R$ and $S^{\prime }$ respectively. Then rewrite above equation into expectation form

$$q_{\pi }(s,a)=\mathbb{E}[R+\gamma q_{\pi }(S^{\prime },A^{\prime })|S=s,A=a],\text{ for all }s,a\text{(expectation form)}$$

where $R,S,S^{\prime }$ are random variables, denote immediate reward, currnet state and next state respectively.

Hence, we can use the Robbins-Monro algorithm to sovle the Bellamn eqaution of action value. We can define

$$g(q_{\pi }(s,a))\triangleq q_{\pi }(s,a)-\mathbb{E}[R+\gamma q_{\pi }(S^{\prime },A^{\prime })|S=s,A=a]$$

We only can get the observation with noise that

$$\begin{array}{rl}\tilde{g}(q_{\pi }(s,a),\eta )& =q_{\pi }(s,a)-[r+\gamma q_{\pi }(s^{\prime },a^{\prime })]\\ & =\underset{g(q_{\pi }(s,a))}{\underbrace{q_{\pi }(s,a)-\mathbb{E}[R+\gamma q_{\pi }(S^{\prime },A^{\prime })|S=s,A=a]}}+\underset{\eta }{\underbrace{\left[\mathbb{E}\right[R+\gamma q_{\pi }(S^{\prime },A^{\prime })|S=s,A=a]-[r+\gamma q_{\pi }(s^{\prime },a^{\prime })]]}}\end{array}$$

Hence, according to the Robbins-Monro algorithm, we can get Sarsa as

$$q_{k+1}(s,a)=q_k(s,a)-{\alpha }_k[q_k(s,a)-(r_k+\gamma q_k(s_k^{\prime },a_k^{\prime })\left)\right]$$

Similar to the TD learning estimates state value in last section, we do some modification in above equation. The sampled data $(s,a,r_k,s_k^{\prime },a_k^{\prime })$ is changed to $(s_t,a_t,r_{t+1},s_{t+1},a_{t+1})$. Hence, the Sarsa becomes

$$\text{Sarsa}:\{\begin{array}{rl}{q}_{t+1}(s_t,a_t)& =q_t(s_t,a_t)-{\alpha }_t(s_t,a_t)[q_t(s_t,a_t)-(r_{t+1}+\gamma q_t(s_{t+1},a_{t+1}))]\\ q_{t+1}(s,a)& =q_t(s,a),\text{for all }(s,a)\ne (s_t,a_t)\end{array}$$

where $t=0,1,2,\dots$. Here, $q_t(s,a_t)$ is the estimated action value of $(s_t,a_t)$; ${\alpha }_t(s_t,a_t)$ is the learning rate depending on $s_t,a_t$.

​ Sarsa is nothing but an action-value version of the TD algorithm. Sarsa is also implemented with policy improvement steps such as $ϵ$- greedy algorithm. There is a point that should be noticed. In the $q$-value update step, unlike the model-based policy iteration or value iteration algorithm where the values of all states are updated in each iteration, Sarsa only updates a single state-action pair that is visited at time step $t$.

Pesudocode:

7.3 TD learning of action value: Expected Sarsa

​ Recall the Bellman equation of action value

$$q_{\pi }(s,a)=\sum _{r}p(r|s,a)r+\gamma \sum _{s^{\prime }}p(s^{\prime }|s,a)v_{\pi }(s^{\prime })\text{(elementwise form)}$$

Regard the probability $p(r|s,a)$ and $p(s^{\prime }|s,a)$ as the distribution of random variable $R$ and $v_{\pi }(S^{\prime })$. Then, we have the expectation form of Bellman equation of action value.

$$q_{\pi }(s,a)=\mathbb{E}[R+\gamma v_{\pi }(S^{\prime })|S=s,A=a]\text{(expectation form)}(1)$$

According to the definition of state value we have

$$\begin{array}{rl}\mathbb{E}[q_{\pi }(s,A)|s]& =\sum _{a\in \mathcal{A}(s)}\pi (a|s)q_{\pi }(s,a)=v_{\pi }(s)\\ \to \mathbb{E}[q_{\pi }(S^{\prime },A)|S^{\prime }]& =v_{\pi }(S^{\prime })(2)\end{array}$$

Subtitute $(2)$ into $(1)$ we have

$$q_{\pi }(s,a)=\mathbb{E}[R+\gamma \mathbb{E}[q_{\pi }(S^{\prime },A)|S^{\prime }]|S=s,A=a],\text{ for all }s,a\text{(expectation form)}$$

Rewirte it into root finding parttern

$$g(q_{\pi }(s,a))\triangleq q_{\pi }(s,a)-\mathbb{E}[R+\gamma \mathbb{E}[q_{\pi }(S^{\prime },A)|S^{\prime }]|S=s,A=a]$$

We can only get the observation with noise $\eta$

$$\begin{array}{rl}\tilde{g}(q_{\pi }(s,a),\eta )& =q_{\pi }(s,a)-(r+\gamma \mathbb{E}[q_{\pi }(s^{\prime },A)|s^{\prime }])\\ & =\underset{g(q_{\pi }(s,a))}{\underbrace{q_{\pi }(s,a)-\mathbb{E}[R+\gamma \mathbb{E}[q_{\pi }(S^{\prime },A)|S^{\prime }]|S=s,A=a]}}+\underset{\eta }{\underbrace{\mathbb{E}[R+\gamma \mathbb{E}[q_{\pi }(S^{\prime },A)|S^{\prime }]|S=s,A=a]-(r+\gamma \mathbb{E}[q_{\pi }(s^{\prime },A)|s^{\prime }])}}\end{array}$$

Hence, we can implement Robbins-Monro algorithm to find the root of $g(q_{\pi }(s,a))$ that

$$q_{k+1}(s,a)=q_k(s,a)-{\alpha }_k(s,a)[q_k(s,a)-(r_k+\gamma \mathbb{E}[q_k(s_k^{\prime },A)|s_k^{\prime }\left]\right)]$$

Similar to the TD learning estimates state value, we do some modification in above equation. The sampled data $(s,a,r_k,s_k^{\prime })$ is changed to $(s_t,a_t,r_{t+1},s_{t+1})$. Hence, the Expected-Sarsa becomes

$$\text{Expected-Sarsa}:\{\begin{array}{rl}{q}_{t+1}(s_t,a_t)& =q_t(s_t,a_t)-{\alpha }_t(s_t,a_t)[q_t(s_t,a_t)-(r_{t+1}+\gamma \mathbb{E}[q_t(s_{t+1},A|s_{t+1})])]\\ q_{t+1}(s,a)& =q_t(s,a),\text{for all }(s,a)\ne (s_t,a_t)\end{array}$$

7.4 TD learning of action values: $n$-step Sarsa

​ Recall the definition of action value is

$$q_{\pi }(s,a)=\mathbb{E}[G_t|S_t=s,A_t=a]$$

The discounted return $G_t$ can be written in different forms as

$$\begin{array}{rl}\text{Sarsa}⟵G_t^{(1)}& =R_{t+1}+\gamma q_{\pi }(S_{t+1},A_{t+1})\\ G_t^{(2)}& =R_{t+1}+\gamma R_{t+2}+{\gamma }^2{q}_{\pi }(S_{t+2},A_{t+2})\\ & ⋮\\ n\text{-step Sarsa}⟵G_t^{(n)}& =R_{t+1}+\gamma R_{t+2}+\cdots +{\gamma }^n{q}_{\pi }(S_{t+n},A_{t+n})\\ & ⋮\\ \text{Monte Carlo}⟵G_t^{(\infty )}& =R_{t+1}+\gamma R_{t+2}+{\gamma }^2{R}_{t+3}+\cdots \end{array}$$

It should be noted that $G_t=G_t^{(1)}=G_t^{(2)}=G_t^{(n)}=G_t^{(\infty )}$, where the superscripts merely indicate the different decomposition structures of $G_t$.

​ Sarsa aims to solve

$$q_{\pi }(s,a)=\mathbb{E}[G_t^{(1)}|s,a]=\mathbb{E}[R_{t+1}+\gamma q_{\pi }(S_{t+1},A_{t+1})|s,a]$$

MC learning aims to solve

$$q_{\pi }(s,a)=\mathbb{E}[G_t^{(\infty )}|s,a]=\mathbb{E}[R_{t+1}+\gamma R_{t+2}+{\gamma }^2{R}_{t+3}+\cdots |s,a]$$

$n$-step Sarsa aims to solve

$$q_{\pi }(s,a)=\mathbb{E}[G_t^{(n)}|s,a]=\mathbb{E}[R_{t+1}+\gamma R_{t+2}+\cdots +{\gamma }^n{q}_{\pi }(S_{t+n},A_{t+n})|s,a]$$

​ The $n$-step Sarsa algorithm is

$$n\text{-step Sarsa}:\{\begin{array}{rl}{q}_{t+1}(s_t,a_t)& =q_t(s_t,a_t)-{\alpha }_t(s_t,a_t)[q_t(s_t,a_t)-(r_{t+1}+\gamma r_{t+2}+\cdots +{\gamma }^n{q}_t(s_{t+n},a_{t+n}))]\\ q_{t+1}(s,a)& =q_t(s,a),\text{for all }(s,a)\ne (s_t,a_t)\end{array}$$

7.5 TD learning of optimal action values: Q-learning

​ It should be noted that Sarsa can only estimate the action values of a given policy. It must be combined with a policy improvement step to find optimal policies and hence their optimal action values. By contrast, Q-learning can directly estimate optimal action values.

​ Recall the Bellman optimal equation of state value in section 3.2

$$\begin{array}{rl}v(s)& =\underset{\pi }{\max }\sum _{a\in \mathcal{A}(s)}\pi (a|s)[\sum _{r}p(r|s,a)r+\gamma \sum _{s^{\prime }}p(s^{\prime }|s,a)v(s^{\prime })]\\ v(s)& =\underset{a\in \mathcal{A}(s)}{\max }[\sum _{r}p(r|s,a)r+\gamma \sum _{s^{\prime }}p(s^{\prime }|s,a)v(s^{\prime })]\end{array}$$

where $v(s)\triangleq {\max }_{a\in \mathcal{A}(s)}q(s,a)$. Hence we have

$$\begin{array}{rl}\underset{a\in \mathcal{A}(s)}{\max }q(s,a)& =\underset{a\in \mathcal{A}(s)}{\max }[\sum _{r}p(r|s,a)r+\gamma \sum _{s^{\prime }}p(s^{\prime }|s,a)v(s^{\prime })]\\ \underset{a\in \mathcal{A}(s)}{\max }q(s,a)& =\underset{a\in \mathcal{A}(s)}{\max }[\sum _{r}p(r|s,a)r+\gamma \sum _{s^{\prime }}p(s^{\prime }|s,a)\underset{a\in \mathcal{A}(s)}{\max }q(s^{\prime },a)]\\ \to q(s,a)& =\sum _{r}p(r|s,a)r+\gamma \sum _{s^{\prime }}p(s^{\prime }|s,a)\underset{a\in \mathcal{A}(s)}{\max }q(s^{\prime },a)\text{(elementwise form)}\end{array}$$

Rewrite it into expectation form

$$q(s,a)=\mathbb{E}[R+\gamma \underset{a\in \mathcal{A}(s)}{\max }q(S^{\prime },a)|S=s,A=a],\text{ for all }s,a\text{(expectation form)}$$

This equation is the Bellman optimal equation expressed in terms of action values.

Rewrite it into

$$g(q(s,a))\triangleq q(s,a)-\mathbb{E}[R+\gamma \underset{a\in \mathcal{A}(S^{\prime })}{\max }q(S^{\prime },a)|S=s,A=a]$$

We can get the observation with noise

$$\begin{array}{rl}\tilde{g}(q(s,a))& =q(s,a)-[r+\gamma \underset{a\in \mathcal{A}(s^{\prime })}{\max }q(s^{\prime },a)]\\ & =\underset{g(q(s,a))}{\underbrace{q(s,a)-\mathbb{E}[R+\gamma \underset{a\in \mathcal{A}(S^{\prime })}{\max }q(S^{\prime },a)|S=s,A=a]}}+\underset{\eta }{\underbrace{\mathbb{E}[R+\gamma \underset{a\in \mathcal{A}(S^{\prime })}{\max }q(S^{\prime },a)|S=s,A=a]-[r+\gamma \underset{a\in \mathcal{A}(s^{\prime })}{\max }q(s^{\prime },a)]}}\end{array}$$

Hence, we can implement Robbins-Monro algorithm to find the root

$$q_{k+1}(s,a)=q_k(s,a)-{\alpha }_k(s,a)[q_k(s,a)-(r_k+\gamma \underset{a\in \mathcal{A}(s^{\prime })}{\max }{q}_k(s^{\prime },a)\left)\right]$$

Similar to the TD learning estimates state value, we do some modification in above equation. The sampled data $(s,a,r_k,s_k^{\prime })$ is changed to $(s_t,a_t,r_{t+1},s_{t+1})$. Hence, the Q-learning becomes

$$\text{Q-learning}:\{\begin{array}{rl}{q}_{t+1}(s_t,a_t)& =q_t(s_t,a_t)-{\alpha }_t(s_t,a_t)[q_t(s_t,a_t)-(r_{t+1}+\gamma \underset{a\in \mathcal{A}(s_{t+1})}{\max }{q}_t(s_{t+1},a))]\\ q_{t+1}(s,a)& =q_t(s,a),\text{for all }(s,a)\ne (s_t,a_t)\end{array}$$

Off-policy vs on-policy:

​ There exist two policies in a TD learning task: behavior policy and target policy. The behavoir policy is used to generate experience samples. The target policy is constantly updated toward an optimal policy. When the behavior policy is the same as the target policy, such a kind of learning is called on-policy. Otherwise, when they are different, the learning is called off-policy.

​ The advantage of off-policy learning compared to on-policy learning is that it can search for optimal policies based on the experiences generated by any other policies.

​ How to determine a algorithm is on-policy or off-policy? The basic reason is that if the algorithm is sovling Bellman equation, then it’s on-policy. This is because Bellman equation is finding the state value or action value under a given policy. Else if the algorithm is sovling Bellman optimal equation, then it’s off-policy. This because Bellman equation does not include any policy, hence, the behavior policy and target policy can be different.

Online learning vs offline learning:

​ Online learning refers to the case where the value and policy can be updated once an experience sample is obtained. Offline learning refers to the case that the update can only be done after all experience samples have been collected. For example, TD learning is online whereas Monte Carlo learning is offline.

Persudocode:

(On-policy version)

(Off-policy version)

---

Reference

赵世钰老师的课程