the discounted return
\[\begin{aligned}
G_t & =R_{t+1}+\gamma R_{t+2}+\gamma^2 R_{t+3}+\ldots \\
& =R_{t+1}+\gamma\left(R_{t+2}+\gamma R_{t+3}+\ldots\right) \\
& =R_{t+1}+\gamma G_{t+1}
\end{aligned}
\]
state-value function/the state value of s \(v_\pi(s)\)
\[\begin{aligned}
v_\pi(s) & =\mathbb{E}\left[G_t \mid S_t=s\right] \\
& =\mathbb{E}\left[R_{t+1}+\gamma G_{t+1} \mid S_t=s\right] \\
& =\mathbb{E}\left[R_{t+1} \mid S_t=s\right]+\gamma \mathbb{E}\left[G_{t+1} \mid S_t=s\right]
\end{aligned}
\]
Bellman Equation
\[\begin{aligned}
v_\pi(s) & =\mathbb{E}\left[R_{t+1} \mid S_t=s\right]+\gamma \mathbb{E}\left[G_{t+1} \mid S_t=s\right], \\
& =\underbrace{\sum_{a \in \mathcal{A}} \pi(a \mid s) \sum_{r \in \mathcal{R}} p(r \mid s, a) r}_{\text {mean of immediate rewards }}+\underbrace{\gamma \sum_{a \in \mathcal{A}} \pi(a \mid s) \sum_{s^{\prime} \in \mathcal{S}} p\left(s^{\prime} \mid s, a\right) v_\pi\left(s^{\prime}\right),}_{\text {mean of future rewards }} \\
& =\sum_{a \in \mathcal{A}} \pi(a \mid s)\left[\sum_{r \in \mathcal{R}} p(r \mid s, a) r+\gamma \sum_{s^{\prime} \in \mathcal{S}} p\left(s^{\prime} \mid s, a\right) v_\pi\left(s^{\prime}\right)\right], \quad \text { for all } s \in \mathcal{S} .
\end{aligned}
\]
two equivalent expressions
First
First, it follows from the law of total probability that
\[\begin{aligned}
& p\left(s^{\prime} \mid s, a\right)=\sum_{r \in \mathcal{R}} p\left(s^{\prime}, r \mid s, a\right), \\
& p(r \mid s, a)=\sum_{s^{\prime} \in \mathcal{S}} p\left(s^{\prime}, r \mid s, a\right) .
\end{aligned}
\]
Then, equation (2.7) can be rewritten as
\[v_\pi(s)=\sum_{a \in \mathcal{A}} \pi(a \mid s) \sum_{s^{\prime} \in \mathcal{S}} \sum_{r \in \mathcal{R}} p\left(s^{\prime}, r \mid s, a\right)\left[r+\gamma v_\pi\left(s^{\prime}\right)\right]
\]
Second
Second, the reward \(r\) may depend solely on the next state \(s^{\prime}\) in some problems. As a result, we can write the reward as \(r\left(s^{\prime}\right)\) and hence \(p\left(r\left(s^{\prime}\right) \mid s, a\right)=p\left(s^{\prime} \mid s, a\right)\), substituting which into \((2.7)\) gives
\[v_\pi(s)=\sum_{a \in \mathcal{A}} \pi(a \mid s) \sum_{s^{\prime} \in \mathcal{S}} p\left(s^{\prime} \mid s, a\right)\left[r\left(s^{\prime}\right)+\gamma v_\pi\left(s^{\prime}\right)\right]
\]