> ## Documentation Index
> Fetch the complete documentation index at: https://aegean.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# SARSA Gridworld Example
> SARSA algorithm on gridworld with Q-value heatmap visualization
It is instructive to apply SARSA to a small, well-known environment so the learning dynamics are visible end-to-end. We use the Gymnasium **CliffWalking-v1** environment described below.
## Environment: CliffWalking-v1
**[CliffWalking-v1](https://gymnasium.farama.org/environments/toy_text/cliff_walking/)** is the canonical Gymnasium gridworld for tabular control. The agent moves on a $4 \times 12$ grid: it starts at the bottom-left (state index 36) and must reach the bottom-right goal (state index 47). Every step yields reward $-1$, except stepping into any of the cells along the bottom edge between start and goal, *the cliff*, which yields reward $-100$ and resets the agent back to the start. The action space is `Discrete(4)` with the convention 0 = up, 1 = right, 2 = down, 3 = left, transitions are deterministic, and the observation is a single integer in $\{0, \ldots, 47\}$ encoding `row * 12 + col`. With only 48 states it is small enough to learn with a plain Q-table, while also being the textbook environment (Sutton & Barto, Example 6.6) used to contrast SARSA's *safe* path along the top edge against Q-learning's *optimal-but-risky* path adjacent to the cliff.
```python theme={null}
import os
from collections import defaultdict
import gymnasium as gym
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
from tqdm import tqdm
class SARSAgent:
"""Tabular SARSA for a discrete-state, discrete-action Gymnasium env.
Update: Q(S_t, A_t) <- Q(S_t, A_t) + alpha * (R_{t+1} + gamma * Q(S_{t+1}, A_{t+1}) - Q(S_t, A_t))
On terminal transitions, Q(S_{t+1}, .) is treated as 0.
"""
def __init__(self, n_actions, learning_rate=0.5, discount_factor=1.0, epsilon=0.1):
self.n_actions = n_actions
self.alpha = learning_rate
self.gamma = discount_factor
self.epsilon = epsilon
self.q_table = defaultdict(lambda: np.zeros(n_actions))
def get_action(self, state):
if np.random.rand() < self.epsilon:
return int(np.random.randint(self.n_actions))
return int(np.argmax(self.q_table[state]))
def learn(self, s, a, r, s_next, a_next, terminated):
target = r + (0.0 if terminated else self.gamma * self.q_table[s_next][a_next])
self.q_table[s][a] += self.alpha * (target - self.q_table[s][a])
```
```python theme={null}
env = gym.make("CliffWalking-v1")
n_actions = env.action_space.n
agent = SARSAgent(n_actions=n_actions, learning_rate=0.5, discount_factor=1.0, epsilon=0.1)
n_episodes = 500
episode_returns = np.empty(n_episodes)
for ep in tqdm(range(n_episodes), desc="Training"):
s, _ = env.reset()
a = agent.get_action(s)
total_r = 0.0
while True:
s_next, r, terminated, truncated, _ = env.step(a)
a_next = agent.get_action(s_next)
agent.learn(s, a, r, s_next, a_next, terminated)
total_r += r
s, a = s_next, a_next
if terminated or truncated:
break
episode_returns[ep] = total_r
env.close()
print(f"Final 50-episode mean return: {episode_returns[-50:].mean():.1f}")
```
```output theme={null}
Training: 0%| | 0/500 [00:00= 0:
ax.text(c + 0.5, r + 0.5, arrow_chars[policy[r, c]], ha="center", va="center", fontsize=18)
ax.set_xlim(0, N_COLS); ax.set_ylim(N_ROWS, 0)
ax.set_xticks(range(N_COLS + 1)); ax.set_yticks(range(N_ROWS + 1))
ax.set_aspect("equal"); ax.grid(True, alpha=0.3)
ax.set_title("Greedy policy from learned Q-table (S=start, G=goal, \u2620=cliff)")
plt.savefig(f"{images_dir}/sarsa_cliff_policy.png", dpi=150, bbox_inches="tight")
plt.show()
```
**Key references**: (Ma & Yu, 2016; Li, 2017; Bellemare et al., 2016; Jaderberg et al., 2016; Lillicrap et al., 2015)
## References
* Bellemare, M., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D., et al. (2016). *Unifying count-based exploration and intrinsic motivation*.
* Jaderberg, M., Mnih, V., Czarnecki, W., Schaul, T., Leibo, J., et al. (2016). *Reinforcement Learning with Unsupervised Auxiliary Tasks*.
* Li, Y. (2017). *Deep Reinforcement Learning: An Overview*.
* Lillicrap, T., Hunt, J., Pritzel, A., Heess, N., Erez, T., et al. (2015). *Continuous control with deep reinforcement learning*.
* Ma, S., Yu, J. (2016). *Transition-based versus State-based Reward Functions for MDPs with Value-at-Risk*.
***
[Edit this page on GitHub](https://github.com/aegean-ai/eaia/edit/main/src/aiml-common/lectures/reinforcement-learning/value-based-algorithms/control/sarsa/gridworld/sarsa_gridworld.mdx) or [file an issue](https://github.com/aegean-ai/eaia/issues/new/choose).