Q-Learning

Introduction

Q-learning learns action values Q(s,a) from experience without a model of the environment. Updates push estimates toward observed reward plus discounted best next action.

Understanding the topic

Exploration epsilon-greedy balances trying new actions vs exploiting known good ones.

Tabular limits Works for small state spaces; large problems need function approximation.

• Exploration — epsilon-greedy balances trying new actions vs exploiting known good ones.
• Tabular limits — Works for small state spaces; large problems need function approximation.

Step-by-step explanation

1. Exploration — epsilon-greedy balances trying new actions vs exploiting known good ones.
2. Tabular limits — Works for small state spaces; large problems need function approximation.

Informative example

Python starter:

# simplified update (tabular)
# Q[s,a] += alpha * (reward + gamma * max(Q[s_next]) - Q[s,a])
alpha, gamma = 0.5, 0.9
Q_sa, reward, max_next = 0.0, 1.0, 0.8
Q_sa += alpha * (reward + gamma * max_next - Q_sa)
print(round(Q_sa, 2))

0.61

Execution workflow

Best practices

• Hold out a test set before hyperparameter tuning.
• Scale numeric columns for distance-based models.
• Track multiple metrics — not accuracy alone on skewed labels.

Common mistakes

• Leaking test statistics into preprocessing fit on full data.
• Training on the same rows you report as test performance.
• Chasing complex models before a simple baseline.

Hands-on exercise

Practice:

• Implement epsilon-greedy on a grid world
• Log episode return over training

Summary

Q-Learning — Model-free action-value updates.