Reinforcement learning (RL) is the branch of machine learning concerned with learning to act. Instead of learning from a fixed dataset of labelled examples, an RL agent learns by interacting with an environment: it takes actions, observes the consequences, and gradually improves its behaviour to maximise a numerical reward. It is the framework behind game-playing systems that beat world champions, robots that learn to walk, recommendation systems that optimise long-term engagement, and — most importantly for this course — the alignment of the large language models you use every day.
Educational note: This course is for learning. Any preference data, human feedback, or training corpus you use in practice must respect data-protection law (GDPR in the EU), dataset licences, and copyright. Alignment work carries real ethical weight — you are shaping how a system behaves toward people — so we return to legal and safety obligations throughout the course. Nothing here is legal advice.
Supervised, unsupervised, and reinforcement learning
It helps to place RL against the other two paradigms. In supervised learning you have inputs paired with correct outputs, and the model learns to reproduce the mapping. In unsupervised learning you have inputs with no labels, and the model finds structure. Reinforcement learning is different in three fundamental ways:
- There is no supervisor, only a reward signal. Nobody tells the agent the correct action. It only receives a scalar reward that says how good the outcome was, and it must figure out which actions led to that reward.
- Feedback can be delayed. An action taken now may only pay off many steps later. Winning a chess game depends on moves made long before the final position. This is the credit assignment problem, and it haunts every algorithm we will study.
- Data is not independent and identically distributed. The agent's own choices determine what it sees next. Explore poorly and you never even observe the states where the good rewards live.
A comparison table makes the contrasts concrete:
| Aspect | Supervised | Unsupervised | Reinforcement |
|---|---|---|---|
| Training signal | Correct label per example | None (structure only) | Scalar reward, possibly delayed |
| Data distribution | Fixed, i.i.d. | Fixed, i.i.d. | Depends on the agent's own behaviour |
| Goal | Reproduce a mapping | Discover structure | Maximise cumulative reward |
| Typical failure | Overfitting to labels | Meaningless clusters | Reward hacking, instability, poor exploration |
| Example | Spam classification | Topic clustering | Game playing, robotics, LLM alignment |
These properties make RL powerful and also notoriously difficult. Much of this course is about the algorithms and engineering tricks that make it work anyway.
The agent-environment loop
At the heart of RL is a simple loop. At each time step t the agent observes a state s_t, chooses an action a_t, and the environment responds with a reward r_{t+1} and a new state s_{t+1}. The agent's goal is to choose actions that maximise the cumulative reward over time, not just the immediate one.