Reinforcement Learning: Theory Notes

Reinforcement Learning

Rather than explicitly modelling the dynamics of the robot and its environment, reinforcement learning (RL) allows an autonomous system (agent) to learn a control strategy (policy) through interaction with its environment.

At each timestep, the agent observes the current state of the environment, chooses an action according to its policy, and receives a reward indicating how useful that action was.

This interaction is typically modelled through a Markov Decision Process (MDP). More formally, a length-$T$ MDP is a tuple:

$$\mathcal{M} = \langle \mathcal{S}, \mathcal{A}, \mathcal{D}, r, \gamma, \rho, T \rangle$$

where:

• $\mathcal{S}$ is the state space. A state $s_t$ describes the state of the robot and environment at timestep $t$. For a robot, this might include joint positions, joint velocities, and camera observations.
• $\mathcal{A}$ is the action space. An action $a_t$ is a command sent to the robot, such as a joint velocity, torque, or target joint position.
• $\mathcal{D}$ represents the environment dynamics: how the current state and action lead to the next state.
• $r$ is the reward function, which assigns a scalar reward to a transition.
• $\gamma$ is the discount factor, which determines how much the agent values future rewards compared to immediate rewards.
• $\rho$ describes the distribution of possible starting states.
• $T$ is the number of timesteps in the trajectory.

As the agent interacts with the environment over time, it generates a trajectory:

$$\tau = (s_0,a_0,r_0,s_1,a_1,r_1,\dots,s_{T-1},a_{T-1},r_{T-1},s_T)$$

A natural question is: what is the probability that a particular trajectory $\tau$ occurs? In general, the probability of what happens next could depend on the entire history of everything that has happened before it. For example, the probability of reaching $s_{t+1}$ might depend on

$$P(s_{t+1}\mid s_t,a_t,s_{t-1},a_{t-1},\dots,s_0,a_0)$$

and the action chosen by the agent could similarly depend on its entire history:

$$P(a_t\mid s_t,a_{t-1},s_{t-1},\dots,s_0,a_0)$$

The Markov assumption simplifies this by assuming that the current state $s_t$ contains all the information we need about the past. Therefore, the next state only depends on the current state and action:

$$P(s_{t+1}\mid s_t,a_t)$$

and the agent's action only depends on the current state:

$$P(a_t\mid s_t)$$

This lets us factor the probability of an entire trajectory into a much simpler expression:

$$P(\tau) = P(s_0) \prod_{t=0}^{T-1} P(s_{t+1}\mid s_t,a_t)\,P(a_t\mid s_t)$$

We already defined the initial-state distribution as $\rho$, the environment dynamics as $\mathcal{D}$, and the agent's policy as $\pi$, so we can rewrite this as:

$$P(\tau) = \rho(s_0) \prod_{t=0}^{T-1} \mathcal{D}(s_t,a_t,s_{t+1})\,\pi_\theta(a_t\mid s_t)$$

We often write the policy as $\pi_\theta$ because it is parameterized by learnable parameters $\theta$, which are optimized to maximize the return associated with the trajectories the policy produces.

To evaluate how good a trajectory is, we define its return as the sum of the rewards collected over the trajectory:

$$G(\tau) = \sum_{t=0}^{T-1}\gamma^t r_t$$

The discount factor $\gamma$ determines how strongly we value rewards further into the future.

However, a policy can produce many different trajectories, so rather than optimizing the return of one individual trajectory, we care about the expected return across trajectories generated by the policy:

$$J(\pi_\theta) = \mathbb{E}_{\tau \sim P_{\theta;\mathcal{D}}}\left[G(\tau)\right]$$

The goal of reinforcement learning is therefore to find the policy parameters $\theta$ that maximize this expected return.

Unlike classical model-based approaches, reinforcement learning does not require us to explicitly derive the dynamics of the robot and its environment. Instead, the policy can learn directly from interaction data, allowing it to handle dynamics that are difficult to model and potentially learn directly from high-dimensional sensor observations such as images.

Learning-based approaches also provide a way to improve performance as more data becomes available, rather than requiring us to manually redesign a controller for every new condition or task.

Despite these advantages, reinforcement learning introduces its own limitations. Learning through interaction can require large amounts of experience, and collecting that experience on a physical robot can be slow, expensive, and potentially unsafe. Simulation makes it possible to generate experience much more easily, but introduces a gap between simulated and real-world physics. RL also depends heavily on the reward signal: poorly specified or sparse rewards can make learning difficult or lead to unintended behaviours.

Soft Actor-Critic (SAC)

Now we know in RL, we need a way to learn the parameters for the policy. For my experiment, I used Soft Actor-Critic (SAC), which is an off-policy actor-critic RL algorithm designed for continuous action spaces.^[1]

SAC is good for robotics because it handles continuous control well, reuses data efficiently through replay buffers, and integrates nicely with demonstration- and intervention-based training.

SAC learns two main components: an actor and a critic. The actor is just our policy:

$$\pi_\theta(a\mid s)$$

Given the current state $s$, it determines which action $a$ the robot should take.

The critic learns a "Q-function":

$$Q_\phi(s,a)$$

Which estimates how much future reward we can expect if we take action $a$ in state $s$ and continue following our policy afterward.

Every time the robot interacts with the environment it generates transitions:

$$(s_t,a_t,r_t,s_{t+1})$$

And these are stored in the "replay buffer".

SAC repeatedly samples transitions from the replay buffer to train the critic. So for a transition $(s_t,a_t,r_t,s_{t+1})$, the basic idea is that the value of taking the current action should be equal to the reward we receive now plus the expected value of what happens afterward:

$$Q(s_t,a_t) \approx r_t + \gamma Q(s_{t+1},a_{t+1})$$

Once the critic can estimate which actions are valuable, the actor can then be trained to produce actions that receive higher Q-values.

So together, the critic learns which actions are good, while the actor learns to choose those actions.

One of the key characteristics of SAC is that the actor does not output one deterministic action, it is stochastic. This is typically represented using a Gaussian distribution:

$$a_t \sim \mathcal{N}\left(\mu_\theta(s_t),\sigma_\theta(s_t)\right)$$

This allows the policy to explore different actions instead of immediately committing to the first behaviour that appears to work. SAC explicitly encourages this exploration by modifying the reward objective with an entropy term:

$$Q(s_t,a_t) \approx r_t + \gamma\left[ Q(s_{t+1},a_{t+1}) - \alpha\log\pi_\theta(a_{t+1}\mid s_{t+1}) \right]$$

The $\alpha$ term controls how strongly exploration is encouraged. A larger $\alpha$ encourages a more stochastic policy, while a smaller $\alpha$ focuses more strongly on maximizing reward.^[1]

HIL-SERL

We can extend SAC into Human-in-the-Loop Sample-Efficient Reinforcement Learning (HIL-SERL). Instead of forcing the robot to discover the task entirely through random exploration, HIL-SERL begins by collecting a small number of successful human demonstrations.^[2] These demonstrations generate the same transitions we saw before:

$$(s_t,a_t,r_t,s_{t+1})$$

but now, the action $a_t$ can be influenced by a human! The human-in-the-loop part comes in when the policy begins to fail. A human supervisor can temporarily take control of the robot, perform a corrective action, and then return control back to the policy.^[2]

So the data used by SAC now comes from three sources:

$$\text{initial demonstrations} + \text{autonomous experience} + \text{human interventions}$$