← back to the work log

HIL-SERL in MuJoCo

last updated

Simulation

I explored the RL pipeline in MuJoCo simulator. The task was simple: train a simulated robot arm to pick up a cube. I first teleoperated the robot and collected 30 successful demonstrations of the task.

Collecting demonstrations through teleoperation in MuJoCo · 2× speed

These demonstrations provided useful transitions for the replay buffer and gave the policy examples of successful behaviour before autonomous training began.

I then trained the policy using SAC.[1] I also used human-in-the-loop interventions when a training run was taking too long or had flagged an error. I temporarily took control to correct the robot’s behaviour before handing control back to the policy, following the intervention approach used in HIL-SERL.[2]

After training, I ran an autonomous rollout of the learned policy.

Autonomous rollout of the learned policy in MuJoCo · 1× speed

The policy was able to control the simulated arm and successfully pick up the cube without human control.

This gave me a simple end-to-end example of how demonstrations, reinforcement learning, and human feedback can be combined before moving onto real-world robot learning.

sources (2) ▾
  1. [1]OpenAI, Spinning Up: Soft Actor-Critic
  2. [2]Luo et al., HIL-SERL: Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning