Researchers from the Georgia Institute of Technology and the University of California, San Diego, have introduced an innovative model-based reinforcement learning algorithm called Policy learning with Large World Models (PWM). Traditional reinforcement learning methods have faced difficulties with multitasking, especially across different robotic forms. PWM tackles these issues by pretraining world models on offline data, then using them for first-order gradient policy learning. This approach can solve tasks with up to 152 action dimensions and achieve up to 27% higher rewards without the necessity for costly online planning.
Reinforcement learning (RL) approaches include model-based and model-free methods. Model-free methods include options like Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC), which utilize actor-critic architectures. PPO relies on zeroth-order gradients, while SAC uses first-order gradients, each with their own unique benefits and drawbacks. There has been a recent shift in the robotics field to large multi-task models trained via behavior cloning, such as the RT-1 and RT-2 for object manipulation.
PWM leverages these large world models and efficiently optimizes policies by using first-order gradients, leading to reduced variance and enhanced improvements in sample efficiency. Their tests, focusing on environments like Hopper, Ant, Anymal, Humanoid, and muscle-actuated Humanoid, showed that PWM yielded higher rewards and smoother optimization landscapes compared to the Study of Human in the Loop Computational (SHAC) model and the time-based decision-making controller (TD-MPC2).
Further tests on 30 and 80 multi-task environments demonstrated PWM’s superior reward performance and faster inference time than TD-MPC2. Nevertheless, PWM heavily relies on extensive pre-existing data for world model training, which restricts its applicability in low-data settings. Also, while PWM offers efficient policy training, it requires re-training for each new task, indicating challenges for quick adaptation. Future research could aim at enhancing world model training and extending PWM to image-based environments and real-world applications.
Overall, the study highlights the potential of large multi-task world models as differentiable physics simulators and the possibility of using PWM in reinforcement learning to improve efficiency and performance in complex and multi-task environments. However, it also outlines the setbacks and the need for future improvements and applications.
