Goal Representations for Instruction Following (GRIF), a novel semi-supervised approach for training robots, has been proposed by a group of scientists led by Vivek Myers, Andre He, and others. This approach aims to bridge the gap between the use of natural language task specifications in language-conditioned behavioral cloning (LCBC) and the performance advantages of goal-conditioned learning.
GRIF works by integrating two key capabilities of a robot: grounding a language instruction in the physical environment and executing a sequence of actions to complete the assigned task. These capabilities can be developed independently, with vision-language data sourced from non-robotic sources aiding in grounding languages, and unlabeled robot trajectories being used for training the robot to reach specific end states.
GRIF suggests using visual goals as a form of task specification that can be scaled and generated using hindsight relabeling. Despite the advantage of training policies on large amounts of unstructured trajectory data, this method is found to be less intuitive for human users who typically find it easier to describe tasks through natural language.
To tackle this, an interface that combines goal-conditioned policies with the ease of task specification through language was introduced. The method, designed to digest large unstructured robotic data sets and generalize to diverse instructions using vision-language data, consists of a language encoder, goal encoder, and a policy network.
In tests, GRIF outperformed existing methods including LCBC, LangLfP, and BC-Z. Using the method, robots were able to effectively utilize large amounts of unlabeled trajectory data to learn goal-conditioned policies. The researchers noted that the current version of GRIF had limitations like struggling with tasks where instructions indicate more about how to do the task than what to do.
The research team suggests future studies could extend the method’s alignment loss to utilize human video data to learn richer semantics from Internet-scale data. This development could potentially improve grounding on language outside the robot dataset, and enable more generalizable robot policies that can follow user instructions.
The research has significant implications for future robot learning, paving the way for the creation of generalist robots that can perform tasks on humans’ behalf, with instructions communicated through natural language.
