4 April 2022

SayCan: filtering language model suggestions through robot skill affordances

A team of researchers showed that combining a large language model with pretrained low-level robot skills lets a mobile manipulator complete long-horizon tasks from abstract natural language instructions, with the robot's physical capabilities constraining what the model is allowed to suggest.

Manipulation and dexterity Language models Reinforcement learning Agents and planning Foundational method Demonstrated

Background

Large language models carry a great deal of knowledge about how tasks are done. Ask one how to clean a spill and it will give a sensible answer. The problem is that the answer may bear no relation to what a particular robot, in a particular room, can actually do. The model has no body. It cannot see the environment, and it has no idea which of its suggestions are physically possible.

At the same time, robot skills trained through reinforcement learning can be very good at narrow, specific actions. What they lack is any understanding of when to use those actions, or how to chain them together to accomplish something a person would phrase in a single sentence. Neither piece, on its own, was enough.

What happened

Michael Ahn, Anthony Brohan, and more than forty colleagues submitted a paper to arXiv on 4 April 2022 describing a system for grounding language models in robotic affordances. The core idea is that a large language model proposes what to do next, but each candidate action is scored by a value function, a learned estimate of how likely that action is to succeed given the current state of the environment. Only actions that are both semantically appropriate and physically feasible rise to the top.

The authors described the arrangement plainly: “the robot can act as the language model’s hands and eyes, while the language model supplies high-level semantic knowledge about the task.” The language model handles the reasoning about what a task requires. The value functions handle the question of what is actually doable right now.

Low-level skills were combined with large language models so that “the language model provides high-level knowledge about the procedures for performing complex and temporally-extended instructions, while value functions associated with these skills provide the grounding necessary to connect this knowledge to a particular physical environment.” That two-way constraint is what makes the system work: neither component is running free.

The team tested this on a mobile manipulator performing real-world tasks. The paper reports that the approach completed long-horizon, abstract, natural language instructions within a fixed library of trained skills in a single real-world kitchen-like environment. A second version of the paper, posted in August 2022, added results with PaLM, a study of new capabilities including drawer manipulation and multilingual instructions, an ablation study varying language model size, and an open-source version running on a simulated tabletop.

Why it mattered

Before this work, existing approaches had not demonstrated this combination at this scale on a real mobile manipulator in a real environment. By having skill value functions filter what the language model could propose, the system could complete complex, multi-step instructions that neither component could handle alone. The method demonstrated one scalable approach to connecting large language models to real robot affordances via learned value functions.

Sources

Do As I Can, Not As I Say: Grounding Language in Robotic Affordances. arxiv.org. Primary source
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances. arxiv.org. Primary source

Cite this page

AI Achievements. (2022). SayCan: filtering language model suggestions through robot skill affordances. Retrieved 2026-08-29, from https://achievements.ai/milestone/saycan-filtering-language-model-suggestions-through-robot

@misc{achievements_saycan_filtering_language_model_suggestions_through_robot,
  title  = {SayCan: filtering language model suggestions through robot skill affordances},
  author = {{AI Achievements}},
  year   = {2022},
  url    = {https://achievements.ai/milestone/saycan-filtering-language-model-suggestions-through-robot}
}