Summary: MIT’s Improbable AI Lab has developed a new multimodal framework called Compositional Foundation Models for Hierarchical Planning (HiP) to assist robots in making detailed, feasible plans. Unlike previous models that relied on paired vision, language, and action data, HiP utilizes three different foundation models trained on different data modalities to capture various aspects of the decision-making process. This approach eliminates the need for expensive paired data and makes the reasoning process more transparent. The research team believes that HiP could enable robots to perform household chores and complex tasks in construction and manufacturing. The system outperformed comparable frameworks in tests by adapting its plans based on new information and accurately completing manipulation tasks. HiP’s hierarchy involves a large language model for task planning, a video diffusion model to understand the environment, and an egocentric action model to determine execution based on the surroundings.
Robots have always faced challenges in planning and executing tasks that humans find intuitive. While humans can effortlessly perform step-by-step chores, robots require a complex plan that involves detailed outlines. MIT’s Improbable AI Lab has tackled this issue with their innovative multimodal framework, HiP.
By leveraging three different foundation models, HiP enhances robotic decision-making and planning. Unlike previous models that rely on paired data, HiP’s foundation models capture distinct aspects of the decision-making process and work together when making decisions. This eliminates the need for difficult-to-obtain paired data, making the reasoning process more transparent.
The possibilities for HiP are vast. The research team envisions robots using this framework to accomplish household chores like putting away books or placing dishes in the dishwasher. Additionally, HiP could assist in complex tasks such as construction and manufacturing by stacking and placing different materials in specific sequences.
In tests, HiP outperformed comparable frameworks by adapting its plans to new information and accurately completing manipulation tasks. For example, in one test, the robot successfully stacked differently colored blocks and adjusted its plans to accommodate missing colors. In another test, the system arranged objects while ignoring unnecessary items and adapted its plans to deal with dirty objects.
HiP operates as a hierarchy, with each component pre-trained on different sets of data. A large language model starts the process by breaking down the task into sub-goals, while a video diffusion model collects physical information about the environment. Finally, an egocentric action model determines the appropriate actions based on the robot’s surroundings.
With HiP, robots can now rely on a multimodal approach that incorporates linguistic, physical, and environmental intelligence. This new framework opens up opportunities for improved robotic decision-making and the successful execution of complex tasks, making robots more efficient and capable companions in various settings.
The source of the article is from the blog publicsectortravel.org.uk