DeepMind, the renowned AI research lab, has recently presented groundbreaking research that showcases an AI agent capable of excelling in various tasks within 3D games that it has never encountered before. While DeepMind has previously focused on developing AI models for strategic games like Go and chess, as well as learning games without explicit rule instruction, this new achievement demonstrates the proficiency of an AI agent in understanding different gaming environments and executing tasks based on natural-language instructions.
To accomplish this feat, DeepMind collaborated with game studios such as Hello Games, Tuxedo Labs, and Coffee Stain, training their Scalable Instructable Multiworld Agent (SIMA) on a diverse set of nine games. In addition to these games, the researchers utilized four research environments, including a Unity-based environment where agents were directed to create sculptures using building blocks. Through this comprehensive approach, SIMA—an AI agent designed for 3D virtual settings—was exposed to a wide range of visual styles and perspectives, spanning from first-person to third-person views.
Each game within SIMA’s portfolio presents a unique interactive world with its own set of skills to master, including navigation, resource mining, spaceship piloting, and item crafting. The researchers at DeepMind emphasized that gaining proficiency in following directions and completing tasks across various video game environments could pave the way for the development of more adaptable AI agents capable of effectively operating in any setting.
To train SIMA, the researchers observed human gameplay and recorded the corresponding keyboard and mouse inputs used to perform actions. This data was then utilized to train SIMA, which utilizes “precise image-language mapping and a video model predicting on-screen actions.” Consequently, SIMA has the ability to comprehend diverse environments and execute tasks to achieve specific objectives.
What sets SIMA apart is that it does not require access to a game’s source code or API; it can operate on commercial versions of games using only two inputs: on-screen visuals and user instructions. By employing the same input method as humans—keyboard and mouse—DeepMind maintains that SIMA can function in virtually any virtual environment.
The evaluation of SIMA’s performance focuses on hundreds of basic skills that can be executed within short timeframes, across categories like navigation (“turn right”), object interaction (“pick up mushrooms”), and menu-based tasks such as opening a map or crafting an item. DeepMind’s ultimate goal is to enable agents to carry out more complex, multi-stage tasks based on natural-language prompts, such as “find resources and build a camp.”
Regarding performance, SIMA has shown promising results across multiple training criteria. Notably, an agent trained on all nine games significantly outperformed an agent trained on just one game. Furthermore, an agent trained on eight games performed almost as well as an agent exclusively trained on the ninth game, showcasing SIMA’s ability to generalize beyond its training.
However, for SIMA to achieve true success, language input is vital. In tests where agents were not provided with language training or instructions, SIMA exhibited aimless behavior, prioritizing common actions like gathering resources instead of following specific directions. This reinforces the crucial role of language understanding in effectively guiding AI agents.
DeepMind acknowledges that this research is still in its early stages, and while the results are promising, further development is necessary to enhance SIMA’s performance and generalizability. The team envisions future iterations of the agent that will improve its understanding and capability to execute more complex tasks. Ultimately, DeepMind aims to develop AI systems that can safely and effectively perform a wide range of tasks, assisting individuals both online and in real-world scenarios.
Frequently Asked Questions:
Q: What is SIMA?
A: SIMA stands for Scalable Instructable Multiworld Agent, an AI agent developed by DeepMind that demonstrates proficiency in executing various tasks within 3D games based on natural-language instructions.
Q: How does SIMA learn to perform tasks in different games?
A: DeepMind trained SIMA by observing human gameplay and recording the corresponding keyboard and mouse inputs. This data was used to train SIMA, which employs precise image-language mapping and a video model to predict on-screen actions.
Q: Does SIMA need access to a game’s source code or API?
A: No, SIMA does not require access to a game’s source code or API. It operates on commercial versions of games using only on-screen visuals and user instructions.
Q: What are the ultimate goals of this research?
A: The ultimate goals of this research are to develop AI agents that can proficiently perform tasks based on natural-language prompts and operate effectively in various virtual environments, ultimately assisting individuals online and in real-world scenarios.
Q: How does language understanding impact SIMA’s performance?
A: Language understanding is essential for SIMA to perform tasks effectively. Tests without language training or instructions resulted in aimless behavior, highlighting the importance of language input in guiding AI agents.
Q: What types of games was SIMA trained on?
A: SIMA was trained on a diverse set of nine games, including games from different genres and visual styles.
Q: How does SIMA generalize its training to new games?
A: SIMA has shown the ability to generalize beyond its training by performing well in games it hasn’t been specifically trained on. This demonstrates its adaptability and proficiency in different gaming environments.
Q: What are some examples of tasks that SIMA can perform in games?
A: SIMA can perform tasks such as navigation, resource mining, spaceship piloting, and item crafting in various games.
Q: How does DeepMind train SIMA to understand natural-language prompts?
A: DeepMind trained SIMA by observing human gameplay and recording the corresponding keyboard and mouse inputs. This data was then used to train SIMA to comprehend natural-language instructions.
Q: What are the limitations of SIMA’s performance without language input?
A: Without language training or instructions, SIMA exhibited aimless behavior, prioritizing common actions instead of following specific directions. This highlights the importance of language understanding in guiding SIMA’s actions.
For more information, you can visit the official DeepMind website: deepmind.com
The source of the article is from the blog yanoticias.es