Google DeepMind just introduced an AI agent that plays video games with you

Imagine playing your favorite RPG, but instead of battling computerized enemies hellbent on defeating you all by yourself, you have a mighty companion – an advanced artificial intelligence (AI) agent that learns and behaves like a human player.

On Wednesday (Mar.13) Google’s AI research subsidiary DeepMind unveiled SIMA, a revolutionary AI agent designed to learn and play video games in a manner akin to human players. Unlike traditional game AI that is programmed to win at all costs, SIMA is trained to run games and follow instructions, mimicking the natural gaming experience of a human participant.

Introducing SIMA: the first generalist AI agent to follow natural-language instructions in a broad range of 3D virtual environments and video games. 🕹️

It can complete tasks similar to a human, and outperforms an agent trained in just one setting. 🧵 https://t.co/qz3IxzUpto pic.twitter.com/02Q6AkW4uq

— Google DeepMind (@GoogleDeepMind) March 13, 2024

How does Google DeepMind’s SIMA work?

SIMA – which stands for Scalable, Instructable, Multiworld Agent – is currently in the research phase, but it’s a significant step toward developing general, capable AI agents. The agent is being developed to eventually learn how to play any video game, including open-world games and those without a linear path to completion.

“SIMA isn’t trained to win a game; it’s trained to run it and do what it’s told,” explained Tim Harley, a Google DeepMind researcher and co-lead on the SIMA project. “It mixes natural language instruction with understanding 3D worlds and image recognition.”

To train SIMA, Google DeepMind collaborated with eight game developers, including studios like Hello Games, Embracer, Tuxedo Labs, and Coffee Stain. The researchers plugged SIMA into various games, such as No Man’s Sky, Teardown, Valheim, and Goat Simulator 3, to teach the AI agent the basics of gameplay.

The training process involved multiple steps, including building a custom environment in the Unity engine where agents learned object manipulation by creating sculptures.

Then researchers recorded pairs of human players, one controlling the game and the other providing instructions, to capture language cues. This data was then fed to SIMA to help it predict on-screen events based on actions and instructions.

But there’s still a long walk to go before SIMA can be rolled out. It’s learning to walk before it can run.

Currently, SIMA has mastered around 600 basic skills, such as turning left, climbing ladders, and opening in-game menus. However, the team envisions that SIMA could eventually be instructed to perform more complex functions within games, like finding resources and building camps.

Who knows, SIMA may one day become the perfect AI party member, helping gamers to victory (or would that be cheating?) while providing a truly human-like gaming experience.

Featured image: Generated by Ideogram