Microsoft Unveils Muse, an AI That Can Dream Up Gaming Scenarios

Microsoft Research has unveiled Muse , a new AI model capable of generating video game visuals and controller actions.

Microsoft's Game Intelligence and Teachable AI Experiences teams worked with Xbox Games Studios' Ninja Theory to develop Muse. They trained it using a large amount of gameplay data from the Xbox game Bleeding Edge. The model can create complex gameplay sequences that stay consistent for several minutes, showing that it understands game dynamics and physics well.

Microsoft is making it easier for researchers to work on new projects by sharing the model weights, sample data, and a WHAM Demonstrator tool. These resources are available on Azure AI Foundry for researchers to use and build upon.

The team faced challenges while trying to scale up model training and make the most of insights from the LLM community. They focused on a multidisciplinary approach, involving game creators from different backgrounds to help shape the technology from the beginning.

Muse has three main abilities: It can create gameplay sequences that stay true to the game rules, produce a variety of gameplay options from the same starting point, and integrate user changes into the gameplay. It could lead to new AI-based game experiences and help with game ideas.