Microsoft just showed how artificial intelligence could find its way into many software applications—by writing code on the fly.
At the Microsoft Build developer conference today, the company’s chief technology officer, Kevin Scott, demonstrated an AI helper for the game Minecraft. The non-player character within the game is powered by the same machine learning technology Microsoft has been testing for auto-generating software code. The feat hints at how recent advances in AI could change personal computing in years to come by replacing interfaces that you tap, type, and click to navigate into interfaces that you simply have a conversation with.
The Minecraft agent responds appropriately to typed commands by converting them into working code behind the scenes using the software API for the game. The AI model that controls the bot was trained on vast amounts of code and natural language text, then shown the API specifications for Minecraft, along with a few usage examples. When a player tells it to “come here,” for instance, the underlying AI model will generate the code needed to have the agent move toward the player. In the demo shown at Build, the bot was also able to perform more complex tasks, like retrieving items and combining them to make something new. And because the model was trained on natural language as well as code, it can even respond to simple questions about how to build things.
While it’s unclear how reliably the system might work outside the demo, similar tricks could be used to make other applications respond to typed or spoken commands.
Microsoft has built an AI coding tool called GitHub Copilot on top of the same technology. It automatically suggests code when a developer starts typing, or in response to the comments added to a piece of code. Scott says Copilot is the first instance of what will likely be a slew of “AI-first” products in the coming years, from Microsoft and others. Code-writing AI “lets you think about doing software development in a different way—so you can express an intention for something that you want to accomplish,” he says.
Scott doesn’t provide specific examples, but this could one day mean a version of Windows that locates a particular document and emails it to a colleague when you ask it to, or an AI-imbued version of Excel that turns a dataset into a chart when you ask. “We’re gonna see lots and lots and lots of big productivity wins for all sorts of routine cognitive work that none of us especially enjoys,” Scott says.
In recent years, AI has proven adept at tasks such as classifying images, transcribing audio, and translating text. Recent algorithmic advances, combined with huge amounts of computer power, have yielded new AI programs capable of more sophisticated feats, including generating coherent text—such as computer code.
The Minecraft bot was built using an AI model called Codex that was developed by OpenAI, an AI company that received funding from Microsoft in 2019. Codex was trained on natural language text scraped from the web, as well as billions of lines of code from GitHub, a popular repository for software owned by Microsoft.
Microsoft’s Copilot was made available to a limited number of testers in June 2021 and is now being used by over 10,000 developers who are producing, on average, around 35 percent of their code in popular languages like Python and Java using Copilot, Microsoft says. The company plans to make Copilot available for anyone to download this summer. To build something like the Minecraft bot, developers would need to work with the underlying AI model, Codex.
Both Codex and Copilot have stirred up some anxiety among developers, who fear they could be automated out of a job. The Minecraft demo could inspire similar concerns. But Scott says the feedback on Copilot has been largely positive, suggesting that it simply automates more tedious coding tasks. “If you talk to a developer who actually uses a Copilot, they’ll say ‘this is such a great tool,’” he says.
