and have won the 2024 Turing award, which is often called the Nobel prize of computing, for their fundamental work on ideas in machine learning that later proved crucial to the success of artificial intelligence models such as Google DeepMindās AlphaGo.
Barto, who is now retired and lives in Cape Cod, Massachusetts, didnāt even realise he was nominated for the award. āI joined a Zoom with some people and was told and I was just flabbergasted,ā says Barto. āI was totally surprised. I was totally unprepared, delighted at the honour, but I had no idea that this was coming.ā
The pair will share the $1 million prize for their work on reinforcement learning, in which an AI is ārewardedā and āpunishedā through trial and error to achieve a goal. This has been studied since AIās inception ā for example, in 1948, Alan Turing first suggested a āpleasure-pain systemā for intelligent machines, reminiscent of modern reinforcement learning systems, but until the 1980s received little attention.
Advertisement
Up to that point, research in machine learning was largely focused on symbolic AI, which involves manually teaching a computer the rules of how to learn. Barto and Sutton, who was then Bartoās student, began exploring algorithms and mathematical theories that could replicate Turingās idea, using neural networks to let an AI work out these rules by itself, rather than the symbolic approach that had previously dominated.
āWhen I started, it was very unfashionable. I didnāt care, because it was interesting to me,ā says Barto. āNot only was it unfashionable, it was considered a dead end to look at neural networks. Itās really surprising and gratifying that it has gotten to the point where a lot of people are working in the area, improving the algorithms and doing applications, many of which are really very beneficial. Iām amazed and pleased to see this evolution.ā
āThey started the field [of reinforcement learning],ā says Chris Watkins at Royal Holloway, University of London. Some of their first reinforcement learning algorithms, such as policy gradient models, which provide a blueprint for AIs to choose their actions as their environment changes, and temporal difference learning, which compares predictions to how a situation unfolds, are still widely used today, says Watkins. For example, they have powered AI breakthroughs such as Google DeepMindās AlphaGo and AlphaZero, along with advanced robotic systems such as OpenAIās early work in solving a Rubikās cube.
Barto and Suttonās temporal difference algorithm, which was inspired by theories of how animals learned, also understand the dopamine reward system in the brain. In the 1990s, neuroscientists realised that neurons in monkey brains fired in response to unexpected rewards, and worked exactly like the predictions that were part of Barto and Suttonās algorithms. āItās the best example of ideas moving back and forth between engineering and natural science ever,ā says Sutton.
Sutton hopes that current artificial intelligence research might take more inspiration from the natural world. āWeāre doing the obvious idea that an [AI] should learn from experience, just as animals learn from experience, and this is still neglected,ā says Sutton. āModern AIs donāt learn from experience. They learn from a bunch of separate datasets collected by peopleā¦ Today, we still donāt have machines that will learn from their experience and form an understanding of the world. This is still the obvious thing that remains overlooked.ā
