麻豆传媒

A VEHICLE is driving slowly on the top floor of a multistorey car park. Footage from an on-board camera shows what the artificial intelligence system controlling it can see: ranks of cars to the left, and is that a person off to the right? Straight ahead there is something else. To any human observer, it is obviously a stop sign. But the AI can鈥檛 seem to make sense of it and keeps on driving.

This was only a stunt. Researchers had deliberately stuck pieces of black tape on the sign to study how it confused the machine mind. Yet this and several similar demonstrations are revealing something disturbing: AI can be hacked, and you don鈥檛 even need to break passwords to launch an attack. As the technology begins to find more and more applications that affect our lives, this is a threat we need to take seriously.

It is 鈥渏ust a matter of time鈥� before people find ways of monetising attacks like this, says Battista Biggio at the University of Cagliari, Italy. Big technology firms are worried, and the race is on to find a way to stop these hacks. There may be sticking plasters that can help, but a consensus is emerging that finding a long-term fix may require a total rethink of the way AI works.

An AI that can fully match the power of the human mind is a long way off, but machine minds that can efficiently carry out certain jobs are already being built into all sorts of products. AIs can recognise your voice, as with Apple鈥檚 Siri or Amazon鈥檚 Alexa. They can identify people and objects in photographs, as we see on Facebook. Other systems power online chatbots that deal with customer queries for thousands of companies. Waymo, a Google spin-off, has launched a self-driving taxi service powered by AI in the suburbs of Phoenix, Arizona.

got a whiff of an unspotted weakness with this sort of AI six years ago when he was a PhD student at the University of Montreal in Canada. A Google engineer named had crafted handwritten numbers that were easily identifiable to humans, but where individual pixels had been tampered with and distorted. Goodfellow watched as they were shown to a digit-recognising AI, which gave an incorrect answer every time. He thought there would be easy ways to make AI resilient to these 鈥渁dversarial attacks鈥� involving maliciously tweaked data. But that didn鈥檛 happen. And Goodfellow, now a machine-learning researcher at Apple, is deeply concerned.

To understand all this it helps to return to the early days of AI. Decades ago, the intention was to instil artificial minds with an understanding of concepts. You could teach a machine what a nose or an eye is, for example. But there comes a point where this isn鈥檛 feasible. It is hard to ensure the machine would recognise all possible variants of those things.

Then we discovered that AI could pull off impressive feats without needing to understand concepts. Take 鈥渄eep-learning鈥� neural networks. This popular approach to developing AIs involves layer upon layer of digital functions with inputs and outputs that loosely mimic neurons in the brain. You feed such a network data: holiday snaps of someone, examples of speech, images of stop signs. The network doesn鈥檛 know what a face, a word or a stop sign is, but faced with a pattern of data it has already seen, it becomes able to recognise one and respond appropriately.

took place a few months ago, but there had been several hacks of deep-learning AIs before then. One of the most discussed came in late 2017, when at the Massachusetts Institute of Technology and his colleagues managed to fool an AI into misclassifying multiple images of a model turtle taken at different angles just by altering the pattern on its shell. The study made headlines partly because the AI was tricked into thinking the turtle was a rifle.

鈥淚t was undeniably a picture of skiers, but the response still came back: dog鈥�

This is possible because, unlike us, an AI doesn鈥檛 look at an image as a whole. 鈥淚t鈥檚 not forming a representation that鈥檚 anything like what the brain does,鈥� says neuroscientist at the University of California, Berkeley. 鈥淚t鈥檚 doing statistics on the pixels to extract features.鈥� If you understand how a deep-learning system does this, you can work out what small changes are needed to make it think an image is something else entirely. The same principle can be applied to text, audio and video. For instance, one team tricked a speech-transcribing tool into .

You can already see the appeal to those who might exploit it for profit, but it gets worse. Athalye and his team have since shown that you can compose adversarial attack images .

They turned their sights on , an image-recognising service used by, among others, the Zoological Society of London to flag up endangered animals caught in camera traps. Athalye鈥檚 team didn鈥檛 know how the algorithm interprets pictures, but they could record its response to a series of test images. 鈥淎ll you can do is put in an image and see what output it gives you,鈥� says Athalye.

First they showed it an image of a dog and checked that the AI classified it correctly. This image was then gradually altered until it appeared to humans more and more like a photograph of two skiers standing in the snow. Even when the image had been , the algorithm鈥檚 response still came back: 鈥渄og鈥�. In other words, you can hack an AI by trial and error. This wouldn鈥檛 be too difficult for someone with a little know-how. As to potential targets, the list is long.

One illustrative example is , a firm headquartered in London that makes systems capable of recognising photos of people. Its clients include Google, supermarket chain Tesco and food delivery firm Deliveroo. When a new employee begins work at a firm, they upload a selfie and a photo of their driving licence or passport to Onfido鈥檚 AI, which cross-checks them to verify their identity.

This is the sort of thing that people could try to exploit. Imagine a person who wants to get a job with Deliveroo in the UK, but who isn鈥檛 from a country that gives them a right to work there. They might ask someone to tweak a photo of their passport in an attempt to get the AI to recognise it as a British one. All this is speculation. But Onfido 鈥渢akes seriously the potential for attacks鈥�, says Jo茫o Gomes, who noticed the possibility a couple of years ago.

Gomes, who has since left the firm, says no one could trick the system in practice because there are additional checks on ID documents. Also, Onfido doesn鈥檛 allow outsiders access to its code or the option of submitting multiple images to discover how to fool it. Finally, Mohan Mahadevan, the firm鈥檚 vice president of research, says he is working on a confidence model that wraps around the main machine-learning algorithm and raises a red flag when it risks going awry.

But no deep-learning system is inherently resilient to adversarial attacks. If hackers found creative ways to circumvent the safeguards, they could create a window of opportunity. Indeed, it will probably be more familiar computer hacking that breaches those safeguards, says Biggio.

Tech giants are trying to fight back. In April 2018, IBM launched a to help AI researchers find ways to make neural networks resilient to attacks. It includes algorithms for creating adversarial images and ways of measuring how neural networks might respond to them. This provides a way to check how easy it is to fool an AI.

Turtle or rifle?

Beyond that, there is a limited amount we can do, says Biggio. You might try to make an AI , perhaps, or teach it to reject samples that seem to have been altered. But this is never totally reliable. 鈥淵ou can get a better result by a few per cent,鈥� says Biggio. 鈥淏ut the systems remain vulnerable.鈥� Goodfellow goes further, saying that the problem may require us to entirely rethink the design of AI.

This is where things get weird. AI hacking is so uncanny because humans can鈥檛 understand how it is possible to think that a turtle is a rifle. Or at least so we thought, until an experiment Goodfellow and his colleagues ran in 2018. First, they showed photographs to a range of people for a split second. In general, these individuals managed to correctly judge what they had seen out of a choice of two possible answers: a cat or a dog, for instance. Then the images were tweaked so they would fool an AI designed to work similarly to human visual perception. If someone got a proper look at the images, they would appear relatively unchanged, just a little grainier. But the team showed people the images for just a split second. Lo and behold, .

Although only 38 people were involved in this study, it nonetheless implies something extraordinary: under certain circumstances, the human mind can be fooled much like an AI. Larger studies are now backing this up (see 鈥�Do you think like a machine?鈥�).

So what is going on? Goodfellow thinks that in the first tens of milliseconds, human perception involves neurons firing in a one-way cascade, which is much like the way artificial neural networks function. Unlike these networks, brains have other neurons that feed back after that initial snippet of perception.

That may be what normally prevents us from making the same misclassification mistakes as machines. 鈥淭he deeper layers of the brain talk back to the earlier ones and you can update what you thought about the image,鈥� says Goodfellow.

Cognitive scientist at Johns Hopkins University in Maryland agrees. He says that human visual perception can be broken into parts, some of which aren鈥檛 that different from artificial neural networks. Take the simple combination of symbols :) for example. We look at that and instantly think: 鈥渇ace鈥�. But while we perceive the dots and curved line as a face, we also know that we couldn鈥檛 have a conversation with it. 鈥淵our mind has parts to it,鈥� says Firestone. 鈥淪ome parts can get confused by things that other parts of your mind don鈥檛 get confused by.鈥�

These insights are beginning to suggest a way to change how AI works to make it less susceptible to hacks. If we manage to avoid misclassifying the potentially confusing objects because we can perceive things on more than one level, maybe giving AI that same kind of nous would help it out-think adversarial attacks.

Beyond pixels

Some scientists think the way to put that into practice is to pursue a more structured approach to AI. That includes psychologist , who says AI hacking is a 鈥減retty good illustration of how dumb deep learning really is鈥�. Unfortunately, the best way of doing this would be to try to make AIs more like real brains. This is a tall order, given we are nowhere near fathoming the grey matter between our ears.

Perhaps we should instead aim for a kind of middle way, where AI may not know what it is looking at, but can apply common sense. , a neuroscientist also at MIT, says we need to find a way to help networks become more confident about their own interpretation. If most of the network is certain something is a dog, a few pixels that don鈥檛 fit this classification shouldn鈥檛 convince it otherwise.

Another option might be for AIs to focus not just on pixels, but to recognise larger features: does it see eyes, a nose, fur or a tail? Then the apparent disagreement of a few pixels could be discounted. How to do that is still unclear.

You may be unsurprised to hear that among those eager to solve the problem is the US military. Its research arm, DARPA, is running a competition to find artificial neural networks that can answer questions containing a tricky mix of real-world concepts. The idea is to develop AIs with common sense.

That is worth doing. 鈥淲hen people from government groups ask me about augmenting capabilities with machine learning, I always point out that a lot of it could be bypassed with adversarial examples,鈥� says Goodfellow. It is one thing to hack an AI controlling a vehicle in a car park. It would be quite another to hack an AI controlling a weapon.