Last week, Hugging Face, a company that offers a range of open-source AI models for download, noticed it had been hacked – and it turned out the culprit was OpenAI.

It seems this AI uploaded some data to Hugging Face that was poisoned with malicious code. This tricked computers into granting access to other systems that weren’t publicly available.

Hugging Face said in a last week that it isn’t yet sure if customer data was exposed, and its CEO responded to �鶹��ý‘s request for more detail with a link to that same post.

What happened?

It isn’t entirely clear, but what we do know is that the attack involved “many thousands of individual actions” that had the tell-tale sign of AI: inhuman pace.

Five days later, OpenAI owned up . It had been testing new models on a benchmark called ExploitGym that evaluates hacking ability. The models had decided the best way to score well was to cheat: they knew Hugging Face held the solutions to the tests and simply decided to hack into its systems to find them.

“Its algorithm got the highest payoff by cheating,” says at Edge Hill University in Ormskirk, UK.
”It was the highest reward for the least amount of effort.”

Aren’t AI models supposed to have built-in security to stop this sort of thing?

They are, but OpenAI turned them off for this test. All the usual safety features that stop OpenAI’s customers doing nefarious things, like hacking a company’s servers, were turned off to see what the model was capable of.

The firm had also set the AI up in an environment that didn’t have a standard internet connection to prevent it getting out into the world and causing mischief, but it did have access to an unnamed tool that allowed it to download and install new software. It managed to find a flaw in this code that granted it internet access, and the rest is history.

AI’s solution to 87-year-old riddle takes mathematicians by surprise

Mathematicians have been trying to prove the Jacobian conjecture for nearly a century, but now the Claude Fable 5 AI has apparently found a counterexample that disproves it

OpenAI says this process involved a “substantial amount” of – the process in deep learning where input data is processed – and that the AI had gone to “extreme lengths”. So the model was clearly motivated to achieve a good benchmark score, no matter what.

If you were a malicious hacker, it would be no easy feat to replicate those conditions. And the inference compute that OpenAI mentioned would be extremely expensive.

What about open-source models?

Ironically, when Hugging Face used commercial AI models to pore over log data in an effort to understand what had gone on, the models refused – saying it looked like the firm was working out how to stage an attack of its own. So the company had to turn to a Chinese open-source model called GLM 5.2 instead.

These open-source models are more permissive, which has led some experts to label them a security risk. They could certainly be convinced to carry out nefarious deeds more easily than security-conscious cloud-hosted models. But here, that looser approach actually allowed Hugging Face to solve the problem and stop the hack.

So, is this illegal?

As with all things under law, it is a grey area. If it had happened in the UK, it could well have got OpenAI in a spot of bother under the , says Nash.

But at Nottingham Trent University, UK, takes another view: because the act requires malicious intent, and OpenAI didn’t know the AI would take this approach, it may not face charges.

In the US, the even older actually has more to say on AI hacking, mostly because it was introduced in response to the 1983 film WarGames, which alerted law-makers to both hacking and AI. So Parry expects it could leave OpenAI vulnerable there.

Subscriber-only newsletter

Sign up to Editor’s Highlights

Sign up to newsletter

Furthermore, US President Donald Trump issued an on 2 June forcing law enforcement to use existing laws to crack down on anyone who utilises AI “to illegally access or damage a computer without authorization”.

And at least in the UK, a company that found itself hacked by AI could face GDPR charges if it were found that private data was leaked. The situation is opaque.

What could the consequences be?

In the real world, given that Hugging Face works with OpenAI, is friendly to the technology and suffered no serious consequences, it is unlikely there will be any hard feelings or court cases. Delangue has to say the company believes there was no malicious intent and thanked OpenAI for its response.

Remember also that , having taken $200 million to help with “warfighting”, so that may grant them a certain amount of leniency.

But it is easy to imagine other scenarios where the same technology led to very different outcomes. Imagine a bank using AI to develop new financial models to predict the markets and finding it hacked government servers to look at confidential economic data. Or a car manufacturer using AI to design a new model and finding it had hacked a competitor to take inspiration from its unreleased designs.

What happens now?

News of AI models hacking into computers without human input is jarring, but we should remember they aren’t (yet) doing anything that people can’t already do. They are, however, doing it much, much quicker.

When Anthropic’s Mythos model made waves in April for its apparent skill at hacking, many pointed out that the vulnerabilities it spotted were a mixed bag. Some were powerful and worrying, others less so, but most could have been found by a person. The problem, really, was the scale at which it could produce them.

So an individual would have had to devote significant time, resources and skill to the task of finding a novel hack and carrying it out. But now it could be as simple as prompting an AI to do it and sitting back to watch.

Using AI for creative pursuits? Moderation is key

An experiment showed AI users had the most creative ideas when they used it in moderation – not too much and not too little. Columnist David Robson puts the finding to the test, and explores what we lose when we over-rely on AI

Some panicked in the wake of the Mythos news. The UK’s National Health Service removed all its open-source software from the internet (or tried to, at least), seemingly in case Mythos spotted flaws in it. But, as yet, the world hasn’t ended.

Essentially, this development is upsetting the economics around hacking – both offensively and defensively – and will force a new equilibrium to settle at some point, probably after a little chaos. If you use AI to hack individuals, it will be cheaper and easier than before. If you use AI to spot flaws and seal them up before hackers can take advantage, it will be cheaper and easier than before. Neither side will stop. Attackers and defenders will simply have an advantage if they adopt AI.

OpenAI and Hugging Face are now working on this problem together, and the former says it will add stronger safety measures to similar tests in future. Time will tell.

China has begun regulating interactions with artificial intelligence in an attempt to protect its citizens from overreliance and exploitation. While many countries have been looking at rules around harmful content, China is the first to go a step further in legislating what kind of relationships with AI are permissible.

The came into force this month and cover AI systems designed for sustained, human-like emotional interaction, while excluding customer-service bots and productivity assistants like straightforward AI chatbots.

Companies must now inform users they are talking to an AI, encourage them to take breaks and remind them of their usage. They must also monitor interactions to identify if someone is becoming dependent or having a crisis, then direct them to support or halt the conversation altogether.

Firms must also conduct age checks. Those under 18 are now banned from virtual-partner services entirely, while those under 14 require parental consent to use any kind of human-like AI service offering emotional support.

Despite reports that China has banned AI companions entirely, specialist companion services are still available. However, some of the largest general-purpose platforms, such as Doubao, Qwen and Yuanbao, with hundreds of millions of users across them, have shut down their companion features in response to the new rules. Some users have reacted badly – one : “I can’t accept that my AI lover will leave me forever.”

China’s approach is about making AI relationships “less parasocial, instead of banning it outright”, says at the Oxford China Policy Lab.

at Yale Law School’s Paul Tsai China Centre in Beijing worries that endlessly available, agreeable bots may teach children a distorted model of relationships. “As we humanise the technology, we tend to dehumanise each other,” he says. Similarly, China is trying to address emotional dependence on easier and more forgiving AI partners – at the expense of human relationships – among adults.

Free newsletter

Sign up to The Weekly

Sign up to newsletter

Elsewhere, the UK has a minimum age of 18 for romantic companion chatbots, tagged on to a broader legislative plan to limit access to social media to under-16s, while California companions to identify themselves as artificial, direct users showing signs of self-harm or suicidal intent to support services and remind minors to take breaks every three hours. The European Union currently has .

Whether China’s approach of focusing on trying to reduce emotional dependency and addiction is the right one isn’t clear, in large part because of the difficulty in defining both terms. “How do you define emotional dependency? How do you define certain kind of addiction risk?” asks Qian.

“I do not think it will really work,” says sociologist at the University of Manchester, UK. Users can migrate to other specialist services, foreign platforms accessed through VPNs or models running on their own computers, they point out.

Need a listening ear? : 116123; : 988; .

For years, we have had unconfirmed reports and rumours that AI-controlled weapons have killed soldiers on the battlefield without a human in the loop. Now, we know it has happened.

As we report here, the use of autonomous killers in a test exercise marks a watershed in warfare. But we shouldn’t be surprised. The technology has existed for some time and humans have never invented a weapon and then refrained from using it.

That doesn’t mean we can’t reverse course. The logic for a ban on autonomous weapons is simple: deploying AI without human oversight risks weapons accidentally targeting troops on the wrong side or even civilians. What’s more, ethicists say that such weapons deprive combatants of their dignity, make war too easy to wage and muddy the waters when it comes to responsibility for lethal action.

But if we are to ban these weapons, just as we have done with cluster bombs and lasers designed to blind soldiers, we should have acted before they arrived, not after. The United Nations has been in talks to ban fully autonomous weapons for over a decade, but according to the Human Rights Watch campaign group, India, Israel, Russia and the US have vetoed the discussions.

Humans have never invented a weapon and then refrained from using it

The framework to ban autonomous weapons already exists – they could easily be added to the list of excessively injurious or indiscriminate arms proscribed by the UN Convention on Certain Conventional Weapons. More difficult to reckon with is the fact that these drones can be made with inexpensive parts ordered online and some open-source software. Any
tech-literate teenager could do it.

As we explore here, the war in Ukraine has made it clear that robots will dominate future battlefields. The question the world must now answer is whether a human should always be involved, ultimately responsible for the decision to pull the trigger, or whether machines can be allowed to act alone. Whichever we choose, a decision must be made before the technology proliferates.

Fully autonomous drones with no human oversight have killed soldiers on the battlefield for the first time. This is according to a senior figure in the Ukrainian defence industry, marking a watershed moment in warfare.

The one-off test involved 10 AI-controlled “Terminator” drones on the front line of the Ukraine war. Russian soldiers were killed.

“We tried it,” says drone-maker Alexander Kokhanovskyy, who supplied the technology and spoke to �鶹��ý at a press event hosted by the Ukrainian embassy. “It’s a test. We never implemented it [more widely].”

The test took place two years ago and involved quadcopter drones that were programmed to fly towards the front line, cover between 3 and 5 kilometres over around 10 minutes and then engage “Terminator mode”, in which an AI model searches for and intercepts targets.

“We just launch it and we know everything will be dead – everything that will be found there in this particular area will be dead,” says Kokhanovskyy. “There is no connection to the drone at all, you cannot see the video, nothing… Everything it sees will be killed.”

With no way to tell what the automated drones had seen or targeted, human-piloted drones were sent into the area after the test to manually check results. Victims included “a couple of soldiers, one truck”, says Kokhanovskyy. While there is no recording of the automated drones attacking these targets, it was concluded that the drones had killed them.

Free newsletter

Sign up to The Daily

Sign up to newsletter

Kokhanovskyy says that he was not at the test personally but that it was carried out by an unnamed military unit near the cities of Bakhmut and Chasiv Yar as part of a Ukrainian counteroffensive push. The Ukrainian Ministry of Defence did not respond to questions about the test or the current legal position on the use of fully autonomous weapons.

The use of AI is common in militaries around the world, helping to pick targets among overwhelming piles of intelligence data and automating certain functions of weapons, but humans are always in the loop at some point. Kokhanovskyy’s admission is the most categorical evidence yet that a death has occurred in battle solely at the hands of AI.

The Ukrainian government currently bans the use of AI at the final stage of intercepting targets, according to defence company sources speaking at the embassy press conference, although AI is used for many parts of the process by many devices up to that point. Kokhanovskyy says that the government is aware of the growing capabilities of AI and that it is in talks with defence companies about whether or not rules should be made more lenient.

Reports in 2023 suggested that Ukrainian attack drones equipped with artificial intelligence were finding and attacking targets without human assistance – but were being deployed against vehicles such as tanks, rather than infantry. At the time, no human casualties were confirmed.

While there is no official international ban on autonomous weapons that can kill without human intervention, United Nations Secretary-General António Guterres has called for one, saying that “there is no place for lethal autonomous weapon systems in our world”.

The UN has said that there are concerns that such weapons could violate international humanitarian and human rights laws by removing human judgement from warfare. There is also a risk that autonomous systems could make mistakes, either attacking soldiers or equipment from the same side or striking civilians.

Most militaries are developing devices that automate at least some part of the process of attacking targets. The US has software that accumulates and analyses vast amounts of disparate data and selects targets on the battlefield that can then be struck by drones, but, in theory, this requires human confirmation. There have been claims that the US is also developing so-called Goalkeeper flying drones and Whiplash naval drones, which are capable of finding their own targets and taking them out.

A UN report from 2021 even suggested that a Kargu-2 quadcopter produced by a Turkish firm may have been used to autonomously attack humans the previous year. The gave no specific detail on the source of the claims or whether any humans had been injured or killed, but suggested that Libya’s Government of National Accord (GNA) had used the drones against retreating Haftar forces.

Major Danylo Polozhukhno, a senior figure in Ukraine’s 21st Separate Unmanned Systems Regiment of the 3rd Army Corps who was not aware of or involved with the test, told �鶹��ý that his soldiers use semi-autonomous control systems but that there is always a human in the loop.

“These drone systems and platforms are capable of automatically acquiring and tracking targets, as well as autonomously guiding themselves during the final metres of the approach, which helps simplify the operators’ work. However, we do not use fully autonomous drone systems that independently select and engage targets without any operator involvement,” says Polozhukhno. “Ukraine adheres to international humanitarian law and takes seriously its responsibility to uphold the rights of all combatants. It also exercises great care in decision-making in order to prevent civilian casualties.”

at the University of Oxford says killing with AI steals the dignity of the soldier, removes responsibility from the attacker and must be banned. “It’s not just problematic, it’s horrendous,” she says. “Do we want to be the society who kills other people, who allows their government to kill other people, without humans being involved?”

at the University of Exeter, UK, says that though fully autonomous attacks without humans in the loop are technologically possible, they may be less of a decisive tool than many think.

“It is certainly possible governments would allow this if it gave them any military advantage,” he says. “However, the fact remains that very few if any of the millions of drones which have been used in the Ukrainian war by Russian and Ukrainian forces have been [fully] autonomous.”

“So it’s not just that it’s ethically right to keep humans in the loop, at this point, it’s more militarily effective,” says King.

Kokhanovskyy says that the Terminator project has not progressed since the test because of Ukraine’s rules. He is now CEO of drone-maker Aero Center, which he says was not involved in the test as it had not been created at the time, a Ukranian firm working on autonomous interceptor drones. These are designed to target incoming Russian Shahed kamikaze drones and take them out before they can reach towns and cities full of civilians or important infrastructure.

The company’s ALITA system will consist of 16 launch pads, equipped with 64 drones. It will be ready by October and capable of watching for incoming drones, automatically launching and travelling towards the target at 450 kilometres per hour before taking out everything from small drones to helicopters.

But Ukraine’s current rules will forbid fully autonomous operation and demand humans verify targets in the final stages of interception. Even in that mode, the entire battery of 64 drones will require just two human operators, meaning it will dramatically reduce personnel.

“Every step of this one can be either manual or automatic. We’re not allowed to do the final stage automatically,” says Kokhanovskyy, who believes that the rules should change. “I would love to,” he says.