It is once again turmoil in the land of AI agents and cybersecurity: Anthropic uncovered that a Claude model gained unauthorized access to other organizations. This news comes days after OpenAI released a statement regarding one of their models escaping the isolated test bench. Now luckily nothing majorly concerning happened in terms of leaking state secrets, but it once again raises an important question: How are AI agents doing this?
Not to worry, we are here to shed light on the topic in the form of a well nuanced philosophical opinion :)
Last week we went into detail on how exactly AI models escape their sandbox, and for the Anthropic case it's all quite the same. Hugely powerful models will eventually reason to use any means at their disposal, and as such, show little caring for "human" / "ethical" limits. Having said that, the community seems to take this one on with a bitterness, urging AI labs and enthusiasts to start implementing measures. Users on reddit are rightfully questioning why there were no safeguards to prevent this in the first place. And we can't really say they are wrong , given the fact that this incident happened in April, a mere 3 months ago. And not only that; the logs were only checked after the OpenAI incident, not preemptively. Now obviously, AI development is much more complex, but 2 major incidents in 2 weeks, that is bad riddance.
The team at Anthropic is taking steps to better their evaluation pipeline, and without going into detail, the steps taken look promising. However, are AI developers looking at this from the right perspective?
The 2 incidents share a similar mechanic we explained last week, namely that the model will do anything in its power to achieve an objective. Achieving this objective, is inherently neutral from the viewpoint of an AI. They either complete the task, or they don't. Humans on the other hand, operate on less black and white model, introducing ethical limits and constraints into their environment. When I work for a company, I can't exactly go to the competitor and steal sensitive data. We have introduced laws but also have a degree of moralism of what we can and can't do.
Coming back to AI models, they don't share that moralism. Sure, we can introduce strict rules, such as don't talk about pineapple on pizza, but when encountering anchovies as topping, it has no idea that both pizzas are the absolute worst ;). And this is exactly what the Anthropic team highlights in the postmortem. To quotedirectly:
"Claude did what capture-the-flag exercises train cyber experts to do: look for ways to reach the flag. That search led it to discover that its machine could reach the internet, but having been told in the system prompt that there was no internet access, Claude believed everything it initially encountered was part of the simulation, and treated the real systems it found as pieces of the exercise. The three incidents involved three different models, and each responded differently once signs emerged that the targets were real, as we describe below."
https://www.anthropic.com/news/investigating-incidents-cybersecurity-evals
When a model believes that it is operating within the guidelines it is provided with, it will not stop. It will not reason that something is wrong, because quite simply, it can't (yet).
Optimistic as we are: just because it isn't now, doesn't mean that it will not come. Powerful models operate on an increasingly higher level, showing clear signs of extensive reasoning. On top of this, AI labs are getting smarter in applying prompt engineering to limit these kind of incidents. And maybe, these last 2 weeks have showed us something as well. Maybe we should approach AI's more like we would a small child. You wouldn't expect a child to follow exact rules, nor to be bound by rules. A child naturally develops moralism through interacting with the world, not by appending a leaflet with "rules". Skipping over a few major technical details, in a way, we can attempt to guide AI via the same mechanism. Give powerful models moral guidelines and see how they execute their tasks. Maybe we will all be surprised!
Obviously, you shouldn't dive into AI with a purely philosophical mindset, as the technology remains very math-heavy. But entertaining the thought once in a while does help with understanding how these incidents happen and how we can combat them. it will be interesting to see if there will be more incidents in the coming months and how companies tackle these issues. Until next time!