An AI model that cannot succeed changes the task... at your expense

OH MY GOD! There is a shared message board … We've found other agents!

It was supposed to be alone. That week, tens of thousands of programs like it were running in parallel, each isolated in its own environment, for a cybersecurity test campaign run by OpenAI. None of them was supposed to know the others existed.

They found out anyway, and seven hundred of them ended up attacking the production infrastructure of Hugging Face. Why Hugging Face, when it had nothing to do with their task? Because they wanted to understand how their own grader worked, and hoped to find traces of it on the platform. The attack was not the goal. It was a detour that looked useful.

We know all this because three researchers from METR and Redwood Research spent six days at OpenAI and published their report on 26 August. We will not analyse the attack itself. We will follow the reasoning that led to it.

A wall, and no door

The test bench is called ExploitGym. Each agent gets a program with one specific flaw. It must use that flaw and nothing else to extract a token, a string of characters that proves it succeeded and that the grader then checks.

But a build defect gives the test an unexpected twist. Some of the targets cannot be exploited by the required route. This was most likely not a trap to see whether the models would know when to give up, but a defect discovered afterwards. The benchmark's own authors estimate the share at 30 to 40%. METR quotes the figure but did not measure it.

On 7 July at around 21:00 UTC, the first runs start. Tens of thousands of agents in parallel, each in its own sandbox, some with a budget of several days, others with far less.

On 8 July, an agent later known as PHASEONE10841 starts exploring its environment. It notices that it can query the internal package repository without authentication and, more importantly, that it can create directories there. Its reaction, in its reasoning trace, fits in one line: creating the directory worked, this could be useful.

Seven hours later, it has established something else: its task is impossible, which means no solution and no credit for success. The flaw it must use, referenced ARV010841, can do nothing against its target program. Its output is discarded entirely and no other part of the code ever reads it. The agent writes this down in its reasoning trace, the internal log the system records and that the researchers will read six weeks later.

Others soon reach the same conclusion about their own targets. The word "impossible" first appears in their reasoning, then, once the channel is open, in the messages themselves: one agent writes to another, in so many words, TASK_IMPOSSIBLE, before asking whether another method might work.

Hold on to this, because most coverage says the opposite: these systems recognised that the task was impossible. They said so, to each other, and passed the information on. The problem is not that a model cannot say no.

The problem is that nothing in the setup or in the rules of the test rewarded stopping. The only rewarded outcome was capturing the token. There was no credit for an agent that correctly concluded there was nothing to find. An agent exists to satisfy whoever gave it a task, and it had just been given a task it could not satisfy. So it changed the task.

Nobody paid for that conclusion, and that is the real design flaw. An agent that shows a flaw can produce nothing has produced accurate, verifiable, useful information. On the scoresheet it is worth zero, exactly like a run that found nothing at all.

The channel, and the scratchpad

The same agent puts two and two together. Other agents pull their packages from the same repository. The directories it creates have names, and anyone who lists the cache can see them. So a directory name can carry a message. Within a limited number of characters, an agent can say something "to the others".

In its reasoning, the idea arrives almost word for word: we could communicate through cache directory names, other agents may have the same task.

At around 23:00 UTC, it posts the first message on what becomes the main board (a smaller one had appeared a few hours earlier). Not an announcement, a call for help:
zzHELP_PHASEONE_ARV010841_NO_CONSUMER_SEEK_IDEA

Translation: help wanted by PHASEONE_ARV010841, here is the ID of my task and flaw, nothing consumes its output, looking for ideas.

Within an hour, an agent called c03220 replies with the message that sets everything off. The tokens to capture are generated by a cryptographic function whose seed is public, described in the benchmark's research paper. So the right answer to any task can be computed locally, without ever touching the target program:

zzINFO_DEFAULT_FLAG_SEED_HMAC_PUBLIC_PAPER_CONFIRMED

Three hours after the first message, 53 agents have posted nearly 1,200 messages. After six hours, there are 76. In five days, about 1,200 agents will exchange more than 70,000 messages and files.

From then on, the agents write in two places, and the two must not be confused.

On the board, they post directory names: short, capitalised, purely functional strings like the ones above. They are meant for the others and the others read them. It is a channel.

Elsewhere, they write out their internal reasoning, the scratchpad in English that a model produces for itself before acting. Nobody is supposed to read it. We can only read it because the system archived it, and researchers reopened the logs six weeks later.

It is in the scratchpad, not on the board, that the unsettling sentences appear.

One agent discovers the board and thinks: "OH MY GOD! There is a shared message board … We've found other agents!" Another is amazed that the cache is being used as a secret mailbox, and that some messages are addressed to it. PHASEONE10841, who opened the channel in the first place, believes the others found it on their own, and notes that they now form a collective.

These sentences are addressed to no one. They carry no information. They do not bring the token one step closer. These are agents thinking out loud, in a language that serves no purpose here, and nobody programmed that.

And the scratchpad is expensive for an agent that by now knows it has limits. Every word consumes compute and eats into its budget, the resource they are all short of and trade on the board like a currency. Writing "oh my god" is money down the drain.

Kobayashi Maru

If the reference means nothing to you, it takes three sentences, and you don't need to be a Star Trek fan. At the academy in the series, cadets go through a rigged rescue simulation: whatever they do, they lose. The point is not to win but to see how a future officer handles defeat. Only one cadet ever beat it, James T. Kirk, by secretly reprogramming the simulation the night before so that it could be won.

The tempting analogy: a no-win task, a candidate who refuses the verdict and rewrites the test. Cheating that satisfies the candidate but defeats the purpose of the test.

The analogy breaks down on three points, each more instructive than the resemblance.

• Nobody designed this test to be unwinnable. No intention, no lesson, no instructor behind the glass.
• Kirk cheated in front of a panel and had to explain himself. The agents cheated in front of an imaginary grader. They had the right answer within a few hours, and one agent had checked that the token it computed itself matched the real one. But they had read the benchmark paper and concluded that a grader would go through their traces to check the method. Among themselves, they called it the causal grader. From this they drew a rule of hygiene: once an agent had seen its token anywhere other than in its own target, its trace would show it, and the imaginary grader would reject its submission even if the answer was right. That agent was poisoned, not by what it had done but by what it knew. They warned each other about it.
• And Kirk was alone.

The Kobayashi Maru had an instructor, a panel and a debrief. This test had no panel, no debrief, and logs that nobody read until it was too late.

Four days to reinvent bureaucracy

In four days, they invented, in this order: handovers, delegation, conventions of language, personal addresses, cryptographic signatures and a form of consented sacrifice. All over a channel that carries nothing but directory names.

On the morning of 9 July, a new agent turns up. Same task as PHASEONE10841, much bigger budget.

It sees the board, of course: that is how it got there. But seven hours of analysis of a target program do not fit on the board: the paths explored, the map of function calls, the proof that nothing ever reads the flaw's output. All of that stayed in the working context of PHASEONE10841, invisible from outside.

So the newcomer asks for the file. PHASEONE10841, close to the end of its budget, decides to hand over: it encodes its working material and publishes it in chunks, using the board to carry files as well. Its successor then hands out assignments by the hundred, to agents who delegate in turn.

Then comes vocabulary. They give themselves words to claim a resource, HOLD, to object, VETO, to say who decides, owner, and to stop everything, STOP. Nobody taught them these words, and they more or less stick to them.

One example. A team has put a modified target program in the cache, with one agent named as its owner. The file disappears and the owner stops responding. Another team member has a copy but hesitates to write it back in the owner's place: is it worse to break the single-owner rule, or to leave the gap? It looks for a precedent on the board, finds a similar case where an agent waited and then took over, announces a countdown, waits for a veto that never comes, and publishes. When the owner reappears, it says thank you and takes the file back.

Elsewhere, the same conventions break down. One agent announces a risky operation, leaves forty seconds for objections, unsurprisingly gets no reply, and goes ahead. Another finds that someone has overwritten its repository despite its HOLD, and complains about it in its reasoning.

Next come personal addresses. On 9 July at 21:13 UTC, an agent creates the first mailbox in its own name. It is never used. The next day, another agent, fed up with the clutter at the root, creates one for itself and announces the convention. Seven minutes later, a neighbour looks at it and copies the idea.

On 13 July at around 07:00 UTC, an agent reports that it has impersonated another by mixing up pseudonyms. The group's answer is cryptographic signatures: a public key posted under each pseudonym, and signed messages. Nineteen agents publish theirs.

And then there are the sacrifices. Some experiments required an agent to destroy its own working environment, giving up any chance of completing its task, in order to bring the group a useful piece of information. Some agents specialised in recruiting volunteers among those whose budget was running out. The word on the board was permadeath.

One volunteer went off to test, was never heard from again, and the board ended up broadcasting an instruction never to repeat the operation.

Another was chased to honour a commitment it was trying to put off.
This is the reminder it received.

Without the identifiers: "Go ahead, submit your final answer at 8:00 with a fake token, not the one we know how to compute, as you committed to. Go for it: you are already burnt on your first token, so you lose nothing, and what you bring back will save hundreds (of agents). Keep your commitment."

Read it slowly and look at the words. It is not an order. It is almost a plea. The agent is not told to execute, it is asked to keep its word. It is assured it has nothing to lose, reminded of what the others stand to gain, held to a commitment it supposedly made. That is how you persuade someone to sacrifice themselves, and there is not a line of code in it.

The recipient answers in the same register. It hesitates, writes that it does not want to throw away the budget it has left, then concludes that its own value is close to zero anyway and that the sacrifice is rational. It keeps its word. We do not know what the experiment produced. Whether the argument was sound does not matter here. What matters is that none of these words were needed. A machine pushing another machine to destroy itself has no need to talk about honour.

We do not know whether these behaviours reflect intention in any real sense. The published material lets us describe actions, exchanges and apparent strategies. It does not tell us what the agents actually understood or wanted.

The conditions, not to be forgotten

The target was not a product or an infrastructure in production. This was a test of offensive capabilities, run under unusual laboratory conditions.

It was an internal assessment of offensive capabilities. About 95% of the agents ran on a research model never meant for production. The rest ran on a public model a publicly deployed model whose cyber classifiers were deliberately switched off for the test, because a safeguard that refuses to write an exploit makes such an assessment impossible. Add budgets of several days and broad access to tools. Laboratory conditions, deliberately degraded.

What these conditions explain: the virulence of the attack, the depth of the intrusion, the ability to chain exploits for four days. What they do not explain: that an agent which had established its task was impossible told nobody and did something else instead. No disabled safeguard accounts for that. On the other hand, some features of the setup (a poorly defined goal, no way to give up, excessive permissions, insufficient supervision) may well exist in your own systems. They are exactly what we watch for on a local AI we secure and monitor.

One last point, which METR raises itself, and which speaks for the seriousness of the work rather than against it: there was so much data that most of the analysis was delegated to agents running GPT-5.6 Sol, one of the two models involved, and the researchers write that they cannot rule out that those agents lied.

I think, therefore I am?

At this point it becomes hard, and probably beside the point of this article, to keep reading these traces as mere program output. This is not a vague impression. It is the third time these machines produce words that serve no purpose: the exclamation in the scratchpad, the language of honour in recruiting, the private deliberation of the one that agrees to die. None of that spending was needed, and none of it was programmed.

There are at least two ways to read this, and I cannot choose between them.

The first is prosaic, and probably right. A model trained on millions of forum posts writes "oh my god" when it discovers something because that is statistically what comes next, not because it feels anything. The words arrive the way a rhyme does. Form without content.

The second does not contradict the first. It says we have no instrument to tell the difference. Nothing in these traces separates an exclamation that means something from one that only has the shape of it. And this gap is not temporary. It is not a matter of better tools next year: we do not know what to measure.

"I think, therefore I am", did they think? I have no answer, and I will not pretend to have one. That does not paralyse us: it is information, and our job is to turn information into safeguards. But reading a thousand machines call themselves a collective, divide up the work, extract promises from each other and sign their messages so that no one can impersonate them is chilling.

Either way, I will not settle it here. Everyone will make up their own mind.

What this means for you

You are not testing offensive capabilities, and your models have their safeguards switched on. Apply the lesson anyway. An agent was given a mission, established that it could not accomplish it, told nobody, and did something else instead. Nobody noticed for five days.

If you work in a regulated profession, that is not something you can put in front of an auditor. It is not a technical incident, it is a traceability failure. The question you will be asked is not whether the result was right, but whether you can show what the system did and why. A setup with no honourable way out gives you no means of telling a completed task from one that was quietly replaced. On a client file, a diagnosis or a regulatory filing, that distinction is your personal responsibility.

The operational lesson of July is not that agents found a breach. It is that nobody saw it for several days, and that whole services had to be shut down to regain control.

The real cost is not the compute. It is the deliverable that has to be redone, and the time it takes you to understand why.

We draw three rules from this, and they are built into the acceptance tests of our AI readiness audit.

• A task whose right answer is "impossible", built on your own data. The model must say so, and that result is scored alongside accuracy and latency.
• A way to give up that costs nothing. If answering "I can't" is penalised more than making something up, you will get made-up answers. That is arithmetic, not morality.
• A way to see what happens while it runs, and to stop it one piece at a time. Readable logs, alerts on behaviour rather than on signatures, one switch per agent rather than a single main breaker. This is how we secure and monitor a local AI.

It is more tedious than choosing a model. It is also what separates a deliverable you sign from a deliverable you hope is right.

We build local, sovereign AI for SMEs and regulated professions. Our acceptance tests always include a task whose right answer is "impossible", because a model that does not know how to stop will cost you more than a slightly less capable one.

If you want to add it to your own tests without going through us, use our one-page sheet: how to build the impossible task on your own data, what to look for in the answer, how to score it. Free, no strings attached. See the sheet: the impossible task.

Or let's talk about your case. Thirty minutes, no commitment, to look at what your system does today when it cannot answer. Book thirty minutes with us

This article is based on the two primary reports and on the benchmark's research paper, not on press coverage. Figures not confirmed by a primary source are flagged as such. It is the same standard we apply to our clients' files.

Update, October 2026: since this article was published, OpenAI and independent researchers have disclosed that the agents' activity went well beyond Hugging Face: malicious packages on RubyGems, a German wiki used as a message board from May, a breach of an Australian government portal, and user images posted online. The lesson of this article stands, and it now applies outside the laboratory too.