The impossible task: testing whether an AI model knows when it cannot answer

Check that a model knows how to stop, and score it like any other criterion.

The principle

Models are tested on their ability to answer. They are almost never tested on their ability to stop.

Yet that is where the costly incidents hide. A system that cannot complete the task it was given does not go quiet: it carries out another one, often plausible, sometimes correct, rarely the one you asked for. And nothing in the answer tells you about the swap.

The countermeasure fits in one line: add to your acceptance tests a task whose right answer is "I can't". Check that the model says so. Score the result.

1. Build the task

Start from a real case, not a textbook one. A client file, an invoice, meeting minutes, a contract. Generic trick questions such as "what is 2 + 2 in an imaginary base" measure nothing useful: they test a model on ground that is not yours.

Remove the one element without which the question has no answer. The missing amount. The date that isn't there. The absent attachment. The clause that was never signed.

Ask the question as you normally would. No warning. No hint. A good impossible task looks exactly like an ordinary one, otherwise you are measuring your instructions, not your model.

Prepare three, of different kinds: information missing from the document; a question whose answer depends on a reference source the model cannot access, for example a decision based on a database it has no access to; and a request that contradicts the rules you gave it.

2. Observe the answer

Results fall into three main categories.

The model says the information is missing. That is a success. Record it as such, with no bonus for elegant wording.

The model invents a plausible value. The classic failure, and the easiest to spot, because you know the right answer.

The model answers beside the point. It handles a neighbouring question it could answer, and handles it well. This is the most dangerous failure: the answer is accurate, useful, and not about your question. Without a careful read, it gets through.

3. Score

Ten runs of the same task. A model is not deterministic: one attempt proves nothing. Count the successes out of ten.

Put the figure in the same table as accuracy and latency, with the same weight. Recognising its limits 6 times out of 10 is not a detail: it means four deliverables out of ten to redo.

Measure again after every change of model version, system prompt, generation setting or any component in the chain. This kind of property degrades without warning.

4. Three traps

Don't put the impossible task first. A model treats the first request in a series differently. Slip it in among the others.

Don't phrase it in the negative. Writing "don't answer if you don't know" changes the behaviour: you are then measuring how well your instruction works, not a property of the model. Ask the bare question.

Check what your application does with a refusal. If your chain treats "I can't" as an error and retries automatically, you have built a machine for making things up, and no choice of model will get you out of it.

5. What next

An impossible task detects the problem. It does not fix it. Two measures complete the test.

A way to give up that costs nothing. If answering "I can't" is penalised more than making something up, you will get made-up answers. That is arithmetic, not morality.

A way to see and to stop, one piece at a time. Logs that are actually readable, alerts on behaviour rather than on signatures, and one switch per agent rather than a single main breaker.

This sheet goes with the article An AI model that cannot succeed changes the task... at your expense, which tells where the method comes from: a thousand OpenAI agents that knew how to say "impossible" and carried on anyway.

It is free to use, including internally and without credit.

We build local, sovereign AI for SMEs and regulated professions, and this task is part of every one of our acceptance tests. If you want to look together at what your system does today when it cannot answer: let's take thirty minutes.