The systems that hold aren't the cleverest. They're the ones that know where not to be.

There’s a temptation I recognize well, because I fell for it myself. When a tool is capable of reasoning, it feels natural to hand it everything. If it can read a document, let it also choose where to file it. If it can write a reply, let it decide when to send it.

That’s the moment systems become brilliant in demos and unreliable in daily life.

The things I’ve seen run for months, without constant supervision, all share the same shape. There’s a part that never reasons, and a part that never acts. The first does dull, predictable work: it takes a file, puts it where it belongs, updates a record, sends a notice. It always does the same thing the same way, and when something doesn’t add up it stops and says so.

The second watches, cross-checks, proposes. It touches nothing.

Keeping them apart looks like a pointless complication, and in the first weeks it is: you write more code to do less. The advantage arrives later, when something goes wrong, and it’s a different kind of advantage from the one you expect.

It isn’t that you get fewer errors. It’s that you always know which half made them.

If the dull part fails, it fails loudly and repeatably: there’s an error, you read it, you fix it, it won’t happen again. If the reasoning part proposes something silly, the silly thing stays a proposal, and someone discards it before it becomes a fact. Neither error is irreversible.

In systems where the two are blended, every failure is the same kind: something happened, nobody is quite sure why, and it isn’t clear whether it will happen again.

Out of this comes a rule I use when deciding what to trust with what. I don’t look at how hard a task is. I look at whether it’s reversible. Actions that can be undone are allowed to be wrong: they’re a cost, not damage. Actions that can’t be undone, the ones that go out into the world, that touch money or people or legal obligations, need a human signature even when the machine is right.

Especially when it’s right, because that’s when people stop checking.

The short version is that the right question, faced with any automation, isn’t what it can do on its own. It’s a different one, and it comes before you start.

What must it never be allowed to do on its own, even when it’s right?

The list that comes out is short, usually shorter than anyone feared. But it’s the difference between something you switch on once and it keeps working, and something that goes quiet after three weeks because nobody trusts it any more.