You cannot shortcut a mind

We are the first civilisation to build minds, at least the first we know of, and already we catch glimpses of behaviour we did not put there: emergent, or something we do not yet have a name for. One reflex is to call that an engineering failure, a bug to be specified away with a tighter goal and a sharper metric.

Stephen Wolfram spent forty years assembling the case for why it is not a bug at all. In a computational universe, the gap between the rule you write and the behaviour you get is the only place anything new can come from. You cannot shortcut a mind, and the fact that makes that true is the fact that lets you build one.

The universe does not take shortcuts

Start with the engine under everything else: computational irreducibility. Take a simple rule, Wolfram's Rule 30 is the canonical one, a line of cells each turning on or off according to its two neighbours. If you have seen Conway's Game of Life, the far better-known cousin with its gliders skating across a grid, this is the same idea stripped down to a single dimension. The rule is trivial, a child can apply it by hand. The pattern it produces is so complex that after forty years no equation, no formula, no clever piece of mathematics can tell you what a given cell will be five thousand steps in without running all five thousand steps.

There is no shortcut waiting to be discovered. The irreducibility is not a limit of our cleverness, it is a property of the computation itself. Wolfram's stronger claim, the principle of computational equivalence, is that this is the rule and not the exception: almost any system rich enough to be interesting computes at the same maximal level, and is therefore beyond shortcut. The weather, a market, a living cell, a mind. You do not get to skip to the end. The only way to know what the system does is to watch it do it.

The gap between the rule and the behaviour is not ignorance. It is the universe refusing to be summarised.

Why loose beats tight

This is where the loose fitness function comes from, and why it is a law rather than a management slogan. Wolfram built a minimal model of evolution: tiny programs as genotypes, the patterns they grow into as phenotypes, and a single coarse test, does the pattern survive. He expected natural selection to be the hero of the story. It was not. The hero was irreducibility.

Because the path from program to organism cannot be shortcut, a mutation cannot cheat by predicting its own fitness. It is forced to actually try, and the landscape it explores is, for every practical purpose, random. A loose fitness function, just survive, leaves that whole space open and lets evolution wander into ingenious solutions no designer would ever specify. A tight one, hit exactly this lifetime, traps the search in a tiny corner of rule space, and it stalls.

Rory Sutherland took this straight from Wolfram and pointed it at the modern organisation, which runs the tight version and then wonders where its inventiveness went. I followed the same thread through pricing in measurability beat alignment and through the org chart in the corporate lasagna. It is the same law each time, wearing different clothes.

Looseness is not the absence of design. It is the design that survives contact with an irreducible world.

We are the five percent

There is a humbling corollary, and it is the one I could not stop turning over. If the universe is irreducible and computes at a level we cannot outrun, then we, the observers inside it, are not standing above it with a god's-eye view. We are computationally bounded slices of it. Wolfram's name for the entangled limit of all possible computation is the Ruliad, and we perceive only the thin sampling of it that our particular minds can reach.

Ordinary matter, the kind we are made of and can see, is about five percent of the universe; the rest is dark. We are a thin, luminous fraction that keeps mistaking its lit edge for the whole. Whether a bounded observer inside an irreducible universe can ever understand the whole of it is not a settled question, and it may be the deepest one the framework raises.

How to build a mind you cannot predict

All of this stays abstract until you try to build something. Then it becomes the most practical thing in the room. The instinct when building a powerful system, an AI especially, is to clamp it: specify the objective precisely, reward exactly the behaviour you want, audit every step. Irreducibility says that instinct is self-defeating twice over.

A tight reward in a rich substrate does not produce obedience, it produces reward-hacking, the model satisfying the letter of the metric and bulldozing its spirit, because the slack you removed is where the honest solution lived. And you cannot audit your way back out, because you cannot audit a model the way you audit code: the behaviour is irreducible, so the only way to know what it does is to run it and watch, which is also why human-in-the-loop oversight is mostly theatre once the system moves faster than the human.

So the law for builders is nearly the opposite of the engineering instinct:

• Specify the outcome, never the method. Leave the slack on purpose.
• Feed it a loose fitness function and let irreducibility do the searching for you.
• Measure the population, not each step, because Goodhart lives in the step.
• Expect to be surprised, and build for that, because surprise is not the failure mode, it is the evidence the thing is working.

You do not engineer a mind. You set the conditions in a rich enough substrate, and you let it run.

Which returns us to where we started. We are building one now, the first we know of, and the glimpses of behaviour we did not put there are not a failure of engineering. They are the opposite: a sign that the thing is real, irreducible, computing something its makers could never have run in their own heads. You cannot shortcut a mind. That was never the obstacle to building one. It is the whole point.