Use a Model You Can't Trust

You cannot trust what a language model says about its own output. The reporter and the reported share a failure domain, so the same defect that produces a wrong answer produces the confident verdict that the answer is fine. Taken at face value, that rules out building anything reliable on a model. It does not. The fix is a standard pattern that language models only made cheaper. Let the model propose. Let an oracle decide.

Split the system into a proposer and a decider

Give the two jobs to two components and never let them merge.

• The proposer generates candidates. It is tuned for recall: cover the space, be creative, be cheap, be wrong often. A language model is an excellent proposer.
• The oracle takes one candidate and returns a ground-truth yes or no. It is tuned for soundness: when it says yes, it is right. A language model is a terrible oracle.

The word is the one from software testing, where the test oracle is the component that decides whether a run passed. It names a source of correct answers, not the divining kind. The connotation that matters is ground truth, not prophecy: a crash, a type error, a balanced ledger. The model guesses; the oracle does not.

The property that makes this work: system correctness is a function of the oracle alone, not the proposer. A wrong proposal the oracle rejects costs nothing but compute. The only way a wrong proposal hurts you is if the oracle accepts it, so all of your reliability lives in the oracle and none of it lives in the model. You are then free to make the model as unreliable as you like, on one condition: it never gets to decide.

System correctness is a property of the oracle, not the model. Let the model be wrong; just never let it decide.

The worked example

Fareed Khan recently built an agentic security pipeline that finds, proves, and patches C memory-safety bugs, running on a small open 7B model on a single GPU, structured around Anthropic's find-and-fix loop. It demonstrates the pattern cleanly precisely because the model is weak and the system is still correct.

The model proposes a proof-of-concept input it thinks will crash the target. The host then re-runs that input against the binary itself, and a crash under AddressSanitizer decides whether a bug is real. The model proposes a patch; a ladder decides: it must compile, the original PoC must stop crashing, benign inputs must still pass, and a fresh discovery agent must fail to find a variant. The model writes; the sanitizer judges.

The control case is what happens when the same model is asked to judge. Pointed at its own confirmed findings as an adversarial verifier, the 7B model voted false-positive three times out of three, at 95% confidence, on bugs that crash the program on demand. The verdict was wrong every time, and the pipeline was still correct, because the vote was advisory and the crash was decisive. A model that rated three reproducible bugs as false positives drove a correct find-prove-patch loop, because it never had to be the judge.

Most "oracles" are fake

The pattern fails the moment you let a plausible-looking thing stand in for the oracle, and the most tempting fake is another model, or the same one. A real oracle has four properties, and a second LLM call has none of them.

• Outside the failure domain. The decider must not share the proposer's bug, bias, training distribution, or address space. A model checking a model fails this, the two have the same blind spots; a model checking itself fails it completely. AddressSanitizer lives in a different failure domain than Qwen, which is the whole reason its verdict carries information the model's does not.
• Ground truth, not opinion. The verdict is a fact the world produces, not a judgment rendered: a crash, a type error, a failing assertion, a signature that verifies or does not, a ledger that reconciles or does not. "Seems exploitable" is not an oracle. A segfault is.
• Decisive. It returns a binary and it overrules everything else. The instant the oracle's output is labelled "advisory," you are back to trusting opinion. In the pipeline the model's vote is advisory and the crash is decisive, and that ordering is the design.
• Cheap enough to run on every candidate. An oracle you run on a sample is a spot check; an oracle you run on every proposal is a gate. Re-detonation is cheap, so nothing passes without it.

A second model is not an oracle. It shares the first one's blind spots and signs the same wrong statement with more confidence.

The pattern is older than the model

None of this is new; generate-and-test is one of the oldest shapes in computing. The compiler's type checker decides whether your program is well-typed, regardless of what you intended. A property-based test decides, regardless of the cases you thought to write. Test-driven development puts the test before the code so the check, not the author, defines done.

Certificate Transparency stops trusting a certificate because a CA signed it, and starts trusting it because it appears in a public append-only log where a forged one cannot hide. TPM attestation moves the verdict to hardware the OS cannot forge. A proof assistant trusts the kernel that checks the proof, never the human who wrote it. In every case the trustworthy component is the cheap deterministic checker, and the creative component is allowed to be fallible.

What changed is the economics of the proposer. Historically, proposing was the expensive half: a human, or later a costly frontier API call, so you economized on proposals and over-trusted the few you made. A language model makes proposing nearly free and nearly universal. That does not make the model trustworthy. It makes it affordable to generate abundantly and reject almost all of it, which is exactly the regime the oracle pattern was built for. The model did not make the pattern possible. It made it valuable.

The cheaper the proposer gets, the more tempting it becomes to mistake its output for the achievement. The loud version right now is the boast that some team's code is ninety-nine percent AI-written. Linus Torvalds answered it with a dare: the same team could say a compiler wrote one hundred percent of their code, and nobody talks that way, because everyone understands a compiler is a tool and not an author.

The compiler earns no byline for the same reason this whole pattern works. Correctness was never a property of the thing that emitted the tokens; it lived in the type checker, the test, the crash. So "ninety-nine percent AI-written" is a number about the proposer, the half the system was already built to distrust, and it stays silent on the oracle, the only half that would carry any information. It is the attribution twin of "a human reviewed it": both name the cheap ninety-nine percent and say nothing about the one percent that decides.

Nobody credits the compiler with the program, because correctness lives in the type checker, not the codegen. The model has earned the same byline.

Building the oracle is the work

The cost did not vanish, it moved. An oracle is cheap to run and often expensive to build. The sanitizer verdict is one signal, but the throwaway-container sandbox, the build with ASAN wired in, the crash parser, and the re-detonation harness around it are real infrastructure. The pattern does not remove engineering effort; it relocates it, from the unwinnable project of making a model trustworthy to the tractable one of building a deterministic checker. The checker is reusable, inspectable, and identical every run, and model trust is none of those, so it is the better place to spend. But "just add an oracle" hides a lot. The oracle is the project.

The pattern does not remove the engineering. It moves it from aligning the model to building the checker, which is the better place to spend.

One gate is the exception; usually it is a ladder. You make "cheap enough to run on every candidate" true by ordering the checks cheapest first and letting each rung filter the stream: is it syntactically valid, does it compile, does it run without crashing, and only then the semantic ground-truth check. A noisy proposer floods the first rung, and each layer thins the flow so the costly check at the top only ever sees survivors.

The patch ladder in the example is exactly this shape: it builds, then the PoC no longer crashes, then benign inputs still pass, then a fresh attack finds no variant, each step stricter and pricier than the last. Design the ladder well and the per-candidate cost stays bounded no matter how much junk the model proposes.

Stateless oracles are the easy ones. A crash, a type error, a single assertion are local: ground truth is a property of one run. The hard oracles evaluate a delta over the whole system state. A data migration is correct only if source and target reconcile across the cluster, not if one row looks right.

An immutable OS update is good only if the entire measured image matches and the services still come up afterward. Stateful ground truth is harder to make deterministic, harder to make cheap, and harder to run on every candidate, which is exactly where most of the oracle-building cost ends up. The further your truth sits from a single local check, the more the oracle costs and the more carefully it has to be laddered.

Four seats, not two

The split names two seats, proposer and oracle, but a deployed system has four, and the human sits in two of them. Keeping the four apart is what stops "so where does the person go" from dissolving into hand-waving.

• The proposer generates candidates: tuned for recall, allowed to be wrong, once a human and now mostly the model. This is the seat "ninety-nine percent AI-written" is counting.
• The oracle decides pass or fail against a fixed criterion: deterministic, grounded in the world or in math. It was never the human, and automation cannot remove it, because a crash is a crash whether or not anyone is watching.
• The principal sets the criterion and owns the outcome: a person, or the organisation standing in for one. Not a vigilance task, but judgment and accountability for the result.
• The watcher is posted to monitor a reliable system and catch the rare miss. This is the seat that is theater.

The watcher is the impostor seat. It pretends to be an oracle, it "decides", while doing a vigilance task humans cannot do, and it owns nothing. Automation deletes it correctly and puts a real oracle plus reversibility in its place.

What it deletes is the impostor, not the principal, and the rule that kills the one cannot reach the other: the watcher fails because vigilance decays as reliability rises, while ownership concentrates as failures grow rarer and costlier.

You can automate the writing, and you can automate the checking, because the oracle draws its authority from the world and not from a model. You cannot automate the seat that decides what counts as success and answers for it when it does not, because a tool cannot be the addressee of that answer. So "how much of our code did the model write" is the wrong brag and the wrong interview question: it scores the proposer, the seat that was always allowed to be wrong, and says nothing about who holds the principal's.

Automation takes the human out of the loop. It cannot take the human off the hook.

Where it stops working

The pattern is only as good as the oracle, and oracles are not free to come by. Two honest limits decide whether you actually have one.

You need ground truth that is cheap to check. Memory safety has a sanitizer: an objective yes or no on every run. Most things do not. "Is this summary faithful," "is this the right architecture," "is this copy any good" have no crash to wait for. Where there is no ground-truth check, you are back to a human decider or to trusting the model, and wrapping a model's opinion in the word "verifier" only hides that you have no oracle, which is the legibility trap in a new costume.

So the first design question for any propose-and-check system is blunt: what is my oracle, and is it ground truth outside the model's failure domain? If the honest answer is "another model call," you do not have one yet.

The oracle gates precision, not recall. It can only rule on proposals that were made. The same 7B pipeline, pointed at a real library instead of a planted canary, found nothing, four reasonable attack ideas and zero proven crashes, because the proposer could not craft the inputs. The oracle had nothing to decide. A stronger proposer raises recall and the oracle stays exactly as it is, which is the quiet benefit of the split: the two halves scale independently. You can drop a better model into the proposer slot without touching the decider, and upgrading the unreliable half can never reduce correctness, because correctness was never its job.

Removing the bottleneck moves it

The reason to build the oracle is that you can then run the maker unattended: a schedule starts it, the oracle gates each result, state lives on disk, and the loop runs to a preset exit. That is the appeal of "remove yourself as the bottleneck," in Karpathy's phrase: put in a few tokens, get a lot of work back. But a working oracle gates exactly one thing, the output against the check, and two costs it cannot see grow the longer the loop runs without you.

• Comprehension debt. The loop merges faster than anyone reads. Every check stays green and the team stops understanding its own codebase. The oracle certifies that the output passed; it never certifies that a human knows what shipped. You removed yourself as the bottleneck on producing code and became the bottleneck on understanding it, and unlike the first, nothing forces you to notice. Green is not understanding, which is the keystone problem turned on your own repository.
• Harness drift. The oracle catches failures inside a run. It does not catch the harness around the loop, the prompts, the tools, the checks, the oracle's own thresholds, eroding as the models underneath change. That repair has no oracle; it runs on a human reading traces. The checker keeps the maker honest, and nothing keeps the checker honest.

The checker keeps the maker honest. Nothing keeps the checker honest.

Propose-and-check is still the right move; it just relocates the scarce human attention from generating output to two jobs that have no oracle: reading what the loop shipped, and maintaining the harness that judges it. Budget for those, or the green is lying again, one level up.

Reliability without a reliable model

This is the constructive side of the diagnostic pieces. The keystone argues a system's self-report is not evidence, because the reporter shares a failure domain with the reported. That reads like a dead end: if you cannot trust the report, you cannot trust the system. It is not. Route every commitment the system makes through a check the proposer cannot author or fake, and the proposer's reliability drops out of the correctness equation. The model can be wrong constantly. The system is right exactly as often as the oracle is.

The leverage moved with the economics. When proposing was expensive, you spent your judgment deciding which proposals to trust. Now proposing is nearly free, and the scarce component is the check the proposer cannot fake. The engineering is no longer in the model.

Stop trying to build a model you can trust. Build the oracle, and a model you cannot trust at all becomes safe to use.