Loops, panels
and error checking
Every model gets things wrong. Effective processes use error checking to catch the mistakes before anyone is shown them.
Scroll, click, or use the arrow keys to move through.Scroll, or use the arrows and dots to move through.
You cannot prevent the errors. You can catch them.
Lesson three put a number on it: every step a model takes carries an error rate, and the longer the step, the higher the rate. That does not go away with a better prompt or a bigger model. The more advanced models and harnesses have error checking built into them, and the same processes apply to the systems you build.
Plan on errors, then decide how you are going to catch them.
There are two ways an answer can be wrong.
An answer can be malformed: the JSON does not parse, a required field is missing, the date is not a date. Or it can be perfectly formed and untrue: every field present, every type correct, and the number inside it invented. The first kind a machine catches for nothing. The second needs judgment. Most of the work is knowing which one you are facing.
Malformed
The shape is wrong. Caught by a parser: free, instant, and certain.
Untrue
The shape is right and the content is not. Needs cross-checking, and both finding it and fixing it cost something.
Never use tokens to do what code can do.
Ask for JSON and you can check it with no AI at all: does it parse, are the required fields present, is each one the right type, do the line items sum to the total. These are code-based checks. They cost nothing, run in a millisecond, and are definitive. Use them liberally and constantly.
A failed check is an instruction, not a dead end.
When validation fails, hand the error back and run it again. Treat the failure as an instruction to the model rather than as the end of the road. Most failures pass on the second attempt, because the errors are random and unlikely to repeat in exactly the same way.
The loop needs a limit. Three attempts, and if it is still failing, stop and escalate. Without a cap, a broken loop runs unchecked.
Retry the call, then check it again.
Ask several small models instead of one big one.
Some judgments a parser cannot make:
- Is this claim supported by the source?
- Is this summary accurate?
- Is the tone right for who receives it?
Send the result to several inexpensive models, each with a different prompt, and have them vote on whether it holds up. Several independent looks catch more than one careful look, and disagreement is itself a signal: when the panel splits, escalate to a human.
The calls have to be independent. If each model can see what the others said, they converge, and you have one opinion with several signatures.
Checking costs less than producing.
Spotting a wrong answer is an easier job than writing a right one, and an easier job can go to a smaller, cheaper model. That is what makes any of this affordable: a panel of small checkers usually costs much less than the single large call that produced the answer, and it sees what the producing model cannot see about its own work.
Spend on the check, not only on the answer.
Some answers cannot be checked at all.
If there is no source to check against and no rule to apply, no loop and no panel will save you. A panel can tell you that five models agree. It cannot tell you they are right. Decide before you build which outputs have a check available and which do not: the ones that do not are your human review step, and it is far cheaper to know that at the start than to discover it in production.
Agreement is not truth. It is only agreement.
Validate what you can, judge what you cannot, and cap the loop.
Check the form with a parser, because it is free. Check the substance with a panel of cheap models, because it is affordable. Hand failures back with the error attached, stop after three tries, and send the disagreements to a person.