Why I run every eval more than once

Before I trust any number out of an AI eval, I run it more than once - because the first time I didn’t, the number fooled me. I ran the same review on the same commit, with the same model and the same settings, five times. It gave a different answer two times out of five. So the common line that these models are basically deterministic at temperature zero is not 99% true, as you can see - it is closer to 60%.

That is a forty percent disagreement rate - too big to write off as rounding error.

Temperature zero is not deterministic

The claim I keep hearing is that if you run a model at temperature zero it is deterministic - same input, same output, every time. It isn’t.

Even at temperature zero, the computations behind one generation run in parallel, and a slight rounding difference on floating-point numbers can lead the model to select a different token in rare cases. Once that one token flips, the whole generation downstream of it changes. So the output is not 100% deterministic, and on a real review that drift showed up as the two-in-five I measured.

The cost is a wrong deployment decision

A single run can put the wrong system into production - and you may never go back to re-check the decision.

On one set of reviews, Haiku - the cheapest of the three models I was testing - caught a bug that both Sonnet and Opus had missed. On the face of it, Haiku looked like a bargain: the cheap model catching what the expensive ones let through. Then I ran it four more times. It caught that bug once in those four runs; the other three missed it entirely. So the real catch rate was two in five. If I had trusted that first single run, I would have concluded that Haiku reviews my repo well and might have deployed it as the reviewer - when in reality it catches that bug 40% of the time. If you are the VP deciding which model to put on your team’s reviews, that single run is the whole basis for the call - and it would have sent you to the wrong model.

The gate fooled me, too

I want to be clear that this is not a mistake other, more careless teams make. It humbled me on my own work.

I had built a multi-agent review gate - several specialist reviewers, each looking at the change from one angle, and a judge on top that consolidates them into one verdict. It is the design a lot of people are copying right now. I built it to test against a single-prompt reviewer - three models, a dozen commits - and that head-to-head has its own writeup. On one of the control commits - the ones I believed were clean - every other model I ran had flagged it as a bug. The gate did not. It marked the finding a false positive and passed the commit. I thought: clearly the gate is working, the extra machinery earning its keep. I predicted it would clear that false positive every single time.

Then I ran it in repeats. It flipped. The clean verdict I had trusted was the minority outcome - the gate passed that commit two times in five and blocked it the other three. What I had read as the architecture working was just noise. Even when I try to practice this rigorously, I get fooled at times and get humbled by the models.

What the popular eval advice skips

The dominant practitioner playbook on LLM evals does not cover this. If you read Hamel Husain and Shreya Shankar’s evals FAQ - and most people building evals have - the guidance is to do error analysis on at least a hundred traces, score binary pass/fail instead of a 1-to-5 scale, and align your LLM judge to human labels so its true-positive and true-negative rates hold up on a held-out set. All of it is correct, and I follow it. None of it is about run-to-run variance or statistical power.

That kind of rigor does exist, and it is under-adopted. Evan Miller’s Adding Error Bars to Evals treats your eval items as a sample from a population, analyzes the paired difference between the two systems rather than each system’s raw score, and uses a power analysis to size how many items you need. Error analysis on a hundred traces is worth doing, but it will not tell you whether the traces you read would come back the same on a rerun.

How many runs is a business decision

Intuitively it makes more sense to me to design the experiment on a statistical footing than to trust a single run. Power analysis is central to inferential statistics. It is how you decide, before running anything, how many measurements you need for a real difference to show up instead of getting lost in the noise. But I am not sure how to translate that cleanly to LLM evals, because the tradeoff is real and you have to be pragmatic - evals are not free. It needs a proper cost-benefit analysis: what error are you willing to accept, and what will you pay to narrow it.

So the number of runs is a business decision. An error bar is the range the true rate could plausibly sit in around the number you measured. Too few repeats leaves that range wide; more repeats narrow it, because the extra runs average out the same run-to-run noise that gave me the 40%. You do not set it with a blanket rule like “always run five times.” And I do not set it by how long the decision will last, either - I first framed this as a “six-month decision” and then rejected that framing. I would rather frame it on the severity of the context the module operates in. Some modules matter more than a normal one.

Take a module that pages on-call the moment a safety-critical signal trips - the alerts a system fires when something is going wrong and someone has to act on it. A mistake there can cause harm that is irrevocably expensive. Where being wrong costs that much, I want to be far more sure the reviewer actually works before I trust it - and the only way to get there is more repeats, which is what pulls the error bar tight. So that module earns the extra runs. Now take a module that sends transactional notifications - a receipt, a reminder. If it fails, no big deal per se. That one can live with a wider bar and fewer repeats.

This is the same risk-tiering I already apply to review depth - critical paths get more scrutiny - moved one layer up, to how hard I measure the reviewer itself. Running something once gives you no real confidence in the answer, even when you are treating the decision as certain. Do the cost-benefit on your eval spend the way you do it on everything else you buy.

The routine at eval time

You never run these repeats in production. That is eval-time discipline - you pay for the repeats once, while you decide what to deploy, not on every live request. The routine I follow:

1. Pilot the flip rate. Run about ten repeats on a few items and see how often the answer flips between identical runs. Mine flipped about 40% of the time, which is high; if yours holds steady, you can do fewer repeats and spend the budget elsewhere.

2. Name the effect size that would change your decision, before you run. Something like “the gate must beat the single prompt by at least half a catch out of six,” or “the false-positive rate must be under 0.2.” Without that number written down first, you cannot tell a real difference from noise after the fact.

3. Two knobs that do different jobs. Repeats measure how often one item’s answer flips between runs. Items per class - a class being one kind of item, like your known-bad commits or your clean controls - measure how much the answer varies from one item to the next, and are what let you generalize past the handful you tested. Five repeats is a floor - below it you cannot even see that an item splits - but five clean runs cannot certify stability. A clean five-out-of-five looks reassuring, but zero failures in five runs is still consistent with a failure rate high enough to matter; to be 95% sure it is at least 95% reliable you would need about 59 clean runs in a row (small samples make the intervals themselves unreliable).

4. After five repeats, add items instead of piling on repeats. Under a fixed budget, once you are past roughly five repeats, an extra item does more for your confidence than a sixth repeat of one you already have - by then you have mostly pinned down that one item’s rate, and what is still uncertain is whether it holds across other items. And be honest about your bars: my own set - six known-bad commits and six matched controls, five repeats each - gives about ±0.18 on a control block-rate near 0.5. Those are fat bars.

5. Analyze paired, and over items. Run the same items through both systems and compare them item by item, with a mixed-effects model or a bootstrap over items - both are standard ways to put an error bar on this kind of grouped data. Do not pool every run into one flat proportion - the five runs of one item are correlated, so pooling makes that bar look far narrower than it really is. Label each item flaky or stable, and never publish a per-item rate as though it were precise.

Open questions

There are a few things I have not settled.

I do not have a clean default repeat count for a given severity tier. Severity points me at which modules deserve tighter bars, but not at how many runs get me there. That number still comes from the measured flip rate, case by case.

My own bars are wide - six items and six controls. So what I trust out of this is the direction and the rough flip rate. The bars are too wide to lean on a second decimal. Tightening those bars means more items and more budget - the same cost-benefit call, now aimed at my own experiment.

And I do not know whether the 40% flip rate holds anywhere else. It came from three models across a dozen items on one codebase, measured once. Your repo, your prompts, or a newer model could move it either way. One codebase cannot tell me which.

---

P.S. Measuring run-to-run variance like this is one part of the paid diagnostic I run for engineering teams: I score your AI-assisted development workflow across 20 areas against the industry frontier, from your own artifacts - rules files, sample PRs, CI config, your own numbers - and hand back where you stand against your peers and a prioritized 90-day plan. If that is worth doing on your team, it starts with a short written application, not a sales call. You can start here.