How many times do I have to run a prompt before the result means anything?
Enough that you are reporting a rate rather than an event. One run is a coin flip. A few runs cannot distinguish a source cited a third of the time from one cited two-thirds of the time. A workable floor is ten runs per prompt in clean sessions, reported as a citation rate — and treat gaps smaller than roughly twenty points as noise unless they persist across several weeks.
You are estimating a proportion, so the math is old
The uncertainty in a proportion estimated from n samples scales with the square root of n. Roughly, the margin of error is about 1 divided by the square root of n. Ten runs gives you a band of roughly plus or minus thirty points on a single prompt. A hundred runs narrows it to about ten.
That arithmetic is the entire reason single-run 'tests' are worthless, and the reason you should be suspicious of any tool that reports a visibility score to one decimal place without saying how many samples produced it.
Breadth beats depth
Given a fixed budget of runs, twenty prompts at ten runs each is far more useful than one prompt at two hundred runs. Precision on a single question you invented tells you about that question. Coverage across the questions your buyers actually ask tells you about your business.
Build the prompt set from real language. Call recordings are the best available source of how people phrase problems before they know the vocabulary, which is exactly the phrasing assistants get asked. That is a practical side benefit of analyzing your calls — the questions are already sitting in your phone system.
Control the session or the numbers mean nothing
- Signed out, fresh context. Prior turns bias the result badly.
- Consistent, recorded location. Local answers are location-bound; a result from one metro does not transfer.
- Recorded date and product version. Vendors ship changes without notice, and a step change in your trend is often theirs, not yours.
- Same wording every time. If you improve a prompt, add it as a new one. Never edit a prompt that has history, or the trend line is a lie.
Report the uncertainty, not just the number
A visibility report that says '40 percent citation rate across 20 prompts, 10 runs each, signed out, Phoenix metro, week of the 14th' can be argued with productively. A report that says 'AI visibility: 62' cannot.
Owners are good at spotting numbers that cannot be audited, and they discount everything else you say once they find one. The same standard applies here as to any revenue reporting: state the method, state what is excluded, and let the number be smaller but real.
Topics: measurement · sampling · statistics · testing
Have a version of this question about your own business?
The useful answer usually depends on which systems you run and how they're connected. That's a conversation, not a blog post.