Why ChatGPT answers change every time you ask

Two different causes produce one symptom. The first is sampling: the model chooses each next token from a distribution rather than retrieving a stored answer, and the temperature parameter providers expose controls how wide that choice is, so identical prompts return different phrasing and a different order of named brands. The second is version change: the assistant you used last month may be running a different model today, because providers ship updates and retire older versions while the product name stays the same. Your own context adds a third layer, since chat history, personalisation and any live web results are all part of the input. Nothing has malfunctioned; this is how the systems are designed to work.

It means single answers are pointless as evidence, which is a smaller claim. Variation between runs is noise around a tendency, and the tendency is what a brand manager actually needs: whether the model reaches for you or for a competitor when it answers the questions your buyers ask. You get at a tendency by sampling many prompts repeatedly, not by asking once and screenshotting the result. That is the same logic behind any share of voice measure, carried into a source that happens to rephrase itself.

At onboarding you give a brand name, a description and up to three competitor names, and Seenby generates a set of 50 to 100 prompt templates from them, with a curated fallback set if generation fails or returns too few prompts. That set is then frozen. The same prompts run every week on OpenAI and Perplexity, so week-over-week movement reflects the market rather than a change in the questions being asked. The score is one headline Share of AI Voice figure per brand per week, computed over that whole set instead of any one answer. The step-by-step arithmetic is on the calculation page linked from this pillar.

Every stored response is tagged with the model version that produced it. When a week moves, you can see whether the answers in that week came from the same model version as the week before, which is the first question to ask before briefing anyone on a decline. Be clear on what tagging does: it separates model-update drift from real drift as a line of enquiry, it does not prove causation. If the version changed and the score moved, treat the movement as unattributed until the next week confirms it.

Yes, with its caveat attached, which is why the caveat text is stored with every computed score rather than living in a footnote somewhere. Seenby surfaces the methodology disclosure on every score display in the product and on a dedicated methodology page, so the number and its limits travel together. Say the limit out loud in the deck: a week-over-week shift may reflect a model update rather than a real competitive change. A score presented without that sentence is being presented as more precise than the underlying source allows.

Spot checks are good for one thing and bad for another. They are useful for reading how an assistant describes you in its own words, which is worth doing before a positioning change. They are unreliable as a trend, because your account carries history and personalisation, you will not ask the same 50 to 100 prompts by hand every week, and a single run tells you nothing about the spread. Use spot checks for texture and a fixed prompt set for direction.

Seenby does not publish a threshold, and we would rather say so than invent one. What we can say is that the sensible reading is directional: look at where the trend sits across weeks, check whether the model version changed, and look at which prompts gained or lost your name, rather than acting on a single week's delta. The editorial call about what counts as material stays with you, because it depends on your category and how contested it is. We give you the score, the caveat, the model version, and the answers underneath.

It will not claim the underlying answers are stable, reproducible or guaranteed, because they are not, and no tool in this category can make them so. It will not attribute a movement to a cause on your behalf; model-version tagging narrows the question, it does not close it. It measures presence relative to up to three competitors you name, so the score is about that comparison and not about the whole market. Seenby is in waitlist stage today, so nothing here describes results anyone has reported.