How the study worked
Ask AI what percentage of salespeople give up after one follow-up, and ChatGPT, Perplexity, Gemini, Claude and Google AI Overviews all say 44%. AI search monitoring company Trakkr followed the citation Google AI Overviews gave: a sales firm's 2020 stats page, whose "source" link pointed to a freelance writer's sample article. That sample listed two slide decks, a blog list and two news stories, none of them a study of follow-ups. Trakkr tried every other route to the number it could find, and each stopped at a page that cites no research. It calls numbers like this, repeated over and over with no study behind them, "zombie stats".
The study, published on October 28, scales the same method up. It asked each of the five assistants 200 questions that call for a number, randomly sampled 1,285 of the statistics they cited (108 from ChatGPT, 277 from Perplexity, and 300 each from Gemini, Claude and Google AI Overviews), and followed each from the cited page to that page's source, up to five pages deep. A model judged every trail, a stronger model re-judged 300, and people re-checked 150 at random, with the rates corrected accordingly.
Results
- 40% (95% range 34%–46%) reached a study that says the same thing;
- 8% reached the study, but the number, who was asked, where or when had changed;
- 40% stopped at a source that couldn't be checked: paywalled, gone, or not actually containing the number;
- 11% had no study anywhere on the trail, with every page repeating the number and crediting no one.
By assistant, ChatGPT did best and quoted the fewest numbers, 1.3 per answer on average against 6.8 for Claude; the other four landed between 38% and 46%, within each other's ranges and too close to rank. To rule out bias from its own questions, Trakkr also traced 300 statistics from Google AI Overviews answers to 52,930 real everyday searches, and again found 40% sourced.
The problem is the citation chain, not invention
The assistants mostly weren't making numbers up: the first page they cited contained the figure 82% of the time. But only 21% of those first pages were original research; 54% were company blogs, and most of the rest were news stories and "statistics roundup" pages. Where a trail started decided where it ended. From a research firm or government page, 61%–62% reached a matching study; from a company blog, 37%; from a stats roundup, 35%. Even among statistics that reached the study, 17% had shifted along the way. The digits themselves moved in roughly a quarter of those; more often it was the context. In one case, a 2004 study of 15 women judging health websites turned into a claim about first impressions of every website.
Trakkr's conclusion: a citation tells you where the AI found a number, not where the number came from. Bear in mind that this is research from a company that sells AI search visibility services; it designed the question set, and trail judgments were mainly model-made with human spot checks, though the method and ranges are clearly stated. For anyone writing reports or choosing stories, the practical upshot is to click through to the original study before using a number an AI gives you.
via: Trakkr Research: Zombie stats: Where the numbers AI quotes come from