Frontier LLMs Disagree on Real-World Fact-Checking

A test fed the same batch of real-world fact-checking questions to several frontier models, and their answers diverged more than expected—none of them can serve as the referee.

What the Test Found

The experiment design isn't complicated: take statements that are contested in reality or need verifying, ask each vendor's flagship model separately, and compare the conclusions. The result is that the models frequently contradict each other, and each gives its answer in the same confident tone. Worse still, the disagreement has no stable pattern—it's not that one vendor systematically leans left or right, but that it wobbles randomly from question to question, so even the folk remedy of "ask several and take the consensus" can't save it.

So How Can You Still Use It

A model's training objective is to generate coherent, plausible text, not to verify truth; whether it's reliable on factual questions depends on how completely and cleanly the training data covers the topic, and real-world controversy is precisely where the data is muddiest. The viable use is to make it an assistant for retrieval and synthesis: give sources, list evidence, and leave the conclusion to a human. If you skip the sources and just ask for the answer, on contested topics you're basically rolling the dice.

via: Hacker News