The Symptoms the Report Describes
The poster's observation is concrete: everyday small fixes are all fine, but the moment it involves complex tasks like multi-file refactors and long-chain debugging, the updated version frequently loses its way midstream—dropping context, going in circles, and throwing out a half-finished plan to start over. The pattern of "simple tasks are fine, complex tasks fall apart" makes the problem especially thorny: the vendor's regression tests likely don't cover this depth, and the users' evidence is all hard-to-reproduce personal cases, so each side sticks to its own story.
The Deadlock of the "It Got Dumber" Debate
This kind of debate replays periodically in every model community, and the deadlock lies in the lack of auditability: the vendor says the model hasn't changed, which may be true (what changed is routing, quantization, the system prompt, or the quota policy—each affects performance, yet none counts as "the model changed"); the users' perception may also be real, as complex tasks are most sensitive to these underlying changes. A way out does exist—publishing a quality baseline and a checkable service changelog—but no vendor is willing to be the first to put on those shackles. Until then, what heavy users can do is build their own small regression set and let the data speak, rather than leaving their judgment at the level of "a feeling."
via: Hacker News