The Cost Structure Is Shifting
People are used to understanding the AI chip race as a race of logic chips, but the cost sheet tells another story: high-bandwidth memory (HBM)'s share of an accelerator's bill of materials climbs generation over generation, and on some products it's already on par with the compute chip itself. The reason is that models keep getting bigger and inference keeps getting more bandwidth-hungry, while HBM stacking is a difficult process with a slow yield ramp, and only three vendors worldwide can expand capacity. The compute bottleneck is shifting from "can't make the chips" to "can't feed the chips."
Who Benefits from This
The beneficiary list is short: SK Hynix, Samsung, Micron, plus TSMC doing advanced packaging. Memory makers stand in a position of power in the supply chain for the first time, and their bargaining power and capital expenditure are both being re-rated. For downstream, the pressure of price hikes gains another source—even if the GPU supply eases, tight memory can still hold up the price of a whole system. From this year on, when looking at AI hardware costs, you can't watch only the GPU—you have to factor memory prices in too.
via: Hacker News