The Background of the Friction
Cloudflare sits at the throat of the internet, with a large share of the world's website traffic passing through its network, which makes it a natural gatekeeper in the war over AI crawling: it has launched a one-click feature to block AI crawlers and is exploring market mechanisms to let websites charge for AI access. Companies like OpenAI, meanwhile, both need web data and are being shut out by more and more doors. Every bit of friction between the two is essentially the same question: in the AI era, who has the right to read the open web, and at what price?
The Formation of a Gatekeeper Economy
Worth noting is the structural change: in the past the default of the open web was "readable," and the deal of search engines crawling in exchange for traffic sustained it for twenty years; AI broke this exchange, hauling away content without returning traffic, and so gatekeepers have found their moment. The Cloudflares are turning "controlling AI access" into a product and even a tollbooth—for content owners a belated bargaining power, for AI companies a cost, and for the web as a whole possibly the curtain falling on the open era: on the internet ahead, licensing and payment will gradually replace default openness. This story is just one small friction in that great turn, but the turn itself is worth watching.
via: Hacker News