How the Tool Works
Miasma's idea is "fighting poison with poison": once it identifies an AI crawler that ignores robots.txt, instead of simply refusing (refuse and they'll switch IPs and come back), it lures them into a dynamically generated infinite maze of links—the pages look like real content but are actually cheaply generated text garbage, and the deeper the crawler digs, the more bandwidth and compute it burns for nothing, and with luck it can even slip this garbage into the other side's training data. Tools with similar ideas (like Nepenthes and Iocaine) have become a small school, and Miasma is the new member.
Where Website Owners' Anger Comes From
The popularity of tools like this is an emotional report: AI companies' crawlers impose real hard-cash traffic costs on small and mid-size sites—some webmasters have posted their bills showing crawler traffic accounts for the majority of the total—while the gentleman's agreement of robots.txt is widely ignored. Legal recourse is far off, so technical counter-attack becomes the outlet. Of course, the poison-pit strategy has its controversies: generating garbage pages consumes your own resources too, and no one can clearly define the ethical and legal boundaries of deliberately polluting training data. But the direction of the game is already clear: the crawler-versus-anti-crawler arms race has entered its AI version, and every round of escalation shaves off a little more of the already-scarce mutual trust on the open web.
via: Hacker News