One robots.txt Group to Opt Out of Several AI Companies' Training: Anthropic Proposes the Shared __usage_ai-training Token at the IETF, With Google and Microsoft Supporting It in 2027

One token instead of a list of names

A site owner who doesn't want their pages used to train AI currently has to write rules company by company: block GPTBot, add Google-Extended, look up the next company's crawler name, and keep up as the list grows. On October 30, Kevin Kelley, a technical lead for standards at Anthropic, submitted an individual Internet-Draft to the IETF that offers a cross-vendor way to write it:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Those three lines say that everything except /press-kit/ should not be used for AI training. The draft calls __usage_ai-training a "policy token": it borrows the position of a User-Agent line but never needs to appear in any crawler's request headers. Path matching and longest-match precedence come straight from RFC 9309, with no new directives and no new grammar, so older parsers ignore it the way they ignore any crawler they don't recognize.

Easy things to misread

• Wildcards don't count. User-Agent: * does not match the token, so existing wildcard rules are not read as training preferences; a file without the token expresses no preference.
• It's about use, not crawling. It is independent of each crawler's own fetch rules; the draft only suggests that a crawler fetching purely for training may skip content where training has been declined.
• "Training" has limits. The definition comes directly from the IETF AIPREF working group's vocabulary draft: using content to modify the learned parameters of a generative AI model. Models used only for classification, ranking or scoring are not included.
• It isn't a standard yet. This is an -00 individual draft that expires on May 3, 2027 and has no formal standing at the IETF; it expresses a preference, not access control.

Who supports it, and when

On the AIPREF working group's mailing list, Kelley wrote that Anthropic has already started implementing support for the draft, complementing rather than replacing its existing crawler controls. Google said on the Search Central blog the same day that it plans to support the token alongside other AI providers in 2027: when a site opts out with the token and does not address Google-Extended explicitly, Google will treat that as disallowing Google-Extended, while sites that specifically allow Google-Extended can still appear and get clicks through grounding in Gemini Apps. Microsoft said on the Bing blog that it will honor the token as it crawls from February 1, 2027, and is adding a separate page-level AI Grounding preference, set with tags like <meta name="Bingbot" content="ai-grounding:disallow">, that controls whether content can inform responses in Bing and Copilot; neither preference affects eligibility for Bing search results, and after March 1, 2027 NoArchive and NoCache will no longer be read as training or grounding preferences. According to The Verge, OpenAI says it will implement the signal "in the coming months"; OpenAI's own page sits behind a human-verification check, so we could not check it directly.

For site owners, adding the group now costs little, since parsers that don't know it skip it. But until the companies actually enforce it, keeping out of training still requires the existing rules for each crawler by name.