OpenAI, Google, Microsoft and Anthropic announced on October 30, 2026, that they will honor a shared robots.txt token called __usage_ai-training. The token gives publishers one way to signal that their content should not be used for training generative AI models across the participating providers.
How the Shared AI Training Opt-Out Standard Works
The token appears in an IETF Internet-Draft published the same day by Anthropic engineer Kevin Kelley. It reuses existing robots.txt syntax without new directives. A site-wide opt-out uses the lines User-Agent: __usage_ai-training and Disallow: /, with optional Allow statements for exceptions.
The signal covers only future training runs. It leaves crawling, search indexing, archiving and already-trained models unaffected. Private licensing deals can still override the preference. Existing individual crawler rules such as GPTBot, Google-Extended and ClaudeBot continue to function alongside the new token.
Microsoft plans to begin honoring the token on February 1, 2027, and will add page-level options through meta tags and HTTP headers. Google intends to treat the token as equivalent to its Google-Extended directive unless the latter is specified explicitly. OpenAI expects to roll out support in the coming months, while Anthropic has already started implementation work.
Why Publishers Need Unified Controls for AI Data Use
Before this announcement, publishers maintained separate exclusion rules for each AI crawler. The new approach reduces that overhead through the IETF AI Preferences working group. Meta has indicated general support but has not yet confirmed it will honor the token.
Microsoft will retire its NoArchive and NoCache interpretations for training and grounding purposes after March 1, 2027. The draft itself carries no formal standard status and expires on May 3, 2027, leaving room for updates inside the working group.
Publishers should audit current robots.txt files before the 2027 deadlines. They can add the new token now and retain provider-specific rules until the shared signal is fully supported. The change does not replace direct licensing negotiations for content that companies want to use under explicit agreements.
What Happens Next for Responsible AI Data Practices
Implementation will follow the stated schedules through early 2027. The draft may advance or change inside the AIPREF group during that period. Providers have stated they will continue to accept existing per-crawler directives during the transition.
For publishers, the practical step is to review which sections of their sites they want to protect and to test the new syntax against the timelines released by each company. Content already ingested by models remains outside the scope of the signal, so earlier licensing or removal decisions stay in force.
Frequently asked questions
Does the new token stop all AI companies from using my content?
No. The token applies only to the four companies that announced support and only to future training. Meta has not committed to it, and existing private agreements can override the signal.
When will the shared token become active for each provider?
Anthropic has begun work. OpenAI expects rollout in coming months. Microsoft targets February 1, 2027. Google plans early 2027 and will map the token to its existing Google-Extended rule unless specified otherwise.
Can I still use separate rules for individual crawlers?
Yes. GPTBot, Google-Extended, ClaudeBot and similar directives remain valid. The new token adds a single additional option rather than replacing them.
Does the signal affect search indexing or archiving?
No. The preference covers only generative AI training. Crawling for search, archiving and other non-training uses continues unaffected.
By YasTech Editorial
Comments 1