Google, Microsoft Bing and OpenAI said on October 30 that they will support one shared AI-training preference in robots.txt. Google’s crawler documentation and Microsoft’s post name the token, User-Agent: __usage_ai-training, which comes from an individual IETF Internet-Draft discussed in the AIPREF working group; OpenAI’s post speaks of “a common signal in robots.txt”. Every company’s wording is a plan. Google says it aims to support the control in 2027, OpenAI aims to implement it “in the coming months”, and Microsoft starts honoring it on February 1, 2027, as it crawls pages. It is one token, but Google and Bing route it to different switches.
What is the __usage_ai-training token?
It is a policy token that a robots.txt group can name in place of a crawler, so a site can state one AI-training preference for every participating provider. It adds no new directives and no new grammar, only a new name after User-Agent. It is a content-use preference, not a crawl directive: the draft says the token “is not a directive for crawling behavior”.
At Google it would also touch Gemini grounding
Google says that when a site opts out of training with the shared token and does not mention Google-Extended, it plans to read that as a Google-Extended disallow. Now read Google’s crawler documentation: Google-Extended covers training of future Gemini models, and also grounding, meaning content from the Google Search index supplied to the model at prompt time, in Gemini Apps and in Grounding with Google Search on Vertex AI. Put together, that is our reading, not a Google sentence: a “training” opt-out through the shared token would also remove a site from grounding in Gemini Apps.
Google’s blog describes the other direction. Sites that specifically allow Google-Extended “are eligible to appear in responses and receive clicks via grounding in Gemini Apps”. The same rule cuts both ways: an explicit Google-Extended allow next to the token keeps a site eligible for grounding in Gemini Apps, but by Google’s crawler documentation it also leaves the content available for training future Gemini models. Through Google-Extended, training and grounding travel together. Google’s crawler documentation also says Google-Extended does not impact a site’s inclusion in Google Search and is not used as a ranking signal there.
Bing keeps training and grounding apart
Microsoft’s Bing blog post splits the two. Training uses the shared robots.txt token, or a page-level setting that overrides it. Grounding is a separate page-level preference with three values: Allow (the default), DisallowExtended and Disallow. In Microsoft’s words, “There is no site-wide robots.txt grounding preference.”
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/That example, from the draft, expresses a preference against AI training everywhere except /press-kit/. Sites that still carry NoArchive or NoCache tags have a date to watch: Microsoft says it stops reading them as a training or grounding preference after March 1, 2027, and NoArchive or NoCache pages without a temporary grounding preference recorded in Bing Webmaster Tools by March 1 default to Allow for grounding until Microsoft recrawls them. Bing says the settings “do not establish licensing or payment arrangements”.
| Bing | ||
|---|---|---|
| Token maps to | Google-Extended disallow, if no Google-Extended rule is set | Training preference |
| Grounding control | Same Google-Extended rule (Gemini Apps, Vertex AI) | Page level; no robots.txt rule |
| Start | Aims for 2027 | February 1, 2027 |
The catch: wildcard rules do not count
Under the draft, a wildcard group such as User-Agent: * never matches the token, and if no group names it, no preference has been expressed. A blanket rule you wrote years ago is therefore not an AI training opt-out under this scheme. The draft is also individual: the datatracker says it is not endorsed by the IETF and has no formal standing. The draft’s author, Kevin Kelley, wrote on the working group list that Anthropic “has already started implementing support”.
Bing now has a date for something we noted on September 16. In our piece on Cloudflare’s Disallow AI Training setting, we reported Cloudflare saying robots.txt support for a no-training preference at Bing was “targeted for early 2027”. Microsoft now says February 1, 2027.
The practical step is to read your live robots.txt, including rules a CDN publishes for you, and see what each token actually says. Then decide per vendor what you want for training and for grounding, because the same opt-out does different things at Google and Bing: Bing keeps them as separate settings, while at Google one Google-Extended rule covers both. Our robots.txt generator builds the rules, and its User-agent field accepts any token.
