🔍 Read the full analysis: What You Should Know About The Shared AI Training Opt-Out on ThorstenMeyerAI.com
Prime made for students and young adults
- Fast, free delivery for dorm and study essentials
- Prime Video and Amazon Music included
- Member-only deals
TL;DR
OpenAI, Google, Microsoft and Anthropic have announced support for a shared robots.txt signal that tells participating companies not to use a website’s content for foundational AI training. The signal is voluntary, is not an official internet standard, and does not block AI answers based on live web content; rollout dates differ, while Meta has expressed support without committing to honor it.
OpenAI, Google, Microsoft and Anthropic have said they will support a shared website signal asking that content not be used for foundational generative AI training. The proposed robots.txt rule is intended to replace some company-by-company opt-out steps, but it is a voluntary preference—not a technical block—and the companies’ implementation timelines differ.
The shared signal is defined in an Internet-Draft titled Machine-Readable Training Preference Signal, published October 30 by Kevin Kelley of Anthropic. A site owner who wants to request exclusion can add a robots.txt group with the user agent __usage_ai-training and the directive Disallow: /. The proposal expresses a preference against using the site’s content to train generative AI models; it does not prevent a crawler from accessing the site.
Anthropic says it has started implementing support, while noting it will take time to extend support across its services. OpenAI says it aims to implement the signal “in the coming months.” Microsoft says it will honor the shared training preference starting February 1, 2027, as it crawls sites. Google says it aims to support the signal in 2027; its crawler documentation specifies “early 2027.”
Microsoft also announced a separate page-level preference for AI Grounding—content used to inform AI answers in Bing, Copilot and Microsoft 365. Microsoft says both preferences take effect February 1, 2027. Its existing NoArchive and NoCache tags will stop counting as training or grounding preferences after March 1, 2027. Meta communications chief Andy Stone voiced support for publishers expressing preferences, but did not commit Meta to honoring this signal.
Infographic 01 · Who backs it, and when
One line in robots.txt, four companies, four timelines.
A shared signal against AI training. Only Anthropic says it has started; the others name dates or months. The standard itself is still a draft.
- 30 Oct 2026Draft published, four companies announce supportdoneIndividual Internet-Draft by Kevin Kelley (Anthropic)
- 30 Oct 2026Anthropic: “has already started implementing support”startedIts standards lead on the IETF list: “It’ll take us some time to add support across our services.”
- Coming monthsOpenAI aims to implement itplanExisting GPTBot and OAI-SearchBot rules keep applying
- 16 + 19 Nov 2026Working group meets at IETF 127, San FranciscoopenOn 21 Oct a chair said no consensus has been declared yet
- 1 Feb 2027Microsoft honors the new preferences as it crawlsplanTraining in robots.txt or per page; grounding per page
- 1 Mar 2027Bing stops reading NoArchive / NoCache as AI preferencesdeadlinePublish the new settings before this date
- 2027Google plans support; its docs say “early 2027”planA “no” maps to Google-Extended unless you set that token yourself
- 3 May 2027Draft version 00 expires unless updatedInternet-Drafts are valid for at most six months
Binding? No, voluntary
- OpenAI calls it a “voluntary effort”.
- The vocabulary draft does not “ensure that preferences are followed”.
- The draft “has no formal standing in the IETF standards process”.
- Private agreements may override it, the draft says.
Who else?
- Meta supports a common signal, but did not say whether it will honor it.
- Apple, Amazon, xAI, Perplexity: no statement in the sources reviewed for this article.
- The draft’s acknowledgments list contributors “for identification only”, not supporters.
Sources: IETF draft, 30 Oct 2026 · Anthropic on the IETF list · AIPREF minutes, 21 Oct 2026 · Google, 30 Oct 2026 · Google crawler docs · OpenAI, 30 Oct 2026 · Microsoft Bing, Oct 2026 · Meta (Andy Stone on X). Company plans are company statements.
Powered by Thorsten Meyer AI
A Simpler Request, Not a Block
For publishers, the proposal could reduce the need to maintain a separate training opt-out for each participating AI company. A common token gives site owners a single way to express the same preference to multiple services, if those services implement and honor it. That may make publisher choices easier to communicate, particularly as the companies’ controls and product uses have developed separately.
But the practical effect depends on which companies adopt the signal, when they do so, and how they interpret it. The rule is a request in a machine-readable file; it does not technically stop access. The announcements also do not establish payment, guarantee that a publisher’s material will be excluded from every use, or promise search traffic. Microsoft’s own statement says neither choice guarantees “inclusion, traffic, or payment.”
Publishers should also distinguish model training from AI answers generated using live web content. The shared token covers training, not those answers. Microsoft has announced its own separate grounding preference, while Google and OpenAI describe the shared control as a training signal. A site owner who wants to set different permissions for training, AI answer products and ordinary search may need to keep using multiple controls.
Popular right now
De'Longhi 15-Bar Pump Espresso Machine with Milk Frother & Temperature Control
Ninja DualZone 6-in-1 Air Fryer, 8-Qt, Grey, DZ801
De'Longhi Magnifica Evo Fully Automatic Espresso Machine with Manual Milk Frother, Silver
As an affiliate, we earn on qualifying purchases.
Before this announcement, website operators generally had to use company-specific crawler rules to express training preferences. Those include GPTBot for OpenAI, Google-Extended for Google and ClaudeBot for Anthropic. Microsoft has also used NoArchive and NoCache page tags for relevant preferences. The new signal is designed to provide a common option, but the companies say their existing controls continue to apply during rollout.
Google says that when a site uses the new token and has no Google-Extended rule, it plans to interpret the shared signal as though the publisher had disallowed Google-Extended. Google also says Google-Extended does not affect Google Search. Microsoft says its preferences do not change eligibility for Bing search results. These distinctions mean opting out of training is not the same as opting out of search.
The proposal is an individual Internet-Draft associated with the IETF’s AI Preferences working group, not an adopted internet standard. At the group’s October 21 meeting, a chair said, “We haven’t declared consensus on anything yet.” The draft’s authorship and status matter: it describes a proposed technical convention, while company announcements describe voluntary plans to use it.
““has already started implementing support for this draft””
— Anthropic, in its announcement of support
Adoption and Coverage Remain Open
Meta’s position is not an implementation commitment. Its communications chief said the company supports publishers’ ability to express preferences and called a common signal an important step, but did not say whether Meta will honor it. The source material also reports no statement from Apple, Amazon, xAI or Perplexity; that does not establish whether those companies will adopt the signal later.
The proposal’s eventual status in the IETF process is unsettled, and the companies have not all supplied the same level of detail about rollout. Anthropic’s crawler help page, as of October 31, still described its existing per-crawler controls. It is also unclear how consistently a voluntary request will be applied across each company’s services and content sources. The signal does not address material already used to train existing models, nor does it provide a mechanism for removing that material.
Watch the 2027 Rollout
Publishers can add the shared training preference while retaining company-specific rules until each relevant provider has implemented its announced support. They should check crawler documentation and company notices as the rollout proceeds, rather than assume the shared line is already recognized everywhere. Sites using Microsoft’s NoArchive or NoCache tags for training or grounding preferences should review Microsoft’s announced March 1, 2027 cutoff for those tags’ role as preferences.
The next milestones are Anthropic’s continued service rollout, OpenAI’s planned implementation in the coming months, and the 2027 start dates announced by Google and Microsoft. Further company commitments and the IETF working group’s consideration may clarify how broad the shared signal becomes. Until then, the key distinction remains: it is a common way to state a preference, not a universal enforcement mechanism.
Key Questions
What does the shared AI training opt-out do?
It lets a website publish a robots.txt preference asking participating companies not to use its content for foundational generative AI training. The proposed user agent is __usage_ai-training, with Disallow: /.
Does the rule prevent AI companies from crawling my site?
No. The signal is a voluntary request, not a technical barrier to crawling. Its effect depends on whether a company supports and honors it.
Does opting out stop my pages from appearing in AI answers or search?
Not by itself. The shared signal covers training, not AI answers based on live web content. Google and Microsoft say their described controls do not change eligibility for their respective search results; Microsoft has announced a separate AI Grounding preference.
When will the companies support the signal?
Anthropic says implementation has started; OpenAI aims to implement it in the coming months. Microsoft’s announced start date is February 1, 2027, and Google says it aims for early 2027. The dates and levels of implementation detail differ.
Is this the setting for my personal AI chats?
No. The proposed rule is for website owners and publishers who want to state a preference about use of their site content. It does not change account-level data or training settings for an individual’s chats with an AI assistant.
Source: ThorstenMeyerAI.com
