How to Opt Out of AI Training in 2026: The New Single robots.txt Line Explained

If you run a website, you have probably watched the list of AI opt-out rules grow longer every year. One rule for Google, another for OpenAI, another for Anthropic, another for Microsoft — each with its own crawler name and its own syntax. On 30 October 2026, Google, Microsoft and OpenAI announced plans to simplify that mess with a single shared line in robots.txt. Search Engine Journal reported the announcements on 1 November 2026, and the story quickly spread through the day's AI news roundups.

There is one catch you need to hear first: none of the three companies supports the new line yet. Adding it today does nothing on its own. This guide explains what the new token is, when each company says it will work, and — most importantly — what you should do with your robots.txt file right now while you wait.

What is the new __usage_ai-training token?

The token comes from an Internet Draft submitted on 30 October 2026 by Anthropic's Kevin Kelley to the AI Preferences working group at the IETF. It is listed as an individual draft, which means it is not an official IETF standard yet — it is a proposal that companies are choosing to adopt.

The idea is simple. Instead of writing separate rules for every company's crawler, a website owner would write one ordinary Allow and Disallow group under a policy token called __usage_ai-training. Every participating provider would read that one group and understand the site's AI-training preference. One line, understood everywhere — at least in theory.

Here is the part that trips people up: the token does not control crawling. It does not stop a search engine from visiting your pages, and it does not remove your content from search results. It only states a preference about training — which the working group's vocabulary draft defines as using content to modify a generative model's learned parameters. Non-generative work, such as classification or ranking, is exempt. And a robots.txt file with no __usage_ai-training group signals no preference at all, which is true of every robots.txt file published today.

What each company actually announced

The announcements sound similar, but the details differ in ways that matter. Here is the honest summary:

Company	What it plans	When
Google	Read a shared opt-out as if Google-Extended were disallowed, covering future Gemini model training and grounding in Gemini Apps and on Vertex AI	Aims for support in 2027; crawler documentation says early 2027
Microsoft	Adopt the shared line for training, plus a separate page-level AI Grounding preference (robots.txt, meta tag or HTTP header)	Honouring new options as it crawls from 1 February 2027
OpenAI	Honour the shared signal; current robots.txt policy stays in effect meanwhile	Within the next few months
Anthropic	Development on support has begun; no newsroom announcement as of 30 October	Rollout across all services will take time

Two deadlines deserve a calendar reminder. Microsoft will stop recognising NoArchive and NoCache as training or grounding signals after 1 March 2027. If you currently rely on those tags to limit Copilot, Microsoft says you must record a temporary preference in Bing Webmaster Tools — a setting that opens on 15 December 2026 and is only available to sites still using the tags on that date. Pages without one will default to allowing grounding after 1 March until Microsoft recrawls them.

How the robots.txt snippet will look

When support arrives, the shared preference will live in its own group in your robots.txt file. A site opting out of AI training would publish something like this:

__usage_ai-training: Disallow: /

And a site allowing training would publish:

__usage_ai-training: Allow: /

The draft adds no new directive of its own — it reuses the familiar Allow and Disallow pattern inside a group named for the token. One technical detail is easy to get wrong: a wildcard group (User-Agent: *) can never match the token. The preference only counts when it sits in its own __usage_ai-training group. Rules you already have in your robots.txt file for other crawlers do not count as a shared training preference.

Until the companies actually support it, treat this snippet as preparation, not protection. Publish it if you like — it is harmless — but do not remove your existing per-company rules yet.

What to do right now (while support is pending)

Here is the practical checklist for today:

1. Keep your existing rules. Google-Extended for Google, GPTBot for OpenAI, and ClaudeBot for Anthropic remain the controls that actually work today. Removing them early would leave you with less protection, not more.
2. Check your robots.txt file. Many site owners have never looked at theirs. It lives at yourdomain.com/robots.txt. If you host your site on a plan that gives you file access, such as the Hostinger 3-in-1 Bundle, you can open and edit it directly. If you built your site with an AI builder, check whether it lets you edit robots.txt — tools like Lovable Pro and Bolt Pro let you publish a site you control, so confirm where your robots.txt settings live before assuming anything.
3. If you use NoArchive or NoCache for Copilot, diarise December 2026. Set the temporary Bing Webmaster Tools preference when it opens on 15 December, or your grounding preference will flip to "allow" after 1 March 2027.
4. Decide training vs grounding separately. This is where Google and Microsoft diverge. At Google, Google-Extended bundles training and grounding together, and Google's crawler documentation describes no way to block Gemini training while still allowing Gemini Apps to use your content for grounding. At Microsoft, the two are separate preferences. If that distinction matters to your business, your policy choice at each company will differ.
5. Re-check in early 2027. Google's early-2027 window and Microsoft's 1 February 2027 start date are the first moments the shared line should actually do something. Verify support in each company's crawler documentation before relying on it.

Common mistakes to avoid

• Assuming silence means opt-out. It does not. No __usage_ai-training group means no preference stated.
• Putting the token under User-Agent: *. A wildcard group can never match it. It needs its own group.
• Expecting it to block crawling or search indexing. It does neither. Crawl control still lives in ordinary robots.txt rules and meta tags.
• Deleting per-company rules on day one. Until each company confirms support, the old rules are the only ones enforced.
• Treating the draft as a finished standard. The draft expires on 3 May 2027 unless a new version replaces it, and its author plans to propose the design as a change to a related Content-Usage draft. Expect details to shift.

Why this matters for small site owners

Big publishers have legal teams and direct lines to the AI companies. Small site owners have a text file. A single shared line, if it truly becomes universal, is the first opt-out mechanism designed to be as easy for a one-person blog as it is for a newsroom. That is why this announcement matters even though nothing works yet: it is the closest the industry has come to agreeing on one language for "please do not train on my work."

It is also a reminder to keep your own house in order. An opt-out is only as good as the file that carries it. If you cannot edit your robots.txt, you cannot express any preference at all — shared token or not. So the most useful thing you can do this week is not to memorise a new token, but to confirm you can open, edit and publish your robots.txt file, and that your current per-company rules are still in place.

Frequently asked questions

Does adding __usage_ai-training to my robots.txt block AI training today?

No. As of November 2026, Google, Microsoft and OpenAI have only announced plans to support it. Google aims for early 2027, Microsoft from 1 February 2027, and OpenAI within the next few months. Keep your existing per-company rules until support is confirmed.

Will the new line stop Google or Bing from indexing my site?

No. The token states a training preference only. It does not control crawling or search indexing, which are handled by ordinary robots.txt rules and meta tags.

What happens if my robots.txt has no __usage_ai-training group?

That signals no preference. Your content is treated the same as every other site without the group — which today means every site, since the token is brand new.

Do I still need Google-Extended, GPTBot and ClaudeBot rules?

Yes, for now. Those are the controls actually enforced today. The shared token is meant to eventually replace that stack, not to sit alongside it forever — but the replacement has not happened yet.

What is the Microsoft NoArchive/NoCache deadline?

Microsoft stops recognising NoArchive and NoCache as training or grounding signals after 1 March 2027. Sites using those tags should set a temporary preference in Bing Webmaster Tools when that setting opens on 15 December 2026.

Can I allow AI grounding but block AI training at Google?

Not under the current setup. Google-Extended covers both training and grounding, and Google's documentation describes no way to separate them. Microsoft, by contrast, keeps training and grounding as separate preferences.

Conclusion

The shared __usage_ai-training line is genuinely good news for website owners — one preference, understood by every major AI provider, instead of a growing pile of company-specific rules. But good news with a future start date is not protection today. Keep your existing opt-outs in place, confirm you can edit your robots.txt, diarise Microsoft's December and March deadlines, and re-check each company's documentation in early 2027 before you trust the new line to do the job alone. That is the whole strategy: prepare now, switch over when support is real, and never assume a file you have not checked is protecting content you care about.