robots.txt gets a shared AI training opt-out. Finally, one that might work.

On October 30, OpenAI, Google, Microsoft, and Anthropic announced they'll all honor a single shared token in robots.txt: __usage_ai-training. Set Disallow on that group and all four crawlers treat it as a training opt-out.

This matters because the previous situation was absurd. Blocking AI training on your site meant maintaining separate directives for GPTBot, ClaudeBot, Google-Extended, PerplexityBot, and however many others had been documented that week. Some bot names weren't public. Some were honored inconsistently. The industry had fragmented into a dozen proprietary signals with no standard. Publishers were maintaining a list that was always incomplete.

One token. Four companies. That's the change.

Source spread

• Thurrott — Anthropic, Google, Microsoft, and OpenAI agree to let sites opt out of AI training — builder. Covers the token, the signatories, implementation timelines including Microsoft's February 2027 date.
• AI Weekly — OpenAI, Google, Microsoft, Anthropic back one AI opt-out token — builder. First major report on the announcement, cites The Verge.
• ToolNavs — robots.txt gets a shared AI training opt-out — builder. Detail on the IETF draft and Anthropic authorship of the draft.
• Dev.to — Controlling AI crawlers with robots.txt: block training, allow citation (2026) — builder. Code-level walkthrough of the directive syntax and path-level exceptions.

Pros & cons

What's actually useful:

• One directive, four crawlers. Maintaining a list of AI bot identifiers is ongoing maintenance work. Maintaining one semantic signal is not. That's the whole value proposition.
• Training is separated from search indexing. Setting Disallow: / under __usage_ai-training doesn't pull your site from Google Search. The signals are independent by design. That's a real improvement over the previous situation where blocking a training crawler could create ambiguous behavior.
• Path-level exceptions work. You can Disallow the whole site but Allow /press-kit/ if you want that content available for training. Granular control without per-bot juggling.
• Anthropic's Kevin Kelley, technical lead for standards, led the IETF draft behind this (draft-ietf-aipref-vocab-08). Anthropic has a real investment in this standardizing.

What deserves a side-eye:

• The signal is advisory. A crawler that ignores it pays no penalty. This is a commitment from the companies, not a technical constraint.
• OpenAI, Google, and Microsoft have 2027 implementation timelines. Anthropic is the only one live today. If you're publishing now, three of the four biggest crawlers aren't honoring this signal yet.
• The IETF draft expires in May 2027. Standards process can rename or restructure the token before it finalizes. Low probability but worth knowing if you're building tooling around it.

What builders need to know

• Add the token now, even before full rollout. Anthropic is already live, which means ClaudeBot honors it today. When OpenAI and Google implement, you won't need to update anything.

User-agent: __usage_ai-training
Disallow: /

• Allow specific paths if you want them trainable. If /press-kit/, /case-studies/, or other content is fine to include, add Allow: /press-kit/ above the Disallow within the same group.
• This does not affect search. Setting Disallow under __usage_ai-training has no effect on Googlebot, Bingbot, or any search crawler. The semantic scopes are separate by design.
• Keep your existing bot-specific directives for now. Until OpenAI and Google ship, GPTBot and Google-Extended are still the signals those crawlers check. Run old and new in parallel through 2027.
• Watch the IETF draft. Current version: draft-ietf-aipref-vocab-08. Expires May 2027. If the token name changes before ratification, you'll want to update. Low risk, but set a reminder.

Further reading

• Thurrott — Anthropic, Google, Microsoft, and OpenAI agree to let sites opt out of AI training
• AI Weekly — OpenAI, Google, Microsoft, Anthropic back one AI opt-out token
• Dev.to — Controlling AI crawlers with robots.txt: block training, allow citation (2026)
• ToolNavs — robots.txt gets a shared AI training opt-out, Google, Microsoft and OpenAI sign on
• IETF aipref working group documents — source for draft-ietf-aipref-vocab-08