AI giants back single robots.txt signal to opt out of AI training
OpenAI, Google, Microsoft and Anthropic plan to honor a new robots.txt signal. It lets website owners refuse the use of their content for AI training with a single entry.
Symbolic image · Photo: Arzhel Younsi / Wikimedia Commons, CC BY-SA 4.0 (cropped and resized)
Four of the biggest AI developers plan to support a shared signal that lets website owners object to their content being used to train AI models. The signal is the entry "__usage_ai-training" in robots.txt, the file that has told search engines and other crawlers what to do for decades. Until now, site owners had to block each crawler individually. The proposal comes from an Internet Engineering Task Force draft submitted on Friday by Anthropic's Kevin Kelley.
The rollout will vary. Anthropic says it has already begun implementing support, while OpenAI promises it in the coming months. Google aims to honor the signal in 2027 and will treat it as if a site had blocked its Google-Extended crawler. Microsoft also announced a separate control for its Bing search engine covering the use of pages in live answers.
The entry covers only the training of AI models, not search indexing or real-time retrieval of content. Compliance is voluntary and there is no enforcement mechanism. The draft is not yet an official standard.