A signal that would let a publisher state one AI training preference for many vendors at once is getting a high-profile backer: OpenAI said on October 30 that it will honor it.
The effort is voluntary, OpenAI said, and it plans to implement the signal in the coming months, sharing the technical specifics at that point. The robots.txt controls it already offers stay in force and remain independent of the new mechanism.
The mechanism lives in a draft from Anthropic’s Kevin Kelley, dated October 30 and expiring May 3, 2027. It adds a policy token that works inside ordinary robots.txt groups.
A site writes the token in the User-Agent line and binds it to paths with the familiar allow and disallow directives. Precedence follows longest match, the same rules published as RFC 9309, and no new directive names or grammar arrive. Parsers that do not recognize the token skip the group the way they would skip any unknown agent, so nothing breaks. Wildcard groups never match the token, and a file that omits it states no preference. Usage preferences are not crawling instructions, and private agreements can override them.
The token’s meaning tracks a companion vocabulary draft from Paul Keller of Open Future and Martin Thomson of Mozilla, still short of working group consensus. It defines AI Training as modifying the learned parameters of a generative model, and where statements conflict the most restrictive preference applies.
Backers listed in the acknowledgments include Google, Microsoft, Meta, Cloudflare, Common Crawl, News Corp, the Financial Times, the BBC, the Electronic Frontier Foundation and Creative Commons, plus working group chairs from Cloudflare and Cisco.
No formal standing attaches to an individual draft, the roadmap notes.
