OpenAI Will Honor a Single robots.txt Signal for AI Training Preferences
The approach lets website owners tell multiple AI companies at once whether their content may be used for training. OpenAI says it aims to adopt the signal within the next few months, and existing crawler blocking rules will keep working.
OpenAI announced on October 30 (local time) that it will honor a common signal expressing a website’s preferences about AI training. The mechanism lets site owners indicate whether their content may be used to train the foundation models behind generative AI. OpenAI said it aims to adopt the signal within the next few months.
One declaration, many companies
The common signal goes in robots.txt, the file where sites spell out which parts automated crawlers may collect. Once an owner writes the signal into that file, the same intent is communicated to every participating AI company.
Today, owners have to look up each company’s crawler name separately and maintain a separate rule for each one. OpenAI said reducing that burden is the point of a common signal. For publishers and site operators, it changes how crawling rules get managed.
What happens to the robots.txt you already use
Existing blocking rules in robots.txt continue to work as before. OpenAI said its current controls will still be respected. The common signal sits on top of that, adding a way to communicate training-use preferences to multiple companies at once.
The signal is based on work underway in the AI Preferences working group at the IETF, the body that develops international internet standards. It is not a legal requirement but something companies choose to follow, so its practical effect depends on how many of them sign on.
When can site owners use it?
OpenAI said it is targeting adoption within the next few months and will share implementation details later. What owners need to watch for is exactly how the signal should be written. There is still time before the announced policy turns into a setting that sites can actually use.
Ultimately, the announcement is about simplifying the process by which content owners declare their preferences on AI training. The rollout timeline, the implementation details, and the range of companies that join in will determine whether the approach proves genuinely useful.