Introduction
OpenAI announced on October 30 that it intends to honor a shared robots.txt signal for generative AI training preferences. Website operators could communicate a choice to participating providers without maintaining separate versions.
OpenAI expects to add support in the coming months. Its existing crawler controls remain available during that transition.
What the proposal covers
The linked technical proposal describes a purpose-specific group in robots.txt that expresses a training preference for selected paths. It keeps the file’s existing rule structure and separates content-use preferences from instructions about crawling.
That proposal was published on October 30 as an individual Internet-Draft. The IETF record explicitly identifies it as unendorsed work with no formal standing in the standards process. It should be described as a proposal, rather than a finished web standard.
Search and training are separate decisions
OpenAI’s current documentation already distinguishes its search crawler, OAI-SearchBot, from GPTBot, which collects material that may be used for training. It says a website can allow the search crawler while disallowing GPTBot. Allowing search access does not guarantee that a particular page will appear in an answer.
For a business publishing guides, product information or a portfolio, this distinction is useful: being discoverable and expressing a training preference do not have to be treated as the same decision.
What we would review with a website owner
Our take at araNET: start with the content and the owner’s intentions. Which pages should people find? What training preference does the owner want to express? Which material requires restricted access?
Then review the current crawler rules with the person maintaining the site. Record the agreed choices and the providers they cover before changing anything. When support for the shared signal arrives, compare its documented behavior with those choices rather than replacing existing rules on the strength of an announcement.
Private documents need their own access controls. The Robots Exclusion Protocol is not an authentication mechanism, and listing a path in the file makes that path public.
Availability and limits
OpenAI has announced a commitment and a broad implementation window, not a launch date. Participation is voluntary; the announcement does not establish universal adoption.
