OpenAI Plans New robots.txt AI Training Controls: What Changes for SEO

Soumya Parmaj Avatar
✨ Summarise and Analyse the Article

OpenAI robots.txt changes are coming. OpenAI plans to support a new shared robots.txt AI training signal that would let website owners express how their content may be used to train generative AI models. Announced on October 30, 2026, the change could simplify AI crawler governance, but it is not active yet. OpenAI says support is planned for the coming months.

For SEO teams, the important question is not whether to block all OpenAI bots. It is how to distinguish AI model training from discovery in ChatGPT search, because those are different uses of website content.

OpenAI robots.txt changes: what was announced?

In its official announcement, OpenAI said it would support a common signal in robots.txt for website owners’ preferences on use of content for foundational generative AI training. Today, publishers often manage different crawler names and policies across AI providers. A shared mechanism could make those preferences easier to communicate to participating services.

These OpenAI robots.txt changes concern a proposed content-use preference, not an announcement that ordinary robots.txt rules have been replaced. It does not automatically govern search indexing, ChatGPT citations, or every other company’s crawler. The exact implementation and provider adoption still matter.

How the proposed AI training signal works

The Internet Engineering Task Force draft discussed in connection with the proposal defines a policy token, __usage_ai-training. Its format resembles a familiar robots.txt rule:

User-agent: __usage_ai-training
Disallow: /

In the proposal, this expresses a sitewide preference against the specified AI training use. The token is not a real crawler user-agent, and the line should not be described as a way to stop all AI bots from requesting pages. Because implementation is still pending, website owners should not rely on this rule alone as an effective OpenAI training opt-out today.

GPTBot vs OAI-SearchBot: what stays the same

OpenAI already documents separate crawlers for different purposes. For a B2B SaaS company, that distinction can affect whether its product comparisons, documentation and help pages are discoverable in ChatGPT search while its training preferences are respected.

CrawlerPurposeSEO implication
GPTBotCollects content that may be used to train OpenAI’s generative AI foundation modelsRestricting it signals an opt-out from that training collection
OAI-SearchBotCrawls websites to surface pages in ChatGPT searchBlocking it can limit appearance as a source in ChatGPT search; navigational links may still appear
ChatGPT-UserFetches content for certain user-initiated ChatGPT actionsIt is not the same as an automated search or training crawler; robots.txt behavior can differ for user-triggered retrieval

This is why an indiscriminate block on OpenAI-related traffic may be counterproductive. An organization might want to decline training while keeping publicly available information discoverable. But allowing OAI-SearchBot does not guarantee that ChatGPT will cite a page. Our guide to ranking and being cited in ChatGPT covers the wider visibility factors.

For broader context, see OneMetrik’s AI search SEO guide, which covers visibility beyond crawler access.

What should SEO teams change in robots.txt right now?

The new shared signal is not yet an operational replacement for OpenAI’s existing crawler controls. Teams can review the current configuration instead of making a rushed change based on a proposed standard.

For example, a business that wants to block GPTBot and allow OAI-SearchBot may use the following rules, depending on its wider robots.txt setup:

User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /

These directives communicate different preferences to OpenAI’s crawlers. They do not override more restrictive network-level controls. A web application firewall, CDN bot filter, IP block or server rule can still prevent permitted bots from reaching the site. The exact result also depends on other rules in the robots.txt file. For a full testing and deployment workflow, see our guide to robots.txt and AI crawler controls.

  1. Inspect current robots.txt rules. Check GPTBot, OAI-SearchBot and broad blocks that might unintentionally restrict important content.
  2. Define separate business goals. Make an explicit decision about model training use and a separate decision about ChatGPT search discovery.
  3. Check access outside robots.txt. Review CDN, WAF and hosting controls as well as crawler requests in server logs.
  4. Protect private data properly. Use authentication and server-side permissions; robots.txt is not a security barrier.
  5. Watch for rollout details. Implement the shared signal only after provider documentation clarifies support and interactions with existing rules.

SEO teams working on discovery in Google’s and other AI-powered search experiences should also keep platform-specific crawler rules separate. OneMetrik’s Google AI Mode SEO guide covers the broader search implications.

What remains unclear about the rollout

OpenAI has not specified an exact implementation date. Website owners also need more detail on how potential conflicts between the shared training preference and provider-specific rules will be resolved, which providers will participate, and how compliance can be verified. The October 30 announcement does not establish that older training data will be removed retrospectively.

OneMetrik takeaway

The most important takeaway from OpenAI’s proposed robots.txt changes is that AI training permissions and AI search visibility are separate choices. A publisher may reasonably decline training use while still wanting its pages accessible for search discovery. A SaaS company may make the same distinction for product documentation and comparisons.

Audit existing controls now, avoid blocking useful discovery accidentally, and wait for the official rollout before treating the new shared token as an enforceable setting. In practical terms, saying no to AI training should not require saying no to potential customers.

Frequently asked questions

Does blocking GPTBot affect ChatGPT search visibility?

Not necessarily. GPTBot relates to content collection for training OpenAI’s generative AI foundation models. OAI-SearchBot is a separate crawler used for ChatGPT search. Website owners can restrict GPTBot while allowing OAI-SearchBot, although being crawlable does not guarantee citations.

Are OpenAI’s proposed robots.txt AI training changes live?

No. OpenAI announced plans on October 30, 2026, to support a shared AI training preference signal in the coming months. Until the company confirms implementation, use existing documented crawler-specific controls rather than relying on the proposed token.

Can robots.txt prevent AI systems from accessing private content?

No. Robots.txt communicates preferences to compliant crawlers and is not an access-control or security mechanism. Protect confidential pages with authentication and appropriate server-side restrictions.