OpenAI backs a shared training signal as publishers seek more control

A publisher may want an AI service to help readers find an article while keeping that article out of the material used to train its next model. Those are two different choices: helping people discover a publication, and allowing its work to become part of a machine’s learning process.

The distinction applies to publishers whether their articles are written by people, produced with AI assistance or assembled through a mixture of the two. The question is how an outside AI company may use the published material.

OpenAI said on October 30 that it would honor a shared training-preference signal in robots.txt, the small file websites use to communicate with automated visitors. The company aims to implement it in the coming months and says its existing crawler controls will continue to apply. The announcement is a commitment to future implementation; website operators can continue using the controls available today. [1]

Today, publishers often write separate instructions for individual companies’ crawlers, the automated programs that visit and collect web pages. A shared signal would let them state a training preference in a common form that participating companies could recognize. The proposed gain is simpler administration across providers, especially for small publications whose editors also manage their websites.

From individual bots to a shared purpose

The proposal comes from Kevin Kelley of Anthropic and draws on the Internet Engineering Task Force’s AI Preferences work. Its current status is an individual Internet-Draft, a work-in-progress submission for discussion. Formal standardization would require further work. [2]

Under the draft, a publisher could state a training preference for the whole website or particular sections, such as an archive of reported articles. That instruction concerns training use. Separate crawler-access rules would still govern which automated visitors are allowed to collect pages, and private agreements could take precedence. [2]

A publication’s reporting, photography, book excerpts and press materials may serve different commercial purposes. Expressing those choices in a consistent format could make them easier for participating companies to interpret and for the publisher to maintain.

The wider standards effort extends beyond this particular proposal. The IETF working group is developing vocabulary and ways to associate preferences with content, including through web protocols and metadata. Its charter assigns technical enforcement and auditing to work outside the group’s scope. [3]

Search visibility is a separate choice

OpenAI already provides distinct controls for training and search. GPTBot collects material that may be used to train its AI models. OAI-SearchBot helps websites appear in ChatGPT’s search features. Its documentation explicitly permits a website to allow the search crawler while declining training use through GPTBot. [4]

Publishers can use those separate controls today. A local news organization could allow OpenAI’s search crawler to find its council coverage while declining collection by its training crawler. The planned shared signal would give participating companies a common way to read the training preference; search visibility would remain a separate setting. An author could make similar choices for a website containing book descriptions and excerpts.

There is also ChatGPT-User, which visits pages in response to certain user requests. OpenAI describes these as user-initiated actions and says robots.txt rules may not apply. Its search crawler remains the relevant control for automatic search discovery. [4]

Publishers therefore need to distinguish three activities: collecting material for training, indexing it for discovery, and retrieving it to answer a person’s immediate question. Each can have a different value to the business and a different technical control.

A crowded field of publishing controls

Cloudflare offers another approach through its Content Signals framework. Its categories distinguish training, conventional search indexing and supplying material to an AI system at answer time. The framework uses its own set of instructions, illustrating how several efforts are approaching the same publishing problem through different implementations. [5]

Maintaining several versions of the same policy creates work for publishers. Broad participation and clear implementation guidance would determine how much maintenance the new approach actually saves.

The preference works like a posted sign: its effect depends on the recipient following it. The established robots exclusion standard treats robots.txt as communication between a website and automated visitors. Enforcing access requires separate measures, such as authentication or server-side controls that determine whether a visitor receives a page. [6]

The economic questions remain

Payment, referral traffic and technical enforcement depend on separate commercial arrangements, product behavior and access-control choices. Those outcomes require their own evidence as providers adopt the signal.

OpenAI’s announcement describes a voluntary effort. On the same day, the company published a defense of fair use in AI training, presenting arguments supporting its position in copyright litigation. The company is accommodating publisher preferences while continuing to contest the legal boundaries of training. [1,7]

Publishers can judge the change by the work it saves and the behavior they observe. Server records can show which crawlers visit; analytics can help identify services that send engaged readers. Those records document visits and referrals. Assessing downstream training use requires additional evidence about a provider’s practices.

The next milestones are OpenAI’s implementation details, adoption by other providers and progress through the standards process. The benefits will depend on how many companies adopt the instruction, how consistently they follow it and what evidence they provide about compliance.