OpenAI, Google, Microsoft, and Anthropic say they will adopt a new standard that publishers can use to opt out of being scraped by AI across multiple crawlers (Emma Roth/The Verge)

AI Giants Agree to Let Publishers Opt Out of Scraping: What the New Multi-Crawler Standard Means for the Future of AI

By · Published October 31, 2026 · Updated October 31, 2026

On October 30, 2026, four of the most powerful companies in artificial intelligence, OpenAI, Google, Microsoft, and Anthropic, said they will adopt a new standard that lets publishers opt out of being scraped by AI across multiple crawlers. It’s a quiet announcement with loud implications. For years, publishers have watched AI systems gobble up their articles, images, and data to train models that then compete with them for attention. Now, the biggest AI developers are agreeing to give content creators a single, unified way to say “no”, or “yes, but only on my terms.”

This isn’t just a technical tweak. It’s a signal that the wild-west era of AI training data is starting to close. The future of AI depends on data, but that data has to come from somewhere, and the people who make it want a say. Here’s what this new standard could mean for publishers, businesses, and anyone who uses AI.

Why publishers have been frustrated with AI scraping

To train a large language model, developers feed it enormous amounts of text from the web. That text includes news articles, blog posts, product reviews, forum discussions, and more. For AI companies, it’s free raw material. For publishers, it’s their life’s work being used without permission or payment.

Until now, the main tool for blocking web crawlers has been robots.txt, a simple file that tells bots which parts of a site they can visit. But robots.txt has big limits. It works one crawler at a time, and it relies on voluntary compliance. There’s no enforcement. If an AI company ignores it, the publisher has little recourse besides a lawsuit. And with dozens of AI crawlers from different companies, managing opt-outs is a nightmare of separate files and inconsistent rules.

The result has been a tense standoff. Publishers want compensation and control. AI companies want data. Lawsuits have piled up, and regulators are paying attention. A unified opt-out standard could break the logjam by making the publisher’s wishes clear, machine-readable, and portable across multiple AI systems.

What the new standard actually does

The standard gives publishers a way to signal their preferences once, and have those preferences respected by multiple AI crawlers. Think of it like a universal “do not enter” sign for AI training, instead of a different sign for every company. The details are still being worked out, but the core idea is simple: a publisher can say whether their content can be used for AI training, and under what conditions.

Because OpenAI, Google, Microsoft, and Anthropic are backing it, the standard covers the crawlers behind some of the world’s most widely used AI models. That’s a huge shift. These four companies don’t just build chatbots; they run the search engines, cloud platforms, and developer tools that power much of the internet. When they agree to honor a common opt-out signal, they create a de facto rulebook for the industry.

It’s important to note what the standard is not. It’s not a law. It doesn’t automatically mean publishers get paid. It doesn’t force every AI company to join, only the ones that choose to adopt it. But it does create a clear, public commitment that publishers can point to. And it gives well-behaved AI companies a way to show they respect content rights, which matters for public trust and future regulation.

The big winners: publishers and content owners

For publishers, this is a major step forward. A single standard means less time spent chasing down individual crawlers and more time focused on content. It also strengthens their bargaining position. If a publisher can easily opt out, then AI companies that want high-quality data have to negotiate. That could lead to more licensing deals, subscription partnerships, and revenue-sharing agreements.

Imagine a news organization that wants to let AI companies use its archives for training, but only if they pay a fee or attribute the source. A standard opt-out mechanism makes that kind of nuance possible. It turns a binary “scrape or don’t scrape” into a negotiation. Over time, a market for licensed AI training data could emerge, with publishers setting their own prices and terms.

Smaller publishers benefit too. They don’t have legal teams to fight AI giants. A simple, standard opt-out gives them a seat at the table without a courtroom battle.

What it means for AI companies

For OpenAI, Google, Microsoft, and Anthropic, adopting the standard is a smart move. It reduces legal risk. Lawsuits over copyright infringement are expensive and distracting. By giving publishers a clear opt-out, these companies can argue that they respect consent and only use content that’s available. That’s a strong defense in court and in the court of public opinion.

It also improves the quality of training data. If publishers can set terms, they’re more likely to share high-quality content, and less likely to block everything out of frustration. A cooperative system could produce cleaner, more reliable datasets than a chaotic free-for-all.

But there’s a catch. AI companies rely on scale. If too many publishers opt out, models could become less accurate or less diverse. The companies will need to balance respect for opt-outs with the need for data. That tension won’t disappear overnight. The standard is a tool, not a magic fix.

The future of the web: a two-tier internet?

One possible outcome is a more fragmented web. On one side, you have content that’s open to AI training, maybe because it’s from sites that want exposure or don’t mind. On the other side, you have protected content that requires a license. This isn’t necessarily bad. It mirrors how other media work: some content is free, some is paid, some is licensed.

But it could also deepen inequality. Large publishers with legal teams and negotiating power might get paid. Small bloggers and independent creators might simply opt out and disappear from AI training data, making their voices less represented in future models. That’s a risk worth watching. A standard that’s easy to use and understand could help, but it won’t solve the power imbalance on its own.

Another concern is enforcement. A standard is only as good as the companies that follow it. Bad actors could ignore opt-outs. Verification will be key. Publishers will need tools to check whether AI companies are actually respecting their signals. Third-party audits and transparency reports could become the norm.

Practical implications for businesses

If you run a business that uses AI, whether you’re building a chatbot, analyzing customer feedback, or generating marketing copy, this standard affects you. Here’s what to do:

For society, the bigger question is trust. People want to know that AI systems are built fairly. A standard that gives content creators control is a step toward that trust. It won’t solve every problem, misinformation, bias, and economic disruption are separate battles, but it sets a precedent: AI should be built with consent, not just convenience.

The road ahead

The announcement is just the beginning. The standard needs to be finalized, implemented, and adopted beyond the first four companies. It needs clear technical specifications, easy-to-use tools for publishers, and independent enforcement. It also needs to evolve as AI technology changes, what works for text today might not work for video or interactive content tomorrow.

But the direction is clear. The era of “ask forgiveness, not permission” in AI training is ending. Publishers are no longer willing to be passive suppliers. AI companies are realizing that long-term success depends on sustainable, ethical data practices. A multi-crawler opt-out standard is a bridge between those two worlds.

For anyone building or using AI, the message is simple: respect the opt-out. The future of AI won’t be built by ignoring creators. It’ll be built by working with them.

TLDR: OpenAI, Google, Microsoft, and Anthropic have agreed to adopt a new standard that lets publishers opt out of AI scraping across multiple crawlers. This gives content creators a unified way to control whether their work is used for AI training, shifting power back toward publishers and opening the door to licensing deals. For AI companies, it reduces legal risk and builds trust. For businesses, it means auditing data sources and building compliance into AI systems. The move signals the end of the free-for-all data era and the start of a more consensual, negotiated relationship between AI and content.