Cloudflare CEO Matthew Prince pushes UK regulator to break up Google search and AI crawlers

After launching a marketplace earlier this year that allows websites to charge AI bots for scraping their content, web infrastructure provider Cloudflare is seeking to increase regulation in the AI ​​sector.

The company’s CEO Matthew Prince says he is in London to speak with the UK’s Competition and Markets Authority (CMA), where he is proposing tougher rules on how Google should be able to compete in the AI ​​race, given its dominance in search.

The Capital Markets Authority earlier this month granted Google special status in the search and advertising markets due to its “large and well-established” position. The move will allow the regulator to impose stricter regulations beyond just search and ads, including in areas such as Google AI Overview, AI Mode, Discovery Feed, Top Stories, and News Tab.

According to Prince, Cloudflare is well-positioned to make recommendations because it is not in the AI ​​space itself, but has a large number of relationships with AI companies themselves.

“We don’t have a dog directly involved in combat,” Prince said, speaking at the Bloomberg Technology Conference in London this week. “We are not an artificial intelligence company.” He added: “We are not media publishers, but we are this network between them. 80% of AI companies are our customers.”

Cloudflare’s president believes that Google should compete on equal terms with other AI companies, which it does not do today. Instead, Google uses its existing web crawler to crawl the content of its AI products and services, in addition to its search engine. This gives Google an unfair advantage, Prince said. (Prince is referring to Googlebot, which crawls search, including its own AI features, such as AI Overviews. Google spokesman Ned Adriance said site owners can opt out of using content to train AI products with Google Extended, which does not affect a site’s inclusion in Google Search. It is possible that some media companies would prefer to completely opt out of AI features, however, the point largely stands.)

“Google is saying: We have the absolute, God-given right to all the content in the world, even if we don’t pay for it, because look at what we’ve done over the past 27 years,” Prince explained. “And they say we can take it and use the same crawler that we use for research to run our AI systems. And if you want to opt out of one, you have to opt out of both.”

This, obviously, is not possible for most people, especially those in the media industry, where losing search means losing about 20% of your revenue, the executive said.

“But it gets worse. If you block Google’s crawler, it blocks their ad safety team, which means your ads across all your platforms stop working, which is just unfortunate,” Prince added.

Because Google compiles its own crawler, it is able to access content that others, like Anthropic, OpenAI, and Perplexity, would have to pay for.

“The problem is that we then effectively hand the game over to Google,” he said.

The solution, Prince said, is to foster a lot of competition in the market, where thousands of AI companies could compete to buy content from thousands of media companies and millions of small businesses. He noted that what the UK CMA was doing by marking Google as a potential regulatory target was a deliberate move, and indicated that they recognized the unique advantage that Google had.

Cloudflare also provided the CMA with data explaining how Google’s crawler works and why it’s nearly impossible for other players to replicate the same success that Google can achieve.

The prince is not the only one who has shared these views in recent days. Last month, Neil Vogel, CEO of People, Inc., the largest U.S. digital and print publisher, which operates more than 40 media brands, said essentially the same thing. In an interview, he called Google a “bad actor,” saying media companies had no choice but to allow Google to crawl their sites for AI content because of the way the crawlers were integrated.

Vogel, whose company adopted Cloudflare’s solution to block non-paying AI crawlers, claimed the system was working, saying deal discussions were underway with several major LLM providers.

Updated using Google comment.