Cloudflare CEO says bot traffic online will exceed human traffic by 2027

Bots are taking over the web, according to Cloudflare CEO Matthew Prince. In an interview at the SXSW conference in Austin this week, he said that with the speed at which AI is growing, AI bot traffic will exceed the volume of human Internet traffic by 2027.

Prince explained that the web’s use of bots is growing alongside the growth of generative AI technology because bots are able to visit more sites to get answers to users’ chatbot queries.

“If a human is doing a task — let’s say you were shopping for a digital camera — and you might go to five websites. Your agent or bot that’s doing that will often go to 1,000 times as many websites as an actual human,” Prince said. “So it might go to 5,000 sites. That’s the real traffic, that’s the real load, that everyone has to deal with and take into account.”

Before the era of generative AI, the Internet accounted for only about 20% of bot traffic, with Google’s web crawler being the largest, according to Prince, whose infrastructure and security company uses a fifth of all websites. But unlike some other reputable crawlers, the only other bots are those used by scammers and bad actors.

“With the advent of generative AI, and its insatiable need for data, we are seeing a rise where we believe that in 2027, the amount of bot traffic online will exceed the amount of human online traffic,” Prince said.

The CEO also noted that this change in the web will require the development of new technologies, such as sandboxes for AI agents that can be quickly turned on and then demolished when their mission is done. This could come into play when consumers ask AI agents to perform certain tasks on their behalf, such as planning a vacation.

“What we’re trying to think about is, how do we actually build that underlying infrastructure where you can — as easily as you open a new tab in your browser — actually spin up new code, which can then run and serve the agents that are there,” Prince said.

He imagines that there will soon come a time when millions of such “sandboxes” are created for customers every second.

Of course, bots using the Internet on this scale requires physical infrastructure in the form of data centers and servers. Prince noted that during Covid, internet traffic increased so rapidly, especially among video players like YouTube, Disney and Netflix, that some parts of the internet were almost collapsing under the pressure.

“this [growth] “It’s more gradual, but unlike Covid, where it went up over a couple of weeks and then kind of plateaued at a new high, we’re seeing increased growth in internet traffic, and we’re not seeing anything that will slow it down or stop it,” Prince added.

All of these concerns about overloading represent great marketing for Cloudflare, a company that focuses its services on helping websites stay highly available, load quickly, and stay safe from attacks. Among its offerings are a content delivery network, a series of security measures and DDoS protection, and “always online” technology that serves up cached versions of websites when the main server fails or goes offline. It also provides companies with tools to block bot traffic they don’t want.

However, Cloudflare’s size gives it the advantage of being able to see the continuing evolution of the Internet and the rapidly emerging challenges facing the age of generative AI.

“I think the thing that people don’t appreciate about AI is that it’s a platform shift,” Prince said, recalling previous web platform shifts, such as the move from desktop to mobile. “AI is another platform shift…the way you will consume information is completely different.”