The Internet is being rebuilt for machines

Cloud infrastructure has long been designed around people searching, clicking, scrolling and streaming in a consistent and predictable way. AI agents behave differently. They can unleash a flurry of activity, spinning up numerous sub-agents that query hundreds of databases, search documents, and call APIs in seconds, and then disappear almost as soon as they arrive.

Under this premise, Amazon is redesigning a key part of its cloud infrastructure. AWS on Thursday launched the next generation of OpenSearch Serverless, a fully managed search and vector database — essentially a large-scale information storage and retrieval system — designed specifically for agent workloads. AWS says the new system can scale up instantly when agents are running tasks and roll back to zero when it’s idle.

The launch reflects a growing realization across the technology industry: Infrastructure originally designed for a human-driven Internet does not perform well in a world with an increasing number of customers.

While AI agents still represent a relatively small portion of Internet activity, machine-generated traffic is already significant, and is expected to grow. Cloudflare says bots accounted for 31% of all HTTP traffic over the past six months. AI crawlers, search engines and assistants accounted for nearly a quarter of all bot requests during that period.

“Non-human traffic will surpass human traffic sometime in the first half of 2027,” Leigh-Yi Olsen, senior product manager at Cloudflare, told TechCrunch.

At the Google I/O developer conference last week, the company said users will be able to start delegating tasks to AI systems, such as searching for purchases, booking travel, browsing the web, and interacting with apps. But the buck doesn’t stop with consumer-focused AI agents. Companies are increasingly deploying agents internally and for their customers, creating new types of traffic that are automatically generated behind the scenes.

As a result, cloud providers and infrastructure companies have been thinking about how to adapt systems designed for humans to a world of agents that continuously and autonomously retrieve information, invoke tools, and generate machine-to-machine traffic.

This is where AWS’s new OpenSearch Serverless comes into play.

“The timing is straightforward,” Tia White, general manager of Amazon OpenSearch Service, told TechCrunch. “Agents are moving from experimentation to production, and they are creating traffic patterns that previous infrastructure simply wasn’t designed for.” “It goes up without warning, it stops working without warning, and the organization needs research that continues without paying for empty or idle compute.”

The main technical change in this new generation is that it decouples compute from storage, allowing compute to be scaled up in seconds to accommodate agents’ traffic flows and reduced to zero, so customers pay $0 when agents are idle.

“Previously, even in our previous serverless version, you had to have at least one instance up and running because storage and compute are coupled,” White said. “You can’t auto-rotate [compute] At the rate you need it, so you always have idle compute reserved for your workloads, whether you’re using it or not.

Think of it like you’re always paying for a parking space, even when you’re not using it. With the Serverless upgrade from AWS, it’s like paying for a specific parking spot.

At launch, OpenSearch Serverless will be natively integrated with AI development platforms like Vercel and Kiro, so developers can deploy production-ready search and vector backends to agents without managing infrastructure.

This shift is beginning to unfold across the cloud industry. Databricks and Snowflake are repositioning themselves as AI-powered enterprise data memory and retrieval systems. Microsoft has rolled out updates to Azure designed to handle AI agent bursts and memory sharing between agents. Cloudflare, similar to Amazon, last month introduced infrastructure aimed at giving agents stable environments and instant scalability.

The more companies deploy AI agents, the greater the pressure to redesign infrastructure around machine-generated workloads, which in turn could make agents cheaper and easier to deploy on larger scales.