A fallen power line reveals a growing problem in an AI data center. Here’s how to fix it.

A power line went down outside Washington, D.C., this week. Typically, the network will only need a few seconds to recover from such an event. But this took more than 10 minutes because more than 3 gigawatts of data centers stopped drawing power almost simultaneously.

The event caused a voltage spike across the PJM network from northern Virginia to Chicago, according to data collected by Ting Labs, a startup that runs an Internet of Things sensor network of people’s electrical sockets.

This event did not cause a power outage, but it did cause lights to flicker throughout the area. The incident demonstrated the impact data centers can have on the network, an outcome that experts believe will become more frequent.

Northern Virginia, within PJM’s territory, is home to the highest concentration of data centers in the world.

“It’s the canary in the coal mine,” Ricardo de Azevedo, CTO at ON.Energy, told TechCrunch. He added that this type of event involving large volumes such as data centers is “happening more and more.”

This event is similar to what happened two years ago, also on the PJM network, and could herald larger events if data centers are not built to handle power supply outages more elegantly. PJM Interconnection operates networks from New Jersey to Illinois and serves 67 million customers, making it the largest network operator in the United States.

When a power line went down this week, it prompted data centers to switch to backup power, and about 3.1 gigawatts of load disappeared in about 30 seconds, according to PJM data. The network appeared to recover somewhat, but after a short time the additional loads dropped. At its peak, the PJM grid had an additional 3.49 gigawatts of electricity. It took another 11 minutes before it stabilized. Discrete data centers represented about 3% of PJM’s total demand at the time, according to Reuters.

A small percentage may not seem like a lot, but the electrical grid needs to operate in a state of near-perfect balance, with supply and demand closely matched. If not, voltages can drop or rise. The network and the devices connected to it can tolerate small fluctuations, but if these fluctuations become too large, they will trigger security operations within the network or within individual facilities, causing them to go out.

When data centers in Northern Virginia sensed fluctuations caused by a power line failure, they turned to backup power, removing their loads from the grid. As more data centers switched, they removed greater amounts of load from the network. What started as a relatively small drop in supply turned into a larger drop in demand, sending supply soaring and causing light bulbs to flicker.

Most data centers make decisions in a split second, and the ones that went offline this week appear to be no different. When the voltage drop reached them, they all decided to disconnect within a few seconds of each other, Ali Zain Banatwala, senior market modeling specialist at the Independent Electricity System Operator, told TechCrunch.

“We need to find a way for these loads that are next to each other to either disconnect or reconnect serially,” he said. A more structured process would allow network operators to develop more robust procedures in advance.

Alternatively, data centers can be built to absorb disruption and not turn their backs on it. One startup, ON.Energy, is working on a product to help data centers — and the grid — survive events like the one this week.

The company has developed an uninterruptible power supply for the entire data center campus, covering not only servers but also chillers and other equipment. The company essentially hides the data center behind a bank of batteries connected to sophisticated power conversion equipment. All the network “sees” is one consistent, well-behaved load instead of the peaks and valleys from each individual part of the data center. ON.Energy allows data centers to ramp up and down compute workloads, including AI training, without disturbing the network.

Perhaps most importantly, this also means that data centers can absorb power fluctuations from the grid. Instead of disconnecting from the grid, the ON.Energy system can use any additional power to charge its batteries, and if the flow drops, the system can send power to the servers. Additionally, it can track the network’s progress within milliseconds, preventing sudden sags or spikes like the one that caused PJM’s problem this week.

ON.Energy is currently installing a total of 3 gigawatts of its systems in four different data centers, De Azevedo said.

Network administrators have also taken notice of the problem.

The ERCOT system, for example, would require loads as large as data centers to “survive” the disruptions, de Azevedo said.

But the clock is ticking. This week’s mass outage was twice as large as a similar event in 2024, when 60 data centers were taken offline simultaneously, pulling 1.5 gigawatts of load from the grid. At the time, data centers accounted for about 6% of PJM’s load, according to Synapse Energy Economics. By 2040, they are expected to constitute 24%. If the problem is not addressed soon, things could get much worse.