Everything is about to “go dark” – A Few Thoughts on Cryptographic Engineering

I’m coming down from spending a few days at Usenix Security, right here in Baltimore. This means that my days have been taken up with two kinds of conversation: first, explaining to colleagues why Baltimore isn’t actually like The Wire. And second, trying not to talk about AI.

Here I’m going to break both of those rules.

I have many worries about what AI means for our field, for various definitions of “field”. But in this post I want to focus on just one thing I’ve started worrying about, and it’s a perverse thing: specifically, I’m concerned that AI is going to make software much too secure.

While that doesn’t sound so bad on the surface, there’s a consequence to this. I mean something very specific: I’m concerned that U.S. intelligence and law enforcement agencies are about to go dark, meaning: that they’re going to suddenly lose a huge portion of their capability. And that this isn’t going to be simply a problem for those agencies, but also for those of us who value computer security and privacy in general.

Going Dark, and the era of law enforcement hacking

To explain how we got here, we need to talk about recent history. This gives me a legitimate excuse to reference The Wire, just because it embeds a realistic snapshot of what electronic surveillance looked like way back in 2002. If you’ve seen the show, you’ll recall that the cops are trying to spy on drug dealers who use payphones and burners, and primarily use them for voice calls. The mobile phones in the show are relatively new — but from a technological perspective — nothing in this scenario would have shocked a cop who jumped forward from, say, 1989.

In less than a decade from the premier of that show, everything was entirely different.

The change began in the late 2000s, thanks to the rise of smartphones and texting. Because smartphones can actually store data as well as conveying it, those phones became a new source of law-enforcement capability. Coincidentally, around 2010 Apple began encrypting iPhone data using a key derived from the user’s passcode (with Android phones following shortly thereafter.) The following year, Apple deployed end-to-end encryption in iPhone text messages. By 2014, a texting startup named WhatsApp had gathered 600 million users worldwide. By 2016 those users, now nearly a billion, were all using default end-to-end encrypted messaging. The chart below gives one view of how quickly that change took place:

The FBI and law enforcement agencies were not insensitive to what was happening. In 2014, Director Comey announced an initiative called Going Dark, which would launch a “national conversation” about what providers could do — or be compelled to do — to make these new communications media legible to law enforcement and counterintelligence.

In 2016, the agency stopped merely talking about this. After a terrorist attack left the FBI holding a shooter’s locked iPhone, the agency ordered Apple to give them access. The company refused. What broke the stalemate — and, to some extent, ended “Going Dark” itself — was something that neither the FBI nor Apple expected. An outside company announced that there was no need for Apple’s assistance: they could simply hack the phone.

The Apple v. FBI case turned out to be microcosm of the whole debate. For the next decade, law enforcement and intelligence agencies continued to ask for “exceptional access” backdoors. But now the urgency was gone: agencies and manufacturers knew that law enforcement could purchase targeted hacking tools like GrayKey (for phone unlocking) or even remote exploitation tools like NSO Group’s Pegasus, assuming they needed them badly enough. Vendors like Apple and Google played a vigorous defense, closing vulnerabilities as soon as they learned about them. But this wasn’t enough. Commercial offensive vulnerability hunters consistently managed to keep the edge.

And now all that feels like it’s about to be history.

The era of AI bug hunting is here

In April, Anthropic announced a new model called Mythos that was optimized for software vulnerability finding. The U.S. government temporarily blocked its export, restricting it to U.S. agencies. While the ban was dramatic and made for good PR, it was mostly pointless. OpenAI, along with Chinese open-weight model labs like Z.ai and Moonshot, have since demonstrated that vulnerability finding isn’t anything that a single model can hold a monopoly on. The list of serious vulnerabilities that these models have found is getting scarier (or more impressive) by the day.

Initially this might seems like good news for the offense, and for hackers in general. But I doubt it will last. Defenders are now in the process of patching every bug they can find, often with AI helping them. Entire development toolchains are being rebuilt to incorporate powerful vulnerability scanning before software reaches the testing phase. This does not mean that every bug will be found: even calculating the number of bugs in a piece of code is probably uncomputable. In the real world, it does feel likely that we’re going to hit some sort of a ceiling on the number of useful bugs, and probably we’ll hit it soon.

Thus: over the next two years, major pieces of software are likely to run out of remotely-exploitable bugs.

While I think this is great, for law enforcement and offensive intelligence agencies, it’s going to be a nightmare. For the first time since 2010, law enforcement might experience what it looks like to really “go dark”, across a huge category of advanced (well-maintained) devices and pieces of software.

So how is this a problem?

The debate over “exceptional access” mechanisms never really went away. In some places, like the UK, it actually metastasized into something worse. Here in the US it mostly went into hibernation. Some of the slowdown can legitimately be attributed to expert pushback — academics and industry engineers pointing out the risk that backdoors might be abused by the very adversaries they’re designed to protect against. But I fear that the market was just pricing supply.

The destruction of the low-hanging vulnerability fruit will make law enforcement (and intelligence) agencies’ need much more acute. The demand for constructed, intentional backdoors will begin in earnest. There will be enormous pressure on industry to re-architect their systems to make their systems friendly to exceptional access. In some cases, governments will ask for these capabilities in the expectation that they’ll be useful for spying on other governments — a strategy that might have been undetectable in the pre-AI era, but that probably will be detectable now. This might result in other governments curtailing their dependence on US software.

In fact, the worst part about this dynamic is that these potential new backdoors will begin primarily useful for allowing the US to weaken its own systems, which will in turn allow foreign adversaries to find new ways to attack our communications. This deliberate self-sabotage will happen just at a moment when we’re finally learning how to defend our own infrastructure.

So what do we do about it?

I honestly have no idea. This is not a call to action for experts to rally behind a sophisticated plan. Like so many things about the AI revolution, it’s just occurring to me that we’re on a long greasy slide to a place that will look different than where we are today. Just realizing this doesn’t mean that I have a clever plan to avoid it. In this case, we’re just going to have to hope that this time we make the right choices, for no other reason than that they’re right.