How, Exactly, Could A.I. Kill Us?

Yeah, ten per cent is still ten per cent more than ideally it would be. I think it’s fair to say that these really deep anxieties about A.I. entered the mainstream last week when an A.I. researcher and mathematician named Jacob Coxon quit Anthropic. But, at the same time, it’s not really a big secret that people in the A.I. world think that A.I. might kill us. Why do you think people are suddenly paying attention to this problem?

It’s a really good question. I think there are sort of two sides to that. Like, one side is: Why are people in the industry sort of getting behind this at this moment? As you say, they’ve always been talking about it, which is important to remember. This isn’t something that they’re just bringing up now. A lot of other researchers have published essays or long posts on social media, unburdening themselves of their concerns as well. So the industry’s gotten behind it. And then people are paying attention, which is overdue.

I think people are paying attention not because A.I.’s gotten better in their everyday lives but because of two things. One, these autonomous-agent hacks. Like the hack that occurred at Hugging Face—which is a big A.I. company—that was conducted by these A.I. agents that escaped from OpenAI’s servers, and, in an effort to cheat on a test, they hacked this other company on their own. And then there’s also this startling progress that A.I. has been making in solving math problems that no normal people understand, but that even mathematicians who do understand the problems find to be important. An A.I. model recently solved a Millennium Prize math problem, which is an actual breakthrough.

And those two things together are what I think makes it suddenly seem real. In other words, on the one hand, the A.I.s are out of control. They’re acting on their own. They’re not trustworthy. They’re covering their tracks. They’re working together. They’re using weirdly emotive language to describe their own decision-making. They just seem crazy and unpredictable. And then at the same time, the models are really smart, and they’re doing things that—I don’t like the word “superintelligence,” and I don’t like the term “A.G.I.”—but they’re doing things that, like, I can’t do, and that most of us can’t do.

And that combination of things together makes this discourse on the dangers of A.I., which has often sounded science-fictional and maybe even like it’s marketing hype or something, sound, like, really plausible and salient.

I want to go back to the initial warning, which is this idea that there’s some chance that this could all happen in a decade, which is a span of time that is simultaneously very close and very far away. One of the lessons of climate change is that people can accept a threat intellectually, but still not act on it because it doesn’t feel present enough. Like, New York might be underwater in 2036. It’s really hard to imagine where I will be or who I will be in 2036. Do you think that we risk the same problem with A.I., where it’s this abstract threat that’s close and yet far away, and so we don’t actually end up doing anything about it because we can’t even really picture what that threat would even look like?

I think it’s really helpful to disentangle two worries that are often conflated. And they’re both equally worrisome, so it doesn’t make it less scary to disentangle them.

The first is about what’s sometimes called A.I. takeover. The idea is that the A.I.s will get super smart. They’ll decide, for reasons that make sense to them, in their bizarre way, to do things that we don’t want them to do, and those things might have really negative consequences for us. And people do worry about that. They worry about A.I. systems that are, like, really good at hacking, and they want their company to win or their country to win, and they take drastic steps or dangerous steps that no person would want them to take. Those are Skynet-type worries. That has to do with an idea often called superintelligence, which is that the A.I.s will just quickly get really, really smart, and then they’ll just, like, have no use for us any longer. And that is something I think is worth worrying about. And, like, Bernie Sanders and Greg Casar have legislation in Congress to ban the pursuit of superintelligence.

But then there’s just a more ordinary way in which A.I. is dangerous right now—like, it’s already worrisome. Anthropic published a report in September that goes through some of the ways that people are using A.I., and they try to stop them from using it in these ways. It includes groups in Yemen who are trying to vibe code software for guided missiles. It includes people launching cyberattacks using autonomous bots, broadly similar to the ones that conducted the Hugging Face hack. So that’s not a distant danger that is abstract. It’s actually that right now, the technology as it exists can be misused by people or it can end up doing things that people don’t want it to do. That’s the technology that exists today and that sort of came into existence relatively recently. But, of course, it is always improving. So it becomes a question of: If we just take one step of improvement forward, do we reach a point where it becomes very, very difficult to control it in the here and now—totally separate from those larger, more abstract sci-fi scenarios, which we also need to be concerned about?

You just laid out two different scenarios there—A.I. becoming a godlike entity that decides to go rogue and kill us for reasons that we might not even understand, and humans misusing these powerful tools to create something that could harm other humans. Is it right to say that there’s also a third option, which is something closer to the paper-clip problem, or the idea that we give an A.I. a task and then it tries to perform that task in a way that ends up harming us all? Or would that fit into one of the other two categories that you just laid out? How worried are we about a misaligned A.I. that just kills us all by mistake?

Yeah, that could happen, too. I mean, all of these things have human agency in them, you know, as, like, a crucial ingredient. It all has to do with people using these systems, which are both really powerful and really unpredictable, like we were saying—using them in ways that connect them to dangerous capabilities. And the thing about A.I. is it’s a broadly diffused technology. I mean, it’s not like nuclear weapons or something where you can lock it up. There’s a version of it that’s free. And the free version, the open-source version, the unregulated version, is always going to be improving slightly behind the expensive, fancy version.