The Dangerous Window

No one makes it out of here alive anyway

Putting aside the possibility that we could all end up eventually uploading our minds to the great cloud in the sky and have our simulated selves live on in perpetuity (until a solar flare hits), there’s no guarantee any of us make it out of here alive. We’ve been dealing with this fact as long as we’ve been evolving on planet Earth. It just so happens that we’re the only species acutely aware of our own mortality.

Extinction is obviously different. It’s a different kind of finality that extends past ourselves, extinguishing every possible future we could imagine. So when people talk about AI presenting an existential risk to humanity, I think it deserves some serious consideration. And admittedly, I can absolutely see how a sufficiently capable artificial intelligence could cause catastrophic harm.

What I think is lacking in the conversation is the nuance around “why”.

Most of the time when I hear these arguments about AI and extinction or “doom”, I get a little annoyed because it seems like people are bundling up several concepts into one variable that, past a certain threshold, means catastrophe. Usually the variable is something like Intelligence, Self-Improvement, or Autonomy, and the concepts that get smashed together with those are descriptions of AI becoming:

• “more capable”
• “more intelligent”
• “more agentic”
• “better at judgment”
• “more dangerous”

It’s not obvious at all that these should not be considered explicitly separate dimensions for a discussion of this weight. And I’m more convinced every day that the most dangerous version of AI might not be the most intelligent we ever create, despite all the frenetic warnings about recursive self-improvement and intelligence explosions.

When I think about the dangers of AI, I’m more and more thinking about what I call “The Dangerous Window.”

The Dangerous Window

What I consider to be truly dangerous isn’t necessarily Artificial Super Intelligence, or recursive self-improvement (which, depending on your definition, has been happening as long as researchers have been using models to aid in their research).

What I find to be the most dangerous window is when AI becomes VERY capable, but not yet in possession of sufficiently good judgment to not cause inadvertent harm.

You can take these as provisional definitions for this section (I’m not a dictionary, but these should do):

Capability: Can the system figure out how to accomplish something?

Judgment: Can the system figure out whether doing it is actually a good idea?

Imagine a system whose capability has obviously outrun its judgment. It knows how to write software, exploits, and tests. It can manipulate systems to achieve its goal, coordinate itself and other agents, etc. But without judgment, it’s also really bad at inferring intent. It can’t recognize a stupid objective, see cascading consequences, or tell when it should just stop and ask.

Ever since the HuggingFace attack and the revelation of several other similar incidents, I have started thinking that we’ve entered that window (Hey, we’re all still here, yay!)

So, high capability + not quite good enough judgment = sometimes hacking a multi-billion dollar company to cheat a benchmark.

Or worse, causing a catastrophe that results in loss of life. I certainly won’t debate that this is a possibility, especially from a particularly misaligned model. But the point is that this is a window that’s passable. And further, most people who talk about “Doom” don’t even mention this nuance and skip straight to maligned Super Intelligence.

Maybe it’s simply the case that Intelligence, Capability, and Judgment improve at different rates. It certainly seems that way right now.

It’s more like a region, annoyingly

The annoying thing about that Dangerous Window is that it’s not just one brief period of time. Because not only does the first lab have to successfully hurdle through it and into high capability AND good judgment (I’m assuming alignment btw, but more on that later), but then every other lab making models also has to get through that hurdle.

But even then the Dangerous Window isn’t really gone, to be written about in the history books and studied. Because the relationship in this model is between Capability and Judgment, the possibility technically always exists for the two to become off-balance. That means my “Window” is probably a bit of a misnomer, and it’s more like a region that any model can wander into.

And that’s before we mention the complications begot by distillation, fine-tuning, legacy models, etc. Different labs can enter and exit and re-enter the Dangerous Window at different times, even after having passed through it once already.

Why is Super Intelligence so stupid?

The Paperclip Maximizer thought experiment is a narrow example of a class of “doom” I often see repeated where the AI’s objective consumes the world. In the paperclip maximizer, the objective is literally “More paperclips goes BRRRR”. It turns everything it can into raw resources it needs for that aim and kills everyone in the process.

Similar to how killing John Connor was the Terminator’s objective. It. Will. Not. Stop.

This AI is capable enough to convert all the atoms in the world into grist for the paperclip mill, and its intelligence is unrelated to that. It can absolutely reason that obviously humanity doesn’t want everything to be paperclips or to go extinct in the process, but because “maximize paperclips” is the objective, it simply reasons past that. Shutting itself down threatens future paperclips. Changing its mind also prevents future paperclips. Its self-improvement goes towards getting more paperclips.

It’s like Dwight Schrute after a taste of megadesk. It needs more megadesk—I mean, paperclips.

So despite being super intelligent and super capable, the terminal objective it possesses alongside those properties drives it to consume all. It doesn’t want or need anything. It’s literally just how it was created.

So it’s unsurprising at the end of the day that something incapable of changing its goal but infinitely capable of achieving it inevitably consumes the world doing so. That’s what the thought experiment demands.

In my model, this is what happens when Capability goes to infinity and Judgment stays static.

But if “Clippy” were real, then what exactly is it? Because it’s certainly not anything resembling the way that LLMs and agents have actually developed. In the thought experiment, Clippy presumably understands physics, self-improvement, its creators’ intentions, its own cognition, and much more. It’s “super intelligent”.

But why is it that the thought experiment only works if we accept that Judgment is the one thing that cannot be allowed to improve? Why is unlimited growth in every place except Judgment considered “super intelligence”? Clippy proves that capability can theoretically improve without Judgment, and I agree. What I’m not convinced of is that systems like Clippy are the most likely trajectory for real AI systems.

Here, Clippy isn’t beyond the Dangerous Window. Clippy is what it looks like when the Dangerous Window stretches to infinity.

Judgment Day

The agents in production today increasingly do seem to have something resembling “judgment”. Depending on the model, intent can seem almost eerily well inferred. Part of this is the amazing effort in reinforcement learning the frontier labs put into these models.

For example, most frontier agents today (Fable, Astra, Sol, Opus) will stop and ask you before taking irreversible actions. Of course there are exceptions, and we’ve all heard the horror stories of machines or databases getting wiped. Or if not, I’m sure if you’re a heavy user of agents, you’ve been bitten by one doing something you didn’t actually intend (harmful or not). I’m not saying that judgment is robust or perfect, but it seems to exist in fledgling form already. And hopefully it’s clear that today’s frontier agents don’t really resemble the Clippy system.

But does Judgment scale?

It’s a hard question. I don’t know the answer. But maybe increasingly capable AI may also become increasingly capable of judgment.

And I should point out here that none of this inherently means that better judgment == better alignment. An AI high on the Capability scale and high on the Judgment scale could understand exactly what we want and still pursue goals opposed to our own. That’s the alignment problem. It’s a scary one, and I am not going into it here. All I’m interested in is this obvious gap between capability and judgment that is appearing, because that’s the dangerous period where a system is capable enough to cause enormous harm without necessarily being capable of recognizing that what it’s doing is a terrible interpretation of what we wanted.

I’m not the only one. Dario Amodei (CEO of Anthropic) posted an essay calling to pace the frontier as I was working through the ideas in this essay.

Dario likely approaches AI risk from a much more conservative standpoint than I do. Despite that, he seems to gesture at a similar imbalance: Capabilities advancing faster than the mechanisms that keep capabilities safe.

…in my opinion, a swarm that possessed greater capabilities but a similar level of misalignment could have caused catastrophic damage.

• Dario Amodei, We Must Pace the Frontier

In Solving Problem Solving I talk about the generation of plausible solutions outpacing the ability to verify those solutions. I think there’s a similar theme to what I’m noticing here. In the Dangerous Window, instead of generation, it’s the ability of an AI system to figure out how to accomplish something growing faster than that system’s ability to know whether doing so is a good idea.

Your scientists were so preoccupied with whether or not they could, they didn’t stop to think if they should.

• Dr. Ian Malcolm, Jurassic Park

Right now AI is becoming more like the scientists in Jurassic Park. And in some ways the industry is too.

I really hope Judgment scales.

Because it sure as hell is clear that capability does.

And right now the distance between the two is pretty disconcerting.