New role: AGI Alignment research

I'm taking a new role within Google -- I'm joining Google DeepMind to help set up a technical alignment team in the Bay Area. [Edit: There’s already a team in London, which will continue to exist and do great work. I’m helping to set up a US branch of that team.]

You might have questions! Here are some answers.

What the heck is alignment?

Alignment is the idea that we should have AIs aim for human goals, and do so in a way that is fundamentally controlled by humans. This sounds kind of obvious, but in fact it turns out to be pretty hard, especially once you consider (future) AIs that are human level or higher for ~all cognitive tasks (this is called AGI, for Artificial General Intelligence).

Wikipedia has a page if you want a more authoritative take.

Why is AI alignment so hard?

A few reasons:

1. We don’t have a way to clearly set goals at all for large language models (LLMs), the most powerful current AI system we have. You can ask it to do stuff in a prompt, and sometimes it will do what you want, but frequently gets it wrong, in part because your request is just very ambiguous because language is ambiguous.
   
   You can use training data to encode your goals, which is more reliable, but takes some translation — you put your idea into guidelines, raters label the examples according to the guidelines, and then the model learns to match the pattern in the data. At each of those steps there is a translation to something potentially quite different, and in practice it takes a ton of iteration to get something kind of right, and is basically impossible to get it totally right.

2. Our understanding of LLMs is very empirical and doesn’t have great theoretical underpinnings, so it’s hard/impossible to know if something will work without doing a lot of experiments. One example of this is if an AI encounters a situation unlike anything in its training, it’s ~impossible to predict what it will do, which seems bad since new experiences do come up from time to time.

3. Once AGI shows up, we’ll have a new problem: how do we supervise AIs that are smarter than we are? We need some sort of solution that scales with how smart AI is, on top of having a solution that works at all. Which we don’t.

Those first two points seem like general points about ML rather than Alignment in particular.

Fair point.

So why are they there?

Because those issues become very risky when we reach AGI. Right now they are barriers to mundane utility — making LLMs actually useful in everyday life — but imagine if you didn’t have clear goals for an air traffic control system¹, or AIs in various kinds of leadership or decisionmaking positions. Generally if you have a really powerful system with a lot of authority you need to have similarly powerful controls.

We have people in positions like that, but we don’t have proof that they will do the right thing.

This is true! And we have lots of examples of people with too much power doing horrible things (see: the 20th century). We’ve managed to mostly find control systems that keep the worst things from happening too often with humans. We need to find that kind of control system for AI too, only the stakes are higher and we may not have the centuries of experimentation it took with human organizations to get it right with AI.

But what will you actually do?

Hire a team of research scientists and software engineers to start to tackle some of the really hard technical (as opposed to, e.g. governance/regulations) problems in Alignment.

And what will that team do?

I don’t know yet. We’ll start by looking for great people, and thinking about where additional investment could make the biggest difference.

That’s pretty vague.

That’s not a question. But yes. My entire career has basically been: this area looks interesting and/or important, I’ll go help figure it out.

Why you? Are you qualified for this?

There are maybe a few hundred people in the world working on AGI alignment, and I think it’s one of the most important problems of our time — like the statement says², as critical and dangerous as pandemics and nuclear weapons.

I’d go further and say that the world is dangerously underinvested in safety for risks like nuclear weapons or pandemics, and investment in AGI safety is way way below those.

I’m lucky enough not to have to worry about career risk, and maybe I can help bring more attention to and investment in this space. Honestly once I started to think about AGI safety in those terms, it became hard to even consider other options.

I still am fuzzy on what this is all about…

Imagine a world where humans aren’t the smartest things around, or the most numerous — like if there were a thousand or a million AIs doing things for every person. And those AIs essentially run the economy, drive research, write laws, collect taxes, you name it.

Is this a good world for humans? Is a human-dominated world good for chimpanzees?

I think we’re going towards such a world, and might be there some time this century. I want to try to ensure that our future world is a great place to be a human, and not some scifi dystopia.

Wow. Ok. Are you hiring?

Yes! But I don’t have other kinds of details yet. Still, drop me a note here or at dmorr at google if you are a research scientist or software engineer and might be interested.

1. Even as task as clear as air traffic control could go wrong in a surprising number of ways. For instance, a goal like “make sure nobody dies in a plane crash, minimize safety incidents, and maximize the number of flights that reach their destinations on time” would result in some very unexpected consequences if given as directions to a system that optimized very hard for those results, as AIs do. For example, we might end up with tons of flights with zero people on board, which scores perfectly on every dimension.

With human supervision, you can just assume a certain amount of common sense and the expectation that a lot of things are taken as understood without being specified. With AI, though, it’s super hard to know what it takes as understood, and more or less impossible to specify all the edge conditions.

2. Yep, I signed it.