Should Humanity Make Way for an AI “Worthy Successor”?

These days, it’s not unusual to encounter “doomers” in AI circles—people who think that superintelligent AI will wipe out humanity either because it wants our stuff or finds us annoying. And there are plenty of serious AI researchers working on “alignment,” or ensuring that AI systems’ goals match our own, so that AI will support human flourishing rather than ending it. But not too many people are thinking about how to provide the best outcome for the universe if, indeed, superintelligent AI is ready to leave humanity in the dust.

Enter Daniel Faggella. The founder of the AI research company Emerj, Faggella argues that it’s critical that we build AI systems that are “worthy successors“ to humanity. A few weeks ago, he hosted a symposium at a cliffside mansion in San Francisco, where AI insiders aired their hopes and fears about a post-human future. IEEE Spectrum caught up with Faggella to hear more about his controversial and provocative vision.

How would you explain the concept of the worthy successor to someone who’s never encountered it before?

Daniel Faggella: The gist of the worthy successor is a post-human intelligence that’s so capable and morally valuable that you would consider it best if it, rather than humanity, took the mantle of the future and determined the future trajectory of intelligence.

The core belief here is that artificial general intelligence is probably unlikely to be aligned [with human goals]. So if the torch of humanity is valuable, what is it about the flame that’s valuable? I think all torches go out eventually, and ultimately marriage to any one torch is scorn for the flame itself. My hypothesis is that the flame is consciousness and autopoiesis, or self-creation. If AGI [artificial general intelligence] has those two things, it would carry the flame into the future. Because we cannot hold this torch forever: I’m arguing that we might have a generation with this torch until it’s turned into something else. So we ought to ensure that that which we create has those two moral traits. Because when we’re gonzo, is the cosmos filled with value, or is it all gone?

“The gist of the worthy successor is a post-human intelligence that’s so capable and morally valuable that you would consider it best if it, rather than humanity, took the mantle of the future” —Daniel Faggella, Emerj

What’s your time frame for when this question of succession becomes important? Is it within our lifetimes?

Faggella: Oh, absolutely. I would suspect there’s a really good shot that within the decade, we’re already feeling the destructive and transformative forces. I think this torch is really within threat within one to two decades. I think we might be dealing with the final flickers here.

Imagine that everything goes great according to your rubric, and the worthy successor is identified. What happens to humans?

Faggella: We should do our damnedest to get the best shake we can get. Some people would say the…