Artificial general intelligence or superintelligence has been one of the most widely cited terms in the world of AI but

there is hardly any consensus on what it means or what its possible implications could be for society. That being said,

leading AI labs like OpenAI, Google and Anthropic are racing to be the first to create a model that could reach AGI

status.

However, Anthropic co founder and Chief Scientist Jared Kaplan, in an interview with the Guardian, explained that

humanity will have “the biggest decision” on whether it takes the “ultimate risk” of letting AI systems train themselves

to become more powerful.

As per Kaplan, the period between 2027 and 2030 may become the moment when artificial intelligence becomes capable of

designing its own successors.

The Anthropic executive says that he is very optimistic about the alignment of AI tools with the interests of humanity

up to the level of human intelligence but not when it exceeds that threshold.

Kaplan on AI training its successor

The moment an AI system begins training its own successor, the guardrails that AI labs currently have on their models

may no longer be enough. Kaplan believes it could lead to an intelligence explosion and may even be the moment when

humans lose control over the AI.

“If you imagine you create this process where you have an AI that is smarter than you, or about as smart as you, it’s

then making an AI that’s much smarter. It’s going to enlist that AI’s help to make an AI smarter than that. It sounds

like a kind of scary process. You don’t know where you end up,” he told the Guardian.

In such a scenario, the AI black box problem would become absolute, where humans would not just be unsure why the AI

made a decision but would not even be able to tell where the AI is going.

“That’s the thing that we view as maybe the biggest decision or scariest thing to do… once no one’s involved in the

process, you don’t really know. You can start a process and say, ‘Oh, it’s going very well. It’s exactly what we

expected. It’s very safe.’ But you don’t know – it’s a dynamic process. Where does that lead?” he noted.

Kaplan says there are two major risks in such a scenario. First, will humans lose control over the AI and will they

continue to have agency in their lives?

“One is do you lose control over it? Do you even know what the AIs are doing? The main question there is: are the AIs

good for humanity? Are they helpful? Are they going to be harmless? Do they understand people? Are they going to allow

people to continue to have agency over their lives and over the world?” Kaplan noted.

The second risk is when the speed of improvement of self taught AIs goes beyond human scientific research and

technological development capabilities.

“It seems very dangerous for it to fall into the wrong hands… You can imagine some person deciding: ‘I want this AI to

just be my slave. I want it to enact my will.’ I think preventing power grabs, preventing misuse of the technology, is

also very important,” he said.