Superintelligence, honestly

Few topics attract more confident claims and less evidence than “superintelligence”. This lesson separates three things that usually get blurred together: definitions, measurements, and forecasts. They deserve very different amounts of trust.

Definitions (these are choices, not facts)

• Narrow AI does one kind of task, sometimes far better than any person: playing chess, folding proteins, recognising speech.
• AGI (artificial general intelligence) loosely means a system that can do most cognitive work a human can do, about as well. There’s no agreed definition. Researchers, companies and contracts each draw the line in different places, which is one reason arguments about “is it AGI yet?” go nowhere.
• Superintelligence usually means going far beyond the best humans at nearly every intellectual task, not just matching them.

When someone claims AGI has arrived, or is decades away, the first question is: by which definition?

Measurements (the solid ground)

What has actually been measured is narrower and more interesting than headlines suggest.

Scaling laws (2020). Researchers at OpenAI found that a language model’s loss, the “how wrong” number from Learning is turning knobs, falls smoothly and predictably as a power law with model size, data size and the compute used for training. Some of the trends held across more than seven orders of magnitude. Bigger models also turned out to learn more from each example.

Compute-optimal training (2022). DeepMind trained over 400 models, from 70 million to over 16 billion parameters, and found that model size and training data should be scaled equally: double one, double the other. They tested this with Chinchilla, a 70B model given the same compute budget as their 280B Gopher but about four times more data. Chinchilla outperformed Gopher (280B), GPT-3 (175B) and others.

Forecasts (where experts disagree)

The largest survey of AI researchers to date asked 2,778 people who had published at top AI venues. Their answers are worth reading as a spread, not as one number:

• On when unaided machines could outperform humans at every possible task, the aggregate estimate was a 10% chance by 2027 and 50% by 2047. That 50% date was 13 years earlier than in the same team’s survey a year before. Forecasts are moving, which is itself a sign of how uncertain they are.
• 68.3% thought good outcomes from superhuman AI were more likely than bad ones.
• Yet between 38% and 51% gave at least a 10% chance to outcomes as bad as human extinction.
• More than half said “substantial” or “extreme” concern is warranted about scenarios including misinformation, authoritarian control and inequality.
• Most expressed substantial uncertainty, and they disagreed about whether faster or slower progress would be better.

So the same group of experts is, on average, optimistic, worried, and unsure, all at once. That’s what an honest picture of an unsolved question looks like.

In 2023 the Center for AI Safety published a one-sentence statement, signed by many AI researchers and leaders: “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.” Other prominent researchers argue that such risks are overstated, or that attention should go to harms already happening, such as bias, misinformation, surveillance and concentration of power. Both camps include serious, well-informed people.