On Superintelligence

I wanted to like this book, I really did, but I can’t say it did it for me. It’s both dated and not. It came out in 2014, which feels like an aeon before large language models, and it’s hard to read now without recent advances getting in the way: when we think of “AI” today we think of LLMs, of ChatGPT and the other chatbots, which Bostrom would call “oracles,” and which still have a long way to go before anything like “superintelligence.” His sense of “AI” is much broader than LLMs, and the book isn’t really about AI at all but about superintelligence, which could just as well come through biological modification, designer drugs, eugenics, or uploading human consciousness — though Bostrom argues, rightly, that AI is probably the fastest route there.

There’s an attempt to predict how an “intelligence explosion” might unfold, but the book is mostly about ethics: how might we make an AI humane, what values could we instill in it, how do we keep it from destroying us, how do we manage resources. Reading it now, it’s hard not to feel pessimistic — several labs set up AI ethics teams, and they’ve either made little progress or been largely disbanded in favor of economics and acceleration. There are sections on economics, social systems, and our “cosmic endowment”: the first two are bleak, with Bostrom warning that we risk falling back into a Malthusian trap in an AI-enhanced world, and the last concerns space colonization, where he fears superintelligence could either let us expand across the universe or squander all our resources and guarantee us a short existence.

My biggest problem is that he seems to take for granted that the superintelligence explosion will happen. That’s been heavily debated over the past half decade thanks to LLMs, and I’m unconvinced LLMs are the path; he also tends to assume that superintelligence means our kind of intelligence multiplied by orders of magnitude, with the fear being a paradigm shift in which superintelligent beings come to see our values and ethics as illusions. I think the real situation is even more alarming. I think it far likelier that “superintelligence” means not an expansion of intelligence but the production of an alien intelligence. We already see it with LLMs, which can’t make sense of human concepts like fingers and teeth and causation — watch any AI video and you’ll know what I mean: the fingers and teeth come out wrong, cars move backward. The model is trying to emulate the world it interprets without sharing our ontology; it’s representation without knowing what things are, or perhaps it does recognize them but sorts them into categories other than ours. There’s a kind of schizophrenia in it, where socially agreed-upon categories are thrown out for something wholly new, and the result becomes illegible to us and harder and harder to control. Bostrom lists techniques for containing an intelligence explosion — “boxing” it in a closed space, building in indicators for an automatic shutdown — but we’ve already seen that even LLMs can be manipulative, so what about other forms?

The real risk is competition. If a single state, firm, or lab dominated AI, there would be little danger — it could work at a leisurely pace with all the proper safeguards. But that’s not what we have. In an arms race, which is what AI research is now, speed matters, and that pushes organizations to drop their ethical limits to maximize progress. To be blunt, that’s extraordinarily dangerous — though I’m not sure it matters, since the more firms are involved the higher the risk, and there are a great many firms in the race today. For all my qualms, it’s an important book, and Bostrom seems to have been the first to really sound the alarm. He cites Eliezer Yudkowsky often, and I may have to turn to Yudkowsky’s writing to see what he has to say.