This reflection by Mahmoud Rasmi is based on the monthly Tech Leaders Salon discussion, where we discuss books on topics including philosophy, strategy, management, and leadership. If you’re interested in joining the conversation, you can learn more about the discussion group here.
From the Enlightenment to AGI
We are trying to create a new species in our own image, a species that can learn on its own, develop an inherent sense of curiosity, perceive the world and represent it adequately, and make decisions and act in the real world.
Ultimately, the goal seems to be the achievement of Artificial General Intelligence (AGI), with entities that are capable of exhibiting human-level intelligence to an extent that they are better than us, helping us achieve unprecedented levels of scientific, digital, and social progress. AGI would allow us to accelerate growth, eradicate disease, combat poverty, and make breakthroughs in record time. That’s the optimistic narrative, or the story the “effective accelerationist” movement is telling.
The techno-optimist worldview has roots in the scientific revolution and the 18th-century Enlightenment. Reason, the scientific method, and new tools made fresh empirical findings and economic progress possible. People had a sense that history was progressing linearly, with new milestones being built on earlier ones.
During the Enlightenment, the prevailing belief was that rational thinking would suffice in solving our problems and answering all our questions. In the absence of a satisfying solution or answer, people thought scientific and rational inquiry would eventually provide one in the future because of its continuous momentum.
For over a century, things seemed like they were heading in the right direction. With unprecedented growth across domains, new fields emerged, new discoveries were made, and new industrial infrastructure was created that triggered even more expansion.
This linear progress, however, was brought to a halt when the very same innovations and discoveries that were meant to improve our lives wreaked havoc in two major world wars, culminating in the atomic bomb, with millions dead, widespread famine, homelessness, and poverty. We also saw the banality of evil, to quote Hannah Arendt, in action, and how quickly things could get out of hand with the rise of totalitarianism and fascism.
The Alignment Problem
The very tools that drove progress also brought about destruction. Technological breakthroughs alone were not enough to curb evil, disease, or poverty. We were still left with perennial questions for which we had no clear answers: what constitutes a good life, or good governance? How should we handle conflict, or reconcile personal liberty with the public good?
These questions carry moral and legal dimensions that scientific or rational inquiry alone can’t settle. What values guide us? How do we treat each other, allocate public budget, or define what justice, fairness, or transparency mean in practice? Can these principles be passed down to future generations? How do we acquire knowledge, and how should we transmit it? Are values relative to each culture, or is there a universal baseline? What moral frameworks align with our principles, and how do we evaluate whether an action is ethical? Most importantly, what do we actually want?