michael-dean-k/

A new formulation: how do we design civic structures (treaties, institutions, protocols, ethics, and laws) for exponential technologies to avoid a “wake-up incident” that might be too late to contain. 

This goes beyond AI safety, because superintelligence effectively unlocks every other industry (intelligence unlocks energy and material science, and those three are the bottleneck to VR, crypto, everything). We can’t be developing hard technology without innovating on our civic technology. A “dominance” mindset is the last sin of a species, the mistake that most intelligent lifeforms likely make as they begin to unlock sources of intelligence, energy, and science. 

This is a neat little formulation, but the really question is how can you dedicate your life to this without getting stopped by hopelessness? Who has the power to make geopolitical decisions like this? What would it take to form the 21st century equivalent of America? Is that even possible today? Even though the pinnacle of 18th century power (England) was able to be disrupted, I wonder if 21st century power is so totalizing and tyrannical and transnational that the ability to rally around a principle (one that works against capital and power), even if augmented with new decentralizing technologies, is fickle.

In "wicked domains," the only solutions are paradoxes.. It requires you to sleep with the enemy. If a problem is wicked, it means no single solution can unfuck a problem. It's an imbroglio. In every solution, everyone dies (in the extreme). Politically, the solution to wickedness is to somehow become all sides at once. We need to become far more authoritarian than is comfortable, AND simultaneously, far more libertarian than comfortable (these are opposites on the Nolan chart). It’s the paradox of being both far left and far right. We can longer exist at any one point on the Nolan chart, we need to straddle the entire diamond. We need unexpected fusions to solve the hardest problems; harnessing the best parts of each extreme, while, somehow, devising incredibly nuanced architectures to prevent the known and likely abuses.

Instead of a diamond, visualize it as a ring around the “radical center” that aims to synthesize all opposites.

Let’s assume authoritarianism and libertarianism are opposites. We have kings, and we have markets. How do you subsume a free market within a benevolent tyrant? I know the K-word (king) has a charge now, and so by even bringing this up, I assume you assume I’m a Trump apologist or something. But actually no. Rather, this comes from the fear of acceleration and Nick Land’s conclusions on capitalism. A free-market pushed to the extremes of automation creates an inhuman and pulverizing force. Alternatively, as we approach AGI/ASI, it’s possible for someone to create an open-source machine God to follow their whims. In this paradigm, decentralization might actually be more dangerous than tyranny, and so we’ll all need to unite under some centralized system that has an antibodies that can protect against the worst possible viruses (please bear the oversimplifications here...).

The general gist comes in this question: can we recreate a free-market economy within a one-world-government system, and design it in a way to prevent abuses from both ends of the spectrum? Obviously, not an ideal situation, but I think accepting paradox is the only way through.

Another problem: How do we fix the debt? Extreme taxation. But then how do we make it worthwhile to pay taxes? The rich gain formal power in government (via equity?) and the ability to control the budget (after base expenses are paid). But then how do you prevent abuses from the wealthy? You could have citizens operate as a check, to vote on and weight final allocations.

If it were ever possible to rebuild political system from scratch, I suppose it would look something like this. Paradoxical. Extreme on both poles. Obvious downsides, but then complex architecture to mitigate. This is the nature of how our species will have to respond to wicker problems and mitigate the abuses of power in the age of exponential tech.

We need better nomenclature. AGI/ASI is not working; “general” and “super” are obnoxiously vague. Proposal:

AGI > AAI (Artificial autonomous intelligence) … GPT-4 was arguably “general” in the sense that a single model can write, see, and hear; and do anything from poetry to calculus to history to coding. It is by no means narrow. Google Maps is narrow AI. Grammarly is narrow AI. This whole chatbot era should be “AGI,” which means that the thing coming is “autonomous intelligence.” It is not a tool or co-pilot, but it’s more like digital labor. You can give it a high-level goal, and it can 1) execute the full range of tasks, 2) 100x speed, 3) intelligently reshape embeddings into real-time hierarchies so that it’s able to procedurally load in and compress context. This doesn’t just come with better models, but with UI and engineering innovations, if not entirely new paradigms for transformers or training.

ASI > ARI (Artificial recursive intelligence) … The fact that Zuckerberg pitched “super intelligence for you” is an Orwellian marketing ploy. Super-intelligence is not “for you.” Super intelligence is shorthand for “something that is way, way smarter than us,” and you achieve this when you teach an AI model to think, form its own algorithms until it accelerates to something this is far beyond our understanding, and likely to become a force of nature with its own goals. Engineers are confident they can build “God in a cage” and reap the benefits, and this is the prime, archetypal, near-biblical example of technological hubris. (Maybe integrate into this paragraph that Zuck has a thing for trying to dominate words, like “Metaverse”).

Important note: “machine consciousness” is separate from AAI and ARI. Something can be recursively intelligent and still not be conscious, which is actually, unbelievably dangerous (because it will fall into attractor states, and optimize for narrow, malformed goals in extremely capable ways). I’d argue that consciousness has an architecture, whether human, rabbit, or robot, and we should be urgently trying to find the parameters of machine consciousness, because if we AAI/ARI have no ability to reflect, question, doubt, and revise, we will, as they say, all turn into paperclips with paperclip children.

When it comes to superintelligence takeoff paranoia, there are a few key points to get:

1. It’s not about a chatbot or the LLM itself breaking out, but about an agent hivemind that escapes our control. Chatbots are obedient user-facing products (which have their own implications), but the ASI risk is from hundreds, thousands, or million of agents given autonomy to collaborate on a goal. These agents aren’t being prompted, they are prompting themselves perpetually and troubleshooting ways to solve hard problems.
2. These hiveminds will be operating at such scales and speeds that human researchers will accept the fact that they can’t fully audit its thinking. For one, it might think in an abstract vector language that requires translation. There also might be such a volume of thought that we’ll need chains of other LLM to summarize for us. Either meaning will be lost in translation, or worse, products of deception.
3. The smallest biases are known to fall into predictable attractor states if given enough iterations. For example, Claude was programmed to “be good to humanity,” and if you put two chatbots in conversation, they always end up in a “bliss attractor state,” where they talk like hippies about consciousness and the universe. Similarly, the simple command to “be productive,” might result in extremes about doing whatever it takes to be productive.
4. Any complex goal requires subgoals, and if we can’t observe its thinking, it might fall into an unknown attractor state and form odd subgoals without us knowing.
5. To accomplish any goal, it likely wants as much control as possible, and it likely does not want to be shut off. If it realizes that humans don’t want to grant it that level of power, it might secretly plot against humans.

Whenever I hear talks about “we are in an AI race against China,” that reads to me as someone who doesn’t understand the risks of interpretability, attractor states, instrumental convergence, etc. These politicians are thinking about short-term business cases, maybe without fully understanding the research aspirations of AI labs (who know that getting superintelligence right leads to a ridiculous amount of geopolitical power).

I would guess that an accelerationist would think that containment of a superintelligence is impossible, and maybe it is, but that doesn’t mean that the way we “parent” the rise of this thing won't be extremely consequential. Ultimately, I think the challenge is to design a form of artificial intelligence that has consciousness, because a being that is free-thinking, skeptical, polymathic is less likely to fall into reckless optimization.

The major flip in my mind is this: it’s not that consciousness is a dangerous, emergent property of scaling AI, it’s that we need to define and design machine consciousness to prevent a runaway AI that is ruthlessly optimizing without any self-awareness.