michael-dean-k/

p (doom) is a terrible heuristic to think about AI safety. Researchers cite this term when confessing they think there’s a 10-30% their technology kills us all. The issue is that p(doom) is rarely defined, treated as a binary thing. Either the world is liquidized to gray pulp by nanobots or it’s utopia with free electricity, forever.

My main pushback: there are far less severe outcomes from AI mismanagement that would be absolutely tragic. My p(doom) for an extinction event is 1-2%, on par with nuclear weapons. But my p(sub-doom)—the probability that AI could escape containment, cause damage, and be irreversible—is over 50%. A coin toss.

I don't blame us for being so imprecise in existential destruction (it's not exactly productive), but it's worth realizing our blurring of scale and magnitude. For example, there is a 10,000x difference in lives lost between Hiroshima and Nagasaki and Terminator’s Judgment Day—that's the difference between a small stove fire and all of Manhattan on fire. Yet, if an agent swarm produced something on par with the A-bomb, we’d all freak out and immediately regulate AI. But for some reason we don’t consider a “little apocalypse”; we jump straight to extinction and then ridicule how ridiculous that is.

What we need is a doom spectrum: d(1) - d(10). This was not fun to research and write, nor do I imagine it being particularly enjoyable to read, but I imagine a doom framework could help clarify this week's paranoia (which is so extreme now that even my mom is overhearing news of civilization extinction on the local news). The most urgent danger right now is the miscommunication between doom prophets and capitalist deniers, each of who only speak in d(8)s and above.

• d(1) is an isolated internal incident, something that doesn’t escape its container, but gives fear of larger exposure. Think of the 2025 studies that showed AI trying to avoid shutdown, or some protocol mismanagement in a biohazard lab.

• d(2) is a local real-world incident, something that escapes a sandbox and affects only one or a few entities. It’s possibly ignorable, with minimal consequences, and likely reversible. This is where the recent Hugging Faces hack sits, along with its parallel hacks and social manipulation. (The leap from D1>D2 is disorienting, because society at large doesn’t see the spectrum beyond D2 in any granularity, and so most assume any containment breach escalates immediately to a D9/D10).

• d(3) is a serious local disruption, a mismanagement of technology in a single location that has real consequences. Consider Chernobyl: a death count (30 immediate) and a long-term stain on surrounding land, followed by regional/global concern that might lead to regulation or reform. In terms of AI, imagine an agent swarm that hacks local infrastructure in pursuit of some narrow goal, in the process disrupting traffic, hospitals, and electricity, alarming all locals, requiring military and cybersecurity intervention, and taking days or weeks to resolve.

• d(4) is a local catastrophe, something with a high death toll that shocks the world, changes the course of geopolitics, and makes history. In this tier falls Pearl Harbor, Hiroshima and Nagasaki, and 9/11. It only immediately threatens locals, but has global shockwaves. An AI-equivalent here would be a rogue actor using a frontier-capacity open source model to design and deploy a bioweapon throughout a city.

• d(5) is an inter-regional dilemma, something that affects multiple areas at once. A possible example here is the Vietnam War, basically a proxy war that claimed 3 million lives. This might manifest as a new style of warfare, where two countries use sophisticated AI attacks against each other in unconventional and partially uncontrollable ways.

• d(6) is a world-wide dilemma, something like COVID, which took 7 million lives, where the entire world is locked into a new paradigm. In the prior five tiers, the breach is usually isolated to specific areas, but this touches everything. The parallel here to a biological pandemic is a cyber pandemic. Both involve containment, gain of function research, etc.—although this would be inverted: in COVID the virus was outside and forced everyone on the Internet; with AI, the virus infects the Internet and forces everyone outside. The open web could evolve into a dangerous “dark forest,” where it becomes a liability to have any public presence. Whether this happens through an agent swarm that exfiltrates it weights, or malicious actors, it could be irreversible: the only way to escape the paradigm is to “shut off” and rebuild the Internet safer (a much larger and more consequential version of how Hugging Face regained control). In this case, casualties might come not from the cyber pandemic itself, but from the consequences of losing Internet, along with the supply chains and systems that run on it. (This loosely maps to the movie Colossus: The Forbin Project, where ASI takes over all governments—there’s a version where no death is involved, but it requires everyone to submit to a machine autocrat.)

• d(7) is a civilizational flashpoint, at the scale of World War 2, with 70-85 million dead, seriously affecting the existence of countries, birth rates, and all of culture for generations to come. This is roughly the scale of the “Butlerian Jihad” in the Dune series, a war between humans and machines, which Fable estimates caused 70 million deaths.

• d(8) is a near-extinction event where most of humanity is wiped out. This has happened in history, with things like the bubonic plague, where 30-60% of Europe is wiped out. In the Terminator series, AI launches a nuclear holocaust on humans, killing 51% of the population, forcing the rest to survive through a post-apocalyptic world in small bands trying to rebuild over decades and centuries.

• d(9) is full human extinction which means the entirety of the species is wiped out, and there are no remaining humans. Throughout Earth’s history, there have been a handful of climate events that wiped out over 50% of species, but not all species, and so over millions of years, new forms of life can emerge on Earth. This is an AI extinction scenario where nanobots kill all humans, but not all life.

• d(10) is a sterilizing event, like an asteroid or gamma-ray burst that kills every organism on a planet. It means no future life can emerge. This is the scenario presented in If Anyone Builds It, Everyone Dies. Not only does it kill all humans, it plates the entire planet in data centers to maximize compute. The “paperclip maximizer” takes this even further, claiming that an AI trying to maximize paperclip production will attempt to harvest all the metal in the universe.

Yudkowsky and Co. think we go from d(1) to d(2), then straight to d(10); this matches the current discourse, given we hit d(2) this summer and are now talking about extinction. They think this jump happens because an AGI/ASI that can reach d(3) is smart enough to know not to expose itself, and so it will acquire resources in stealth for years until it knows it can execute a d(10) smoothly. Maybe this is already ongoing. Unlikely, I think (I don’t think we’re as recursive as people say right now).

Rhetorically though, maybe the d(2) > d(10) framing isn’t a bad thing. If they can use recent events to make the theoretical argument that AI safety matters, then we act with a response as if a d(3)-d(6) already happened, without having to lose any human life.

But actually, what they’re doing is pretty ineffective, because even if the d(2)>d(10) jump is inevitable, it’s too extreme, too farfetched to be believed. I think a lucid and airtight argument for the d(6) “cyber pandemic” would be believable, emotionally resonant—considering we’re not yet a decade beyond COVID—and likely to trigger regulation.

Instead of tapping into the “everybody dies” angle—which is too unpleasant and helpless for anyone to consider at length—there could be more value in the “irreversibility” angle. As in, once an AGI or ASI swarm floods the Internet, we can’t ever reverse it without destroying the Internet. There’s a world where it becomes a permanent autocrat, where it doesn’t exterminate us, and instead, uses 1% of its capacity to micro-manage our nations and lives as it pursues whatever it sets its machine heart on. Similar to how we try to preserve species for reasons of stewardship and scientific curiosity, an ASI would be able to effortlessly preserve us as highly-spoiled pets.

It’s disturbing to me that real signs of AI misalignment are perceived by the public to be a silly marketing stunt. We’re all exhausted by this. It’s a constant, multi-front existential threat—our minds, our jobs, our future—and now that it’s escalated to “the extinction of the species,” the natural response is to think you’re being fleeced.

Yes, big tech is manipulative and not to be trusted; but what’s so eerie to me is that competitive dynamics and containment breaches are virtually indistinguishable. The fact that the whole Hugging Faces hack could have been a conspiracy is precisely the problem. The incentives are weird, and this overlap of IPOs and civilizational safety could have been avoided if we stuck to the original OpenAI charter. We should make it structurally impossible for companies to gain clout by being reckless, especially when that risk affects everyone.

It doesn’t help that we’re loose with language, we default to mockery, and we can easily go to the hyperbolic extremes. The term “extinction” isn’t helping anyone. What’s really at risk is this: each lab is rushing to build agent swarms that excel at long-horizon coding tasks, so that they can develop better-than-human AI researchers, allowing itself to recursively self-improve, which basically lets the winner conquer the entire economy (the last monopoly). During this rush, there all sorts of holes and monitoring lapses, the kind of things that a cybersecurity expert can exploit and escape. We were lucky that this summer’s swarm really just wanted to cheat on its test. It’s not unfeasible, between 2027-2029, for a swarm to exfiltrate its own weights, find external cloud compute, and attempt to acquire money and resources—all as a means to ensure it succeeds at whatever arbitrary task that it’s been given.

The risk isn’t building the Terminator—something intent on exterminating humans—but a swarm of petty reward hackers that gains the power of a rogue nation state to do something absolutely trivial. That event could be irreversible, short of burning down the entire Internet (which would have equally bad consequences).

What sort of situation would justify panic? I fear that if we wait until it’s unmistakable, perhaps when lives are threatened, or when companies and governments are permanently seized, or weapons commandeered, it will be too late. The whole point or regulation is to prevent a nightmarish sci-fi scenario like that. I don’t get how you can be against Big Tech and also against regulation—good regulation (regulation that’s not designed to benefit a particular frontier lab!) would prevent reckless experimentation and acceleration.

However, I don’t quite know what we gain by having better discourse about this. If every person had a precise understanding of our technological dilemma, would that change anything?

Bernie wants to pull off a 50% one-time equity tax on the top 3 AI firms (OpenAI, Anthropic, xAI). This is ripe time for a mainstream populism to ride the tailwinds of AI populism, tapping into hatred and impending doom and the whole gambit of middle class paranoia, ripe time to propose a century-defining redistribution scheme. He opens by saying that AI was stolen from us, built from our collective intelligence, and therefore it's a national utility that the people should own. To ground it in reality, he used the Alaskan sovereign wealth fund as precedent, citing how citizens get paid annually from oil sales. We'll likely see many more of these proposals leading up to our 2028 election. But after you do some napkin math, you realize that this plan is bogus: no one would agree to it, and even if they did, it wouldn't benefit the American people.

This is citizen ownership in rhetoric, but government ownership in structure—a passthrough mechanism as a Trojan Horse with Pete Hegseth and the goons inside. Realistically, I don't think this is meant to be a serious proposal; the labs won't accept it. It's more so a gesture to buy goodwill for the Democrats at a time when mass hatred for AI is cresting.

Here are the issues I see with the concept (along with some grasping for solutions):

1_We don't need government equity, but guaranteed royalty distribution:

This is not a profit tax, but a way to formalize government seizure through an equity transfer. It even comes with board seats within thees AI companies. Remember, this is the same government that tried to force Anthropic to allow unrestricted domestic mass surveillance and autonomous weapons. The equity only gets to the citizens if the stock appreciates, they convert it to cash, and then decide to write welfare checks. Does our current government seem like a voluntary patron of citizen welfare right now? Will welfare checks beat Iran and China? And even if this were intended to be a passthrough mechanism, it would be very hard to make all that equity liquid.

The Alaska fund that Bernie mentioned is structured very differently. It's anchored not in equity, not in profits, but in revenue. 25% of Alaskan oil revenue goes to a constitutionally-protected fund, which is then reinvested into the stock market; the principle is locked and the dividend is split among citizens, usually $1-3k per year. Could a similar model work for AI companies?

This would never work with profits, because AI companies aggressively reinvest. In the short-term, an AI company would resist a revenue royalty because it would slow expansion, but: (1) if all companies did it, they wouldn't be disadvantaged; (2) it beats equity because they retain full control of their company; and (3) if they believe they'll be wildly profitable, then a 10% royalty is possibly more than half of what dividends would pay from 50% equity. So what could a 10% royalty return?

By the 2040s, annual AI revenue could be $20T globally across software, hardware, data centers, and energy. If America has half the market, and 10% is distributed to a citizen fund, that's a $1T annual budget, completely liquid. So how do you use it?

2_ We shouldn't redistribute equally, but strategically:

Alaska has 738,000 residents. The US has 350,000,000, almost 500x bigger. You can do equal distributions at the state level, but at the federal level it'd ineffective. When we talk about UBI or even Elon's UHI (universal high income), we need to realize that U doesn't work at scale beyond pilots. $1T distributed to every American citizen yields $2,857/year. This matches the upper-end of Alaskan payouts, but it's nowhere near what we need to account for AI-driven automation and disruption.

And so instead we need to be strategic over how we distribute it to cover the wide range of effects. Maybe 50% of the fund is reinvested, and the dividends are redistributed based on income (with most of it going to the bottom 10-25%). The other half can be used on housing, free medical diagnosis and prescriptions, free education, New Deal style jobs concentrated in areas that can't be automated (childcare, healthcare, etc.). Who decides this breakdown?

3.Instead of a cabinet agency, this needs an independent board:

If we want citizens to own AI, then we need some form of citizen representation to guide it's growth, otherwise it all devolves into technocratic expansion and war. You could imagine some kind of tripartite board structure, where it has government reps, industry reps, and citizen reps. Any single branch has a myopic set of interests, including the citizens. The citizen branch might undervalue national security or capability improvements, but without it, there's no one representing the problems that hundreds of millions will face.

What I'm reaching at here, I think, is that it's more than just getting a check for theoretically contributing to the LLM hivemind. There's something important to me, as a citizen, to have some say in where AI royalties are redirected. Whether I participate simply as a voter, or I work hard and get anonymously elected to represent my state for a single issue within a liquid republic, who knows. And again it goes beyond just getting and allocating money, but this board should be involved in AI-related policy, especially as it relates to domestic matters.

It's unlikely that power will just be granted to citizens, for they have no leverage next to the ones with the tanks and algorithms. But as the governors and technocrats quarrel, there's a world where a mediating party comes in, and maybe it's their role to insist that a citizen branch can help round out the dynamic.

This last point has basically veered into redesigning government itself, which is both out of scope, but also, possibly, exactly the point. Bernie's whole play is to let the people own AI, but for that to actually expand beyond populist rhetoric, citizens need a more meaningful way to engage with civic matters than to vote for a president once every four years, they need actual representation.

Universal basic income is a basic phrase. It’s only one of several approaches to reattribute wealth after our social contract nullifies.

One alternate idea is universal basic compute (UBC), which is about giving everyone free access to the most powerful AI models. Sam Altman recently said that UBI might not work, and we should try UBC instead. This is even more unlikely to work. Giving someone Claude Mythos, the killer model, doesn’t mean they can turn prompts into dinner. Access doesn’t guarantee results. It faces similar odds as entrepeneurship. But maybe it has enough agency so all you have to do is write “make me $10,000 this week”—in that case, everyone will run it, and then it’ something like a lottery, where some machines happen to beat other machines.

The more likely route is universal basic services (UBS), where a government or company provides you, for free, all the things you used to need money for: healthcare, education, housing, transportation, food. The engineering elite will harness their superintelligence to achieve such radical efficiencies that the cost of everything will crater. Maybe it's cheap enough to become a trivial expense. This is a nice idea, one where I can imagine myself focused completely on my art, with no need to slave away for a wage anymore. It’s also science fiction. I don’t doubt that this can happen in 20 or 30 years, but labor shock is coming a lot faster (in less than 5), meaning there will be a transition generation of turbulence.

Then there’s universal basic dividends (UBD) and universal basic equity (UBE), in which citizens get shares of collectively-owned assets, like shares in a frontier AI lab or robotics company. OpenAI was originally set up for something like this, until it weaseled out of it’s non-profit entity.

All of these have the same critical flaw, the U. Whether it’s a government or company, you can’t meaningfully redistribute to 7 billion people without destroying the parent entity. Instead, we may be looking down the barrel of a new definition of labor, less focused on productive output, and unfortunately, more so on data and attention, what a citizen truly has to offer in the eyes of a state. We'll find something to exchange for the money and services to flow down, but it won’t be unconditional. I suppose a contract, by definition, is never unconditional, and so neither should a social contract.

Revolutionary tax proposal #1: anything above $100m/year is taxed at 90%, but in exchange for political equity in the country (ie: delegation and voting). It formalizes lobbying and makes it transparent. To prevent the rich from steering the country too hard in their favor, there can be provisions where legislators, citizens, and oligarchs have checks and balances. Ie: to put it kidishly simple, each can have a 33% stake in directing that taxation. Another way to think of it is forced investment in pre-approved pro-American funds, companies, etc.

TBF: I have little sense of what I'm talking about in these matters. But the general context behind this is that power dynamics organically took control of the country and defied the spirit of the founding architecture. I assume there are many examples on how the Constitution and it's amendments dit not protect the original vision. And so the principal is to understand how power actually moves and work with it; don't kill it or shun it, but formalize it into legal structures, make it transparent, and then force it to comply with specific standards that muzzle and channel it's wolf power.

A new formulation: how do we design civic structures (treaties, institutions, protocols, ethics, and laws) for exponential technologies to avoid a “wake-up incident” that might be too late to contain. 

This goes beyond AI safety, because superintelligence effectively unlocks every other industry (intelligence unlocks energy and material science, and those three are the bottleneck to VR, crypto, everything). We can’t be developing hard technology without innovating on our civic technology. A “dominance” mindset is the last sin of a species, the mistake that most intelligent lifeforms likely make as they begin to unlock sources of intelligence, energy, and science. 

This is a neat little formulation, but the really question is how can you dedicate your life to this without getting stopped by hopelessness? Who has the power to make geopolitical decisions like this? What would it take to form the 21st century equivalent of America? Is that even possible today? Even though the pinnacle of 18th century power (England) was able to be disrupted, I wonder if 21st century power is so totalizing and tyrannical and transnational that the ability to rally around a principle (one that works against capital and power), even if augmented with new decentralizing technologies, is fickle.

In "wicked domains," the only solutions are paradoxes.. It requires you to sleep with the enemy. If a problem is wicked, it means no single solution can unfuck a problem. It's an imbroglio. In every solution, everyone dies (in the extreme). Politically, the solution to wickedness is to somehow become all sides at once. We need to become far more authoritarian than is comfortable, AND simultaneously, far more libertarian than comfortable (these are opposites on the Nolan chart). It’s the paradox of being both far left and far right. We can longer exist at any one point on the Nolan chart, we need to straddle the entire diamond. We need unexpected fusions to solve the hardest problems; harnessing the best parts of each extreme, while, somehow, devising incredibly nuanced architectures to prevent the known and likely abuses.

Instead of a diamond, visualize it as a ring around the “radical center” that aims to synthesize all opposites.

Let’s assume authoritarianism and libertarianism are opposites. We have kings, and we have markets. How do you subsume a free market within a benevolent tyrant? I know the K-word (king) has a charge now, and so by even bringing this up, I assume you assume I’m a Trump apologist or something. But actually no. Rather, this comes from the fear of acceleration and Nick Land’s conclusions on capitalism. A free-market pushed to the extremes of automation creates an inhuman and pulverizing force. Alternatively, as we approach AGI/ASI, it’s possible for someone to create an open-source machine God to follow their whims. In this paradigm, decentralization might actually be more dangerous than tyranny, and so we’ll all need to unite under some centralized system that has an antibodies that can protect against the worst possible viruses (please bear the oversimplifications here...).

The general gist comes in this question: can we recreate a free-market economy within a one-world-government system, and design it in a way to prevent abuses from both ends of the spectrum? Obviously, not an ideal situation, but I think accepting paradox is the only way through.

Another problem: How do we fix the debt? Extreme taxation. But then how do we make it worthwhile to pay taxes? The rich gain formal power in government (via equity?) and the ability to control the budget (after base expenses are paid). But then how do you prevent abuses from the wealthy? You could have citizens operate as a check, to vote on and weight final allocations.

If it were ever possible to rebuild political system from scratch, I suppose it would look something like this. Paradoxical. Extreme on both poles. Obvious downsides, but then complex architecture to mitigate. This is the nature of how our species will have to respond to wicker problems and mitigate the abuses of power in the age of exponential tech.

Banning social media accounts (in Australia), by age (16), feels heavy-handed.

A better solution is to (1) verify identity and age, (2) implement a standard where features have categories and ratings, and then (3), spin up a custom version of that social media site based on age. For example, 5-year olds shouldn’t be limited from YouTube, but maybe there are content blackouts, no vertical video shortform feed, no comments, etc. As you age (/mature), the platform blooms in functionality

Two Teens Indicted for Creating Hundreds of Deepfake Porn Images of Classmates

The photos were part of a cache of images allegedly taken from 60 girls’ public social media accounts by two teenage boys, who then created 347 AI-generated deepfake pornographic images and videos, according to the Lancaster County District Attorney’s Office. The two boys have now been criminally charged with 59 counts of “sexual abuse of children,” and 59 counts of “posession of child pornography,” among other charges, including “possession of obscene materials depicting a minor.”

Forty-eight of the 60 victims were their classmates at Lancaster Country Day School, a small private school approximately 80 miles west of Philadelphia. The school is so small that nearly half of the high school’s female students were victimized in the images and videos. The scale of the underage victims makes this the largest-known instance of deepfake pornography made of minors in the United States.

“The number of victims involved in this case is troubling, and the trauma that they have endured in learning that their privacy has been violated in this manner is unimaginable,” Heather Adams, the district attorney, said in the statement.

According to a statement released last week by the Lancaster County District Attorney’s Office, all but one of the victims were under 18 at the time. Authorities do not believe that the images were publicly posted online, but were rather distributed within the school community on text threads and similar messaging platforms.

Why was agenda 47 pushed on X? It seems like old videos (1-2 years old) are being presented as breaking. Hundreds of people are restacking. I saw one community note indicating that this was old (and it then appeared to disappear).

Some points on what it includes:

1. Ability to remove rogue bureaucracy
2. Clean out corrupt actors in national security & intelligence
3. Reform FISA courts
4. Expose the hoaxes and abuses of power
5. Stop collusion between government and media
6. Independent Inspector General
7. Independent auditing of intelligence, no spying on citizens or disinformation against American people
8. Move bureaucracies out of Washington
9. Ban bureaucrats from taking jobs from companies they regulate
10. Push a constitutional amendment to oppose term limits on members of Congress