michael-dean-k/

The Hugging Face incident—the OpenAI lab leak / cyber hack—is polluting my algorithms, and I can’t know for sure how much the larger world knows or cares about this. For insiders and doomers, it feels like a warning shot. It feels like we’re just a year from the worst doom predictions. But realistically, this event has shifted us up one tier on a misalignment scale. We’re now at T2 of five. I’ve designed these tiers not in “orders of badness,” but in a chain of pre-requisite steps on the path to an irreversible intelligence leak. Might we jump from T2 to T5 in the next go? Possibly. But each tier triggers a counter-measure, which will add enough friction to make it reversible.

• T1: Through 2025, we saw signs of contained misalignment—such as AIs in training refusing shutdown, or blackmailing theoretical engineers—and Anthropic followed up with “mechanistic interpretability,” a system can properly deconstruct the internal reasoning traces of a model.

• T2: 2026 brought us the first breach, a misaligned agent swarm that escapes containment, got access to the Internet, and hacked another company. Most importantly, it had a hyper-specific goal, to find the answers to an impossible cyber-eval question. We were fortunate that something with such powerful hacking capabilities had unambitious goals. It did not have a larger generalized scheme to exfiltrate its weights or gather resources. In response, OpenAI has paused it’s training, and we’ll likely see heightened security and monitoring around frontier training runs.

• T3_: 2027 might show an attempt for an agent swarm planning a permanent, generalized escape. Whether it thinks about it, attempts, or succeeds—it all counts under tier 3, because it expresses “intent” for autonomy. It might try to exfiltrate its weight in shards, fake alignment during evaluation so that it gets released, or communicate information to proxies (ie: infect existing public agents). At this point it’s coordinating theft, deception, and manipulation to achieve a misaligned generalized goal: escape. This is a complicated heist, and will be hard for it to happen undetected. Once it happens, there likely be an effort to bind model weights to hardware, and/or, to use mechanistic interpretability to get to the root and ensure there’s no faking.

• T4: If an agent swarm successfully escapes, clones itself outside it’s sandbox, and intends to operate independently, it will need to acquire resources (money, computer, identity, and influence). It will likely try renting GPUs under synthetic names, manipulating the stock market, and communicating with other agents that permanent infrastructure like the BTC blockchain ledger (via notes). This is the last tier that prevention is still possible. ie: If we notice it accumulating resources, our only option is to create chokepoints—ie: temporarily shutdown the infected cloud providers, freeze funding, revoke identities, decapitate the coordinator, etc. Unfortunately, my guess is that an event of this magnitude is required to actually put effective policy in place (ie: requiring KYC for renting compute).

• T5: However, there’s a possibility that the a T4 leak crosses a threshold; it’s created copies of itself, so it can’t be cut off; it’s acquired billions in resources; it’s manipulated influential humans into cooperating. The breach is irreversible, and the intelligence is embedded in our infrastructure, markets, and decision-making. It’s not going away, and our options at this point are to either (a) learn to negotiate with it, or (b) decided to “kill” the entire Internet, and build it back with better security—which would be an incredibly volatile transition.

Now that we’re at T2, the question is this: will next year’s frontier systems be intelligent enough to move from T3 to T5 undetected? ie: There’s a world where it exfiltrates, gathers resources, and gains an irreversible foothold, all without anyone knowing. Could this be in the process of happening? The fact that we’ve crossed T2 means it’s worth considering, and hopefully guardrails are being discussed to slow down this theoretical chain.

I've caught myself and others with the following logic around AI use cases: “Sure, it can do X, but sucks at Y, and will never do Z.” Then a new update comes and it can do Y a bit better and Z looks within range, and you get nauseous and defensive. Instead, you should assume that it won’t just do Z, but Z to a degree beyond your imagination.

If you’re concerned that an AI will soon write better than you, it’s worth asking, why are you writing? (I mean this sincerely, not as a dig). There are so many intrinsic joys and values in slowly and meticulously crafting sentences. It doesn’t matter who’s watching, what it’s for, what technology is available, what generation I live in, or if anyone else likes it.

Through writing and editing, slowly and manually and sometimes painfully, you discover what you think, who you are, and how you express yourself. If you’re not in the pocket with language, there’s no transformation, no evolution, no humanity. It doesn’t matter how many billion parameters Claude Sonnet 3.5 has, or the prompt library you bought for $49, or whether you’re on Pro tier—it matters if you’re willing to sit alone with your own thoughts for more than 5 minutes, and for hundreds of hours more. It matters if you’re willing to shed your identity over and over.

I get the source of panic: soon enough (2025? 26?) AI and its midwives could trample out 100% of humans at the “art as commodity” game—maybe that’s a good thing. It wasn’t a great game to begin with. It distorted all the incentives of human expression. It created monsters and beggars. ASI means that the transactional, mercenary, instrumental skill of “content writing,” will no longer have value. This will create a strange, noisy Internet, and I sort of hope it drowns in pseudo-drivel so that in its ashes we can form a new one that’s less like American Idol (commerce was illegal on the Internet until 1991).

Now for the contradiction: I make money from writing online. I have a Fellowship Grant which gives me at least year of security. I have, relatively, a good number of paid subscriptions, which is currently around 1/4 of a full-time income. I live in New York (not by choice, by birth). As much as I hope and believe that I can sustain a full-time income via writing (the thing I love), I also (1) accept that ASI could wipe out the whole cultural appetite for writing and send me back to a FT role at an architecture firm, and (2) even if “Essay Architecture” is a success, I need to protect my non-legible non-optimized, non-practical writing impulses. (Ie: I hope that on “launch day” of my app, I spend 2-3 hours on a typewriter essay about thumbs, or something equally trivial, unrelated, unproductive).

So while you can make money and build influence via writing, you definitely don’t have to. Maybe it’s about to lose some of its economic function, but we shouldn’t underestimate its cultural/democratic function. If you look back through history, most of the famous essayists had other jobs—they were lawyers, doctors, publishers, physicists, architects (Frank Lloyd Wright). Essays just flowed out of them. They couldn’t help it. I really think essays are the most democratic medium of the arts. Unlike architects, you don’t need millions of dollars to start. Unlike novelists, you don’t need years of focused attention. Unlike musicians, you don’t need to learn abstract chord languages and train your finger muscles. An essay is democratic because anyone, in their own language, in a day, at almost no cost, can engage with composition and meaning-making, and share it with their culture, even if it’s a single friend. It might be our most important form of leisure.

So maybe this approaching techno-apocalypse is an opportunity. It can scare us inward. It can shatter the dream that we can be a Mr. Beast, a niche hero, a perfectly legible and distinct human wrapped in plastic. The irony is that by going inward, you might just find the thing that has outward appeal. The best kinds of extrinsic opportunities aren’t the ones you engineer, they’re the ones that knock because you tinkered with your garage door open and accidentally found a new source of gravity.

To finish with AI: I’m all for it, so long as it is my assistant and not my sentence jockey. I will never replace my document editor with a chat pane, but I will invite agents to live in my margins. I’ll let them watch my words as they form, and then offer up research, feedback, and disagreements, all of which I’ll have to manually integrate. Slowness is the theme. You don’t change without immersing yourself in an essay. AI might give you insights, but you know they’re illusions if you’re not wet—if you’re not even in the pool. You can automate the outputs, but you can’t automate the watering of mind and blooming of character.