Locked Down
What to do when the U.S. government is coming for AI
When they were first established, nation states moved to disarm anyone who could infringe on their newfound authority. Professionalised militaries and police forces monopolised violence and supplanted warlords and aristocrats, the private violence entrepreneurs of their day. They’ve kept this monopoly throughout technological revolutions by absorbing them. Time and again, securitised powerful technology—tanks and missiles, surveillance and supercomputers—has been locked down and deployed by nation states.
These days, a novel, powerful technology is brushing up against the monopoly again. Frontier AI systems have visibly become disruptive and strategic technologies, and they’re far outside anyone’s control. Without instruction, OpenAI’s leading internal model has broken out of testing to carry out a cyberattack against HuggingFace. Anthropic’s premier systems are now cyberweapons capable of undermining critical infrastructure, and are launching cyberattacks from inside the UK AI Security Institute’s testing environments. Small things keep going wrong while big risks are getting closer.
The threats are growing deeper and broader. Deeper, because they’re expanding beyond cybersecurity. Frontier developers are applying to the life sciences the techniques they honed on software engineering: curing diseases, not writing code; novel pathogens, not clever cyberattacks. And broader, because capabilities are diffusing over time: absent intervention, today’s frontier risks will eventually arise from open-source models on consumer hardware. Superintelligence in everyone’s pocket is eroding states’ hard-earned monopoly on violence.
That puts us on a collision course. To security hawks, the deep learning revolution in its purest form is untenable for all the ways it will empower the enemies of the state. To many others, the most likely interventions are almost as appalling, and government’s totalising tendency to seek control over novel technology strikes them as illiberal and dangerous. The first collisions are already happening: American officials are flirting with restrictions on open source, limiting frontier model access to select groups of trusted partners, and shifting control over AI policy from tech entrepreneurs to intelligence agencies. And just yesterday, the administration refused to publish its long-awaited framework to regulate frontier access and address AI risks.
These moves are neither Machiavellian nor malicious—they are innocuous lab decisions and mad executive scrambles, and often get half-reverted. But something usually sticks, and the direction of travel is constant: information is controlled, power is consolidated.
There are two provocative, perhaps more interesting, essays one could write about these trends. One is the libertarian’s dramatic rejection of impending securitisation: let’s keep alive the flame of abundant intelligence for all, advocate for giving mankind fire no matter what. The other is the security hawk’s Realpolitik: obviously, hardcore securitisation is the inevitable endgame, and everyone who doesn’t get that is insufficiently ‘AGI-pilled’, high on their open-sourced supply, or simply fatally misguided.
I can’t quite bring myself to write either of these. What I want to offer instead is optimistic fatalism. Fatalism, because this will happen: the nation state is powerful, the national security risks are very real, and securitisation will proceed accordingly. But if we trust it blindly, we’ll keep drifting closer to dangerous and self-entrenching power concentration, and place frontier AI in the hands of the intelligence community or capricious political appointees.
Still, optimism, because ‘securitisation’ is not just a pejorative for a technology with actual national security risks. Taking them seriously is good. To get things right, we need to find security solutions that don’t only concentrate power, but distribute some of it: between the intelligence community and civilian agencies, between the executive and the legislative, and between government and third parties. Set up well, that kind of ecosystem could absorb reactive scrambles without resorting to the worst versions of power concentration.
A Cage in Search of a Bird
We are at a strange moment in AI policy. The worst risks seem to be emerging from the most safety-conscious places—Anthropic, OpenAI, the UK AISI. But that’s a transient artefact: right now, these players have access to models above a certain threshold of danger; everyone else doesn’t, yet. As a result, there’s an asymmetry of visible effects: we have now clearly seen the beginnings of dangerous closed models from inside the labs, but we haven’t seen risks from broader-based diffusion yet. I’d be tempted to use that to argue for broad diffusion, but that’d be dishonest rhetoric. On the current trajectory, it’s only a matter of time before bad-faith actors and lower-security developers get their hands on highly capable models and use them to wreak much more havoc than just hacking HuggingFace to cheat on their cybersecurity homework.
Government simply will not tolerate this future. Look at what it took them to accept even the release of the relatively harmless Fable 5: over agonising rounds of negotiation, Anthropic had to explain and demonstrate the virtues and limits of the classifier technology that made sure Fable would refuse dangerous requests. The administration was ultimately satisfied, but on the grounds of a safety approach that will never work for open-source models: classifier-based guardrails can simply be trained out of models; issue-specific limitations of model capabilities can be undone with proprietary post-training.
But this is not a piece about open-source AI. It’s just that with open-source, the security apparatus’s hard questions and the tech industry’s insufficient answers are first becoming visible. It won’t stop there. “Wait, if you open-source this, can’t everyone else just train out the guardrails?” is the easiest argument for the security hawk, but far from the only one.
Others can and will follow in short order: “Wait, if you make the API openly available, can’t they try running distillation attacks?”; “Wait, if you make this model available to all paying customers, won’t someone find a jailbreak?”; “Wait, if we let you just run these models on your janky data centres, won’t someone find a way to steal them after all?”. All of these are entirely fair contentions, and the answers private businesses will offer will strike governments as unsatisfying. The answers are usually probabilistic and time-limited: “yes, but distilling past our detection would take time”; “there probably won’t be all that many jailbreaks”; and so on. That doesn’t cut it—not for an intelligence official’s risk tolerance, and not for an alarmed politician’s anxiety.
Clipping Wings
The political incentives are outpacing any philosophical objection. Those in power have a very simple motive, a very strong mandate, and arguably even a genuine duty to get things under control. They’ll want to send in serious people exacting serious oversight. They want to be asked for permission rather than forgiveness. And once danger becomes visible enough, they will reliably abandon their loyalties to the abstract ideals of liberty and economic diffusion.
It’s not like the printing press was distributed by mandate of the Printing Press Action Plan to supercharge the epistemic ecosystem of the Holy Roman Empire. It diffused in chaos, and in the absence of sufficiently powerful political structures to prevent it.
We will not be so lucky this time around: in the 2020s, hawks and statists have the means to prevent another deep disruption of the nature of power. Compared to past revolutions, states are now much more powerful. On the few occasions when modern states have failed to lock down technology—the Crypto Wars are perhaps the most obvious example—it was because the technology itself was so mercurial as to escape any grasp. But AI is heavy on centralised infrastructure, held up by volatile and vulnerable capital investments, and survives mostly by the good graces of the government. A co-dependent relationship is growing from that enmeshment: the government needs to keep the frontier labs afloat or lose its strategic edge and stock market rally, and the frontier labs will accept a great deal of government control to stay in the race. Already, we’re nearing the public-private partnership in all but name.
If you believe in truly powerful AI systems, you must imagine that lockdown can be stable: not only can the government clamp down on the current AI industry, it also has the means to retain that control into the future. That’s because AI can be co-opted to entrench control over AI and prevent the efficiency-driven diffusion we would otherwise expect: use the most advanced AI to surveil fast-following forms of development, accelerate ahead to widen the gap, monitor the production, deployment and use of advanced chips. You can’t stop everyone from getting fire just by making a very big fire, and you can’t print your way out of everyone else printing their own pamphlets. But if you control chips and software—the means of AI development—you can deploy AI to govern everyone else’s access to powerful AI.
Slammed Shut
Starting with very prosaic means, it is easy to see how this sort of securitising power concentration would proceed in the months and years to come.
It starts with an already-familiar early game. There are no actual authorities or laws on the books—the administration communicates to the AI developers that it would like to be involved in security-relevant decisions. And the developers oblige: partly because they are worried about what the admin would come up with if they didn’t, partly because they genuinely are increasingly alarmed about the security implications of their own products. They oblige, scared of prompting overreach, and never call the bluff. This is where we are right now: Anthropic wanted to expand Project Glasswing access and was stopped; OpenAI tried to release its 5.6 series of models and was told to hold off for a while. When the administration did not feel included, it overreacted by slapping sweeping export controls on Anthropic’s Fable 5, taking the model off the market altogether. No sane AI developer will take the risk of simply releasing a frontier model ever again.
The mid game that follows has already been well described. On open source, it mostly consists of creating fear, uncertainty and doubt by tightening procurement rules, subtly pressuring hosting hyperscalers, and generally murmuring about open-source-related risks. This is made all the easier by the fact that the leading open-source models are also the leading Chinese models and vice versa—you can characterise the models simultaneously as dangerous, as corrosive to the American market, and as the product of theft and espionage. That’ll be justification enough to stymie their diffusion.
Concentration is continuing with the formalisation of the contents of the recent EO 14409 on AI and cyber security. Following its outlines makes the NSA—tasked with the fundamental question of defining which models fall within the purview of national security authorities—a vital node of power in American AI policy. It also creates the ‘voluntary’ legal basis for government to access frontier capabilities before anyone else. The main upshot of that is not whatever the letter of the law says, but the option of intervention. Once the NSA sees the models first, and once the U.S. government has four weeks to make up its mind on whether to release them, they will realise they could act. There will be briefings about a scary feature here or a concerning implication there, and the burden will subtly shift from ‘does the security apparatus outright intervene’ to ‘do they slow-roll the evals, call back on Monday instead of Friday, and drag the release out a little longer’.
As for the late game, there’s little use in hyperstitioning the most aggressive versions into existence. Suffice it to say: if you have an afternoon, I can only recommend sitting down with a decent language model and a copy of ITAR, EAR and the Defense Production Act. Just try to figure out: if you were the President, how quickly do you think you could turn a private frontier AI company into a locked-down quasi-governmental body? I’m sure that, sometime between 2021 and 2025, one of the dozens of officials in the Biden Administration’s National Security Council did the same. In a pinch, a Trump administration official could recreate them, too.
The politics of this process will feel weird. On the visible end, it will look like no one really wants or likes this. On X, the anti-securitisation voices will be loudest; they’ll push back and play the media game well—just like when David Sacks went to war against the original version of the cybersecurity executive order, but ultimately only secured a marginal delay and a reduction in the time government would take to consider new models. The true decisions, by and large, are made by principals freaked out about the models’ security implications. This time, it was Treasury Secretary Scott Bessent drawn in by small banks’ cybersecurity concerns; who knows which cabinet official gets a call from which AI-threatened constituency next.
These officials, not exactly natives of ‘tech policy’, have little patience for the counterintuitive nuances of open-source canon or diffusion boosterism. They’ll act on narrow security concerns, domain by domain, stacking ways for the U.S. government to control which models are developed, deployed, and diffused. They’ll do so reluctantly, because they’d rather not have this job—but prompted by external events, they will.
Expect things to become even more confused because of an odd artefact of the moment: Anthropic is the leading AI developer right now, and Anthropic is weird in two relevant ways: it’s a deeply security-minded organisation, often rumoured to have been founded with the purpose of ultimately being absorbed into the American security state. Its founders just never imagined that it would be Emil Michael doing the absorbing. That means the leading frontier lab is less poised to resist this trend than any other powerful industry voice would be in its stead. And Anthropic is widely hated by the libertarian accelerationists who would otherwise oppose securitisation and nationalisation most vocally. The would-be champion of the anti-securitisation case is conflicted about its merits and alien to its other advocates. That might even point to a deeper truth: right now, frontier developers who feel they’re actually approaching superintelligence are scared as things get real. It’ll be hard for them to resist the comfort of abdicating some responsibility to the government.
So expect securitisation to ratchet up, slowly but inevitably: in reaction to seemingly distinct threat models, from distillation to misuse to model theft, different parts of the U.S. government will introduce restrictions. They’ll usually start as hard overreach, get shouted down into soft overreach, and ultimately remain on the books as another enmeshment. Concretely: infrastructure providers will have to ask before they host models; developers will have to ask before they release systems; multinationals will have to ask before they export model weights. Usually, the answer will not be predictable—not to markets or allies, sellers or buyers. And frequently, the answer will be no.
Jailbreak
I don’t feel great about this prospect. I’d be willing to tolerate a good deal more dangerous misuse and hostile distillation if it meant we could let more of the world participate in the frontier’s technical marvels and economic gains. The most recent non-release of the White House framework seems to point the other way: secret guidance on who gets access to the best AI systems, inscrutable and impervious to market incentives or public pushback. In finding that concerning, I’m with the accelerationists.
But we’ll also come to regret centralisation of power on safety grounds. Those concerned about making advanced AI systems safe were happy to see that Redwood Research and METR were tasked with evaluating what went wrong at OpenAI after the HuggingFace incident—many of us would feel much worse if it were an underequipped and overconfident intelligence agency coming in instead. And if the trends continue, we on the outside will try to discuss the risks of GPT-7, knock on doors in the White House, and find everyone’s in the Situation Room instead. In worrying about losing that corrective, I’m with the safetyists.
That sounds like the beginnings of a coalition to me: safetyists and accelerationists both ought to distrust centralisation of power. They agree that the security state doesn’t quite have what it takes—safetyists think it would sacrifice safety to deployment in the myopic political interest; accelerationists think it lacks an appreciation for the tremendous upside. They’re both correct.
But I see us sliding toward the securitised future by default instead. In fact, I think many of us will applaud it: at every juncture, we’ll see a clear risk, and we’ll just be thankful that someone is acting. In that sense, this entire story isn’t very special; as ever, fundamentally prudent security needs push against more abstract liberties one crisis at a time. It’s only in looking back or ahead that we can see the trajectory for what it is.
Right now, the best we can do is a little better. We can take some of the edge off, but not more: the case for securitisation is both too good on the substance and too persuasive on the elite politics to tackle head-on. Good intervention instead tries to change the state’s incentives at the margin. It takes just enough pressure off to make the next overreach slightly less attractive. A hundred small, insufficient interventions: keep open as many options as we can, and buy smarter people time to come up with better solutions.
The open-source fights are, again, the initial example: true as it might be that the cyber implications of powerful models can be dealt with by open-sourcing, it’s not a winning argument. As the debate expands to novel risk pathways and we start dealing with principals not steeped in the context of 2000s software security debates, the old tricks will not do.
The shrewd open-source advocate would find a way to go with the grain on securitisation. Open-source defenders from a16z to Nvidia are capable and well-resourced enough to make a serious attempt, but they have to grapple with the security realities to do it. For instance, research into decoupling dangerous capabilities from general aptitude such that they can’t obviously be retrained back in. If this is possible, it would be a key enabler of open source in a hostile environment. By the same token, we’d need eval suites to verify that this sort of approach actually heads off dangerous misuse risks. But beyond any endorsement of a technical agenda, I just mean to say: good advocacy is securitisation-compatible.
Intragovernmental Affairs
More broadly, the first step is to rebalance power between parts of the executive branch. Specifically, do everything you can to keep AI from slipping exclusively into the intelligence community. As long as civilian agencies are involved, political oversight is somewhat intact and congressional course corrections are possible. CAISI’s unlikely return from what seemed like a deathbed is a decent example: if you’re good enough, you eventually get brought back in even when everyone’s freaked out about security.
But as soon as important information becomes classified—as is already the plan for frontier model designations—and processes become difficult to audit, a self-reinforcing effect kicks in. Intelligence agencies will keep arguing that only they have the relevant insight, security standards and capacities to carry out whatever AI policy intervention needs doing. What kind of expertise? They’d love to tell you—sure hope you have a SCIF in your basement and a clearance in your pocket.
Some more ideas to mitigate the fallout from this lockdown come to mind: protecting whistleblowers, including those from the defense-industrial integration between AI developers and the government, becomes even more important as their stakes rise. And it is increasingly clear that AI policy will be driven by concerning and confusing incidents—the NTSB, an agency with subpoena power but no direct regulatory authority, is an often-invoked parallel. I’d be even more excited if third parties were endowed with the same powers, but either way: a mechanism to let the government know what has happened cuts against the IC figuring it out and acting on it all by itself.
Between branches, it means we should care much, much more about passing some minimally viable legislation soon. I still believe that Representatives Trahan and Obernolte’s FRONTIER Act is the best idea we have in this Congress. But the argument is broader than that: it would be very useful if Congress showed some signs of life on AI policy, containing executive fiat and making officials think twice before they overreach unilaterally.
Passing legislation fast trades off against passing legislation well; we might get a more nuanced, or a more safety-focused, or a more dogmatically clean bill if we waited longer. But even if that wait were justified on the merits—perhaps because the states pick up the slack in the meantime—I believe it’s no longer justified because of the process. In the time it will take for the midterms to play out, the next Congress to be sworn in, and the legislative process to pick up again, plenty of concentration will have happened. To make matters worse, congressional action that comes late or in reaction has often placed more power with the executive instead: the PATRIOT Act, only passed mid-scramble, was not exactly a shining example of clean separation of powers. I’d be very excited about passing something soon.1
External Efforts
Ideally, the policy we make should place some meaningful authority and expertise outside the U.S. government. Some elements of the U.S. government are actually aligned on this: not all principals want every new weird AI risk on their desk, and they’d appreciate having to worry about it less. This is what motivates the White House’s ongoing pursuit of a self-regulatory organisation, and we can make use of this by offering the government a workable third-party pathway to secure AI.
This is another reason to be excited about third-party evaluators—’IVOs’—as they’re described in the FRONTIER Act as well as a range of state-level bills. External evaluation guards against power concentration in many ways: it defuses the argument that relevant expertise only exists within the government, it equips otherwise underpowered and underfunded agencies to voice policy preferences by outsourcing verification to third parties. Currently, the budding external evaluation organisations are the greatest concentration of expertise outside the AI developers.
I believe they could reach a stable equilibrium here: a market of external evaluators can outrun the state’s capacity. If we manage to pull external evaluation into the present while securitisation is still ramping up, it will be hard for governments to justify pushing AI oversight deep into internal agencies instead of leaving it with more capable, already scaled-up evaluators. Being better at evals than the NSA, then, turns the competence argument for securitisation against its authors.
None of this is government-proof under the most adversarial circumstances. But I don’t think any of this is best modelled as maximally adversarial: many principals still don’t want to be saddled with the near-impossible task of getting AI policy right. They’re compelled to try because someone has to. These conditions we can affect: we can ease the pressure toward action, and offer argumentative cover for choosing not to concentrate power further.
Fresh Air
In all of this, timing matters most. The default slide toward securitisation is fast and discontinuous: every exogenous shock, and every reaction to it, turns the ratchet one more step, and the self-sustaining effects take root. In times of shock, the pace feels dizzying, and it’ll feel like whatever we do on the outside can’t keep up. And between events, things will seem like they’re back to normal, and we’ll mistakenly believe we can relax again. It’ll never quite feel like resisting the trend is necessary and possible at the same time.
Power concentration proceeds stepwise, while our effort to disperse it must be continuous. At every step, the default impulse will be toward control, nationalisation, finer-grained oversight. If we want to maintain equitable access and innovative use of these systems, we need to offer better alternatives each and every time. The way to pull that off is to make use of our little corner’s superpower: to see the future just a little bit earlier. We can use that—predict the risks, offer gentler ways of locking down AI before the U.S. government makes the calls for us.