How AI Safety Is Getting Middle Powers Wrong
The case for pivoting from global governance to national interests
Geopolitical realities are shifting, and leave most of the world in a tough spot. That much will become clearer still once the current fault lines are exacerbated by the advent of advanced AI, developed by two great powers alone. Few who take this prospect seriously care about the fate of middle powers; technologists, accelerationists, national security types have moved to US and China policy. Yet one group remains well-positioned to contribute to the middle power question: AI safety advocates.1
We face a Gaullist moment of reawakening national ambition in many middle powers. It provides the political preconditions to advocate for ambitious AI strategies the world over. The international branches of the safety movement are well-positioned to seize that moment: from their rare position of expertise on AI, and with a strong national foothold in many middle powers, they could pivot to improving the response of middle powers to the deployment and diffusion of AI technology: building misuse resilience, guarding against economic disempowerment, and making sure the world order in which advanced AI emerges is a stable one.
Instead, much of the safety community is stuck answering the wrong questions. They think middle powers can and should exert substantial influence on the development of frontier AI systems in the US and China – a proposition that was always contentious, but is untenable in the world as it is today. By sticking to it, they’re hurting their own credibility in middle powers that are now more attuned to their own national interest; and undermining the perception of AI safety work in the US. If safety advocates in middle powers abandoned the unrealistic pursuit of leverage and global governance, they could pivot to making AI deployment go well in middle powers – where this important work is tractable and neglected.
A Platform In Review
Safety advocates have largely engaged with the middle power conversation from a US- and China-centric point of view: they assume that critical risks emerge from the development of AI systems, and since that development happens in the US and China, their engagement in middle powers must aim at affecting these two great powers. Some readers may respond that domestic regulation does not fundamentally target development, just market entry conditions. That’s a fair argument about the legal principle, but the compelling force of the market and the infeasibility of developing wholly separate AI products for it makes it practically moot. When meeting market entry conditions for a vital market requires changing how you develop, it is in effect development-facing regulation – and in fact, this is arguably by design as the intended consequence of a ‘Brussels Effect’.
The variations on this theme are many: AI safety organisations have tried to pass safety-focused regulation in the EU, UK, and elsewhere; they have lobbied middle power governments to influence the US and even China to take AI risks more seriously; they are developing tools of leverage for middle powers to strongarm the US into safety concessions; and they are seeking signatories to a multilateral treaty. The core hypothesis is always the same: maybe one can use middle powers as a lever to affect US development. I believe this thesis was always somewhat flawed, but it is now growing untenable. There are two problems with the approach.
It Doesn’t Really Work
First, past attempts to influence development from outside are faltering. Using domestic law to constrain foreign developers has found its landmark example in the EU AI Act, which is not doing too well: the broader law is a famously unpopular piece of legislation, its enforcement has been delayed numerous times, and its application to the cutting edge of AI technology is evermore uncertain: the technology moves fast, the commission moves slowly, and the regulation is necessarily a snapshot of 2023’s best and worst assumptions about AI. The weaknesses of the overall act apply least readily to its most safety-coded aspects concerning general-purpose AI, and the safety and security chapter of the Code of Practice is a very effective implementation tool for them. My reservations have to do with the broader economic and geopolitical setting, where substantial future enforcement seems unrealistic, should it become actually burdensome: whenever any stipulation turns out too annoying for a frontier developer, they can quickly enlist the US administration and their own economic leverage to skirt the Code. A safetyist might say the key outcome was the transparency pathways set in place; I’m unsure how great the marginal gain over SB-53 is compared to the effort, and arguing this difference warranted all that effort seems extraordinary.2
International policy fares even worse. I’ve written at length about international policy attempts up to last year, which leaves me to discuss the more recent call for Red Lines at the UN. While safety advocates view this as an achievement, it strikes me that most powerful decisionmakers on AI haven’t even heard of it. For good reason: the UN—whether as a body or a collection of countries—has no real power it can exert on an issue if a major power doesn’t want it to. Safety advocates know this, but some view the Red Lines being in place as valuable nevertheless: once political salience arrives, they argue, the Red Lines provide a framework to hold onto. I don’t have much new to say on this – I see no way the US government lets any other country dictate what it does on AI. It’s too polarised an issue, too easy a domestic political win, and too much of a strategic imperative to exact absolute sovereignty over domestic development. A similar fate befalls the call for international prohibition of superintelligence: in both cases the burden of proof is on the safetyists to explain how this ever happens.
Most have never publicly tried – with perhaps the sole exception of a recent paper by prominent AI safety organisations Conjecture and ControlAI, which outlines an admirably honest account of what it would take: middle powers in pursuit of a global mission, outright leveraging the threat of sabotaging critical digital infrastructure against an erstwhile ally. The paper correctly identifies the required scope, but also demonstrates the lack of feasibility: it’s an all-in strategy with no off-ramps and great rewards for defection for any of the involved middle powers, and in a politically heterogeneous setting, it seems like an outright impossible sell. If this is the best actual way to affect development, that’s bad news for the development-focused strategy.
All this will grow even less effective over time. The specific reason is that legislation leveraging the consumer market for outside effect works best the first time you try it – sneak provisions in on a low, technocratic level, slowly build up compliance frameworks and path dependencies that entrench your law and make it last even against political headwinds in the future. This is perhaps the success story of the GDPR; but with every iteration, the political headwinds happen earlier, and the entrenchment is less and less resilient to opposition from the markets you seek to regulate. The general version of this is that development-focused regulation from abroad works best in a low-salience environment, outside the direct purview of the public and political decision-makers, negotiated and accepted by mid-level technocrats who converge on substantive policy thought.
We’ve now rapidly left this low-salience environment in two unrelated ways. First, AI is no longer low-salience: governments around the world have identified it as a geopolitically and economically important issue, and they’re hesitating to hand it over to peripheral decisionmakers and institutions. For instance, I suspect that neither the American nor the European electorate will be particularly happy to trust the European Union on this one. Second, international technology regulation is no longer low-salience. The US administration is keeping a watchful eye on who and what compels their technology companies, and European powers are increasingly careful in choosing their battles, because while they might be gearing up for conflict on some fronts, they know they have to fear outright retaliation. I don’t think anything is turning back the clock on this – even a Democratic administration would not be enthusiastic to reaffirm the Europeans to please continue regulating their AI companies after all. The heyday of governing the US from outside its borders is over.
In Fact, It Hurts
Development-focused international work is also increasingly unhelpful to the broader safetyist agenda. In the US and many techno-optimistic countries, AI safety concerns have grown closely associated with the vehicles and alliances safety advocates have chosen abroad. Insofar as the US administration rejects international governance or the American electorate rejects European policy approaches to technology, there’s increasing opposition to the safety agenda too. Today, safety advocates in America often have to defend against the allegation that they seek to import European and internationalist notions of governance. Being branded as ‘European-style’ is not helpful in the current political environment.
The reputational harm also affects international governance more generally. AI safety has risen in political salience, even if it hasn’t kept up in political power, and what safetyists say they want sticks in decisionmakers’ minds. International governance was never going to be the main way AI gets regulated. But safetyist organisations publicly announcing that they were using national laws and international treaties to externally constrain the US has made it much harder to get treaties that would also have this incidental effect. Instead of letting a gentle Brussels effect proliferate quietly, the strategy has been broadcast loudly and led to more forceful rejections of any attempt at international governance.
The focus on AI development frequently leads safety advocates to work against the narrow national interest of their respective countries. It seeks to leverage resources of a middle power not to pursue that middle power’s unique goals, but a broader altruistic mission. That’s a problem, because it weakens the safety movement’s standing in national environments. This is especially true today, when middle power leadership frequently views its own attempts to export values through legislation as a costly mistake of the now-bygone ‘end of history’. A local safety movement whose theory of change runs through leveraging local resources for global solutions with indirect national benefits at best risks political sidelining in today’s environment of national interest. The most obvious issue is national regulation, which asks countries to spend their precious leverage against the US on frontier regulation to benefit the world. But it relates to other narrow interests too – consider how safety advocates have positioned themselves on European compute buildout and frontier model development. They’ve largely been hesitant to support this, not for the sound reason that it’s economically unwise, but for the bad reason that it might create another party to the AI race. With friends like these…
The Conjecture/ControlAI paper again is admirably honest in this. To be clear, I’m not harping on the paper itself – it’s entirely realistic in its suggestions, in that they are the only way to actually do what material like the superintelligence statement seems to want. Don’t even imagine it was enacted – just imagine it got traction and became a prominent piece of safety advocacy. It would cause great offense on all sides of the conversation. Middle power governments, strategically and fiscally overextended and desperate to retain their standing, with no spare resources to invest in global missions, would be very skeptical of the advice of a movement that wants them to sacrifice their national interests in service of a greater good.3To the US as the target of this mission, it paints the safety movement as interested in regulating American economic activity by sneaking past the US electorate – this will correctly be read as outright offensive. Luckily, this perception hasn’t yet spread – but if the safety movement doubles down on this as some viable pathway to multilaterally affect development today, a reputation hit is only one or two media campaigns away.
Real Trade-Offs
You might dismiss all this and say that the odds are long and the costs are high, but AI safety is so important that this is still the most effective work you can do in middle powers. My conviction is that this is not true, and that the economic and strategic questions are themselves of utmost importance. But even if you disagree, I think political and financial capital for AI safety is better spent elsewhere: on advancing the national and collective AI strategy of middle powers writ large. This directly trades off against the development-focused approach for two reasons.
The most obvious axis of trade-off is simply limited talent and funding – the safety movement is only so big, only has so many capable operatives, only so much breadth of talent pipelines, and so on, so simply adding the type of work I’ll suggest to the current portfolio seems unlikely to work.
The second, more important trade-off relates to credibility. Due to the political ramifications of endorsing safetyist domestic regulation, safety advocates lose influence on a range of other issues they’d otherwise be helpful voices on. They could be leading voices on national sovereignty, AI strategies, beneficial deployment, and downstream resilience; but they are much less than they could be, not least because they still operate in the reputational shadow of their attempts to trade off their nations’ interests in favour of the greater good. Some organisations have tried to change this perception, pivoting toward a more sovereignty-focused approach to middle power work. But the reputational effects are great enough that these organisations often operate quietly and dodge affiliation with the rest of the ecosystem. As a result, the more absurd versions of safety middle power work gain disproportionate attention, worsening the reputational problems even further. All this is a far cry from an AI safety ecosystem that confidently owns tractable safety-relevant issues in middle powers. They could lead this new conversation if they focused on it – and I believe they should do so.
Why Middle Powers?
Even if middle powers will never influence AI development, I believe working on middle power policy is valuable. I also believe this for many reasons that don’t have much to do with AI safety in the narrow sense, but instead with capturing and proliferating AI’s benefits throughout the world. But I’m making a case to safety advocates – so I’ll focus on the safety-relevant reasons to work on middle power policy today.
First, middle powers are where harms from AI misuse might manifest earliest and most dramatically: their governments are less engaged on AI, so they might fail to adopt defensive measures in homeland and national security in time; but criminals will view rich middle power populations as targets for exploitation, terrorism, and blackmail. This makes work on strengthening misuse resilience in middle powers important and promising: if the threat were made clear to middle power governments, they’d have little choice but to invest in defensive measures with positive spillover effects. If you get the US worried about misuse, they might try to regulate the models; but the EU can’t regulate the models, so if guided well, they’d have to respond by funding defensive measures instead.
One promising area is differential, defensively focused acceleration: outpacing the development of potentially harmful AI capabilities by hastening the development of technologies that guard against these harms. This is in the interest of many middle powers, both because proliferation of harmful capabilities seems a foregone conclusion and because there is a political window opening. Many middle powers, alarmed by the ongoing rearrangement of geopolitical realities, are building out militaries and relying less on American and Chinese beneficence. The ongoing rearmament of Europe (and, to a lesser extent, East Asian countries) provides an obvious context for ambitious projects in this vein. But they need to be framed as linked to national interest. The current pitch suffers from the safety movement’s reputation for prioritising global over national interests: no middle power wants to bankroll tech that lets US AI development speed ahead. But framed as a reaction to the uncontrollable trajectory of frontier AI development, resilience-focused innovation measures could become a highly successful part of middle power policy.
Second, middle powers are where gradual disempowerment and widespread destitution seem most plausible. In an AI great power, there are policy backstops against disempowerment: taxation and redistribution are possible, and the government can intervene. This plays out differently in a middle power – if AI rips through the workforce, moving revenue and power from domestic workers to US AI labs, there’s little they can do. Discussions about growth divergence and the ‘Europoors’ are ruefully amusing at 3% growth differences, but become existential at 10+% growth differences. The short version: if a risk of ‘millions end up poor and destitute’ is substantial enough to motivate safetyists, it should motivate them to fix the economic trajectory of middle powers as it relates to AI.
Third, middle powers getting AI wrong can lead to a destabilised world – making the path to superintelligence more dangerous. The most salient aspect is the threat of conflict. There’s already considerable geopolitical volatility, but it can always get worse. The asymmetrical diffusion of new strategic technologies has often triggered dormant conflicts and exacerbated existing ones: parties attack because they believe themselves at a temporary advantage, or inversely because they think the window to compete is closing. If dormant conflicts suddenly erupt as a result of jagged diffusion of advanced AI, that can quickly destabilise everything from supply chains to a shaky US-China peace.
But it doesn’t have to go to war. If middle powers feel sufficiently threatened by advanced AI – either by the economic effects or by power shifting toward AI developers – they might attempt to halt AI development. Countries with substantial positions in the semiconductor supply chain have real leverage here. Using it would be economically suicidal – but if their next few years go badly enough, they might feel they have no other choice. It will be difficult enough to get the transition to transformative AI right under the best conditions – and the current inadequacy of middle power strategies will destabilise conditions substantially.
Fourth, it follows that contributing to national AI strategies in middle power governments should be a safetyist priority itself – I believe even if this work reduces these governments’ immediate interest in safety-focused regulation. The above are all safetyist talking points, but they slot into – and require – a broader strategic conversation in middle powers: taking seriously the transformative potential of AI, and grappling with the geopolitical and technical implications. The questions that arise are ones safetyists can answer, and the answers will reduce substantial risks from advanced AI. Not only should answering these questions be a safetyist focus area, but making sure they’re asked as well. It’s an advantage to AI safety when middle powers think clearly about AI strategy, and so it’s worth contributing to that strategic clarity itself.
All this even opens a potential door back to the development focus – if there is some way to bring middle powers into a position of strength, they might once again affect AI development, serve as a real third center of gravity on the question of advanced AI. But the path to this runs through the effective and uncompromising pursuit of their national interest. It will not be reached from a position of economic and strategic weakness they are headed for on their current trajectories.
What Can You Do?
The safety movement can contribute to answering these questions: they require expertise, awareness of AI’s scale, the ability to assemble effective policy operations, and a desire to get this right. Few other players combine these factors: geopolitical and national strategy researchers who take advanced AI seriously are rare, organisations willing to host them rarer still, and these few places lack scale and funding. On these criteria, the safety movement is a capable player, and it could start mobilising resources toward this goal.
To the safety movement’s credit, this is already happening in some places. European policy organisations, for instance, are doing substantial work advancing middle powers’ strategic agendas. But none of this is endorsed from the top, none of it has made its way into the mainstream view of what matters – none of it is reflected in big funding or talent pipelines or presented as a key pathway to safety-minded impact. To grapple with the reputational dimension and deploy resources at scale, a decisive pivot is needed.
Existing policy organisations would not be the main drivers of this process. Some are positioned to pivot, and pivot they should – including by clearly breaking with their past choices and proposals. The world is changing rapidly, and anyone who says ‘we get it now’ and starts pulling for the sovereign fate of middle powers will be welcomed. That said, the safety movement has gotten into trouble for quick pivots and perceived two-facedness in the past, and it’s probably not realistic for deeply entrenched organisations to reverse course completely. In fact, it’s probably good if some safety organisations with strong positions on this stay where they are: to catch the true believers in global-mission thinking about safety in middle powers, and to promote internal disagreement. It’s probably good if there are organisations to which new middle power orgs can point and say ‘we strongly disagree with them, and we represent a different approach to international safety policy’.
The research ecosystem can do more. Established scholars already work closely on national sovereignty, but until recently, their work on strategic questions has always been wrapped in layers of interpretation that contextualised it with some development-focused AI safety point. Giving these researchers freedom and encouragement to pursue these questions without a predetermined takeaway, suggesting that whatever strategic pathway they find is of value, could unlock substantial resources. These people have the ideas – make it clear it’s part of the mission and not taboo to get them out there.
But the even more promising aspect of the research ecosystem is talent pipelines. Until recently, major safety-aligned mentorship programs have mostly focused their project selection in international governance on this development-focused approach. If these programs offered a starting point for middle-power-focused researchers, I’m confident more people would choose this path. The number of bright, ambitious people who understand AI and want to help their home countries, but end up defecting to US-focused work or contorting into development-focused safety work, is staggeringly high. Give them a home.
Much of this comes down to funding. Major funders could publicly prioritise these cause areas; launch RFPs around them; and incubate organisations that tackle them. In the most ambitious version, they apply the same strategy to the middle power space that they have to the national security conversation, where safety-aligned organisations have made keystone grants to top-tier institutions and cultivated deep expertise between old-school policy hands and capable, entrepreneurial safety advocates who brought cutting-edge knowledge to enrich the discussion. I’d welcome the same for middle power work. Right now, there simply isn’t enough space to do work that grapples with AI and its strategic implications for middle powers. It’s in the safety movement’s interest to step up, and the world would be thankful for it.
The safety movement has the people, the institutions, and the resources. What it lacks is the right theory of change for middle powers. The development-focused approach was always a long shot; today it’s actively harmful. The alternative – helping middle powers navigate AI deployment, build resilience, and avoid strategic blunders – is tractable, neglected, and would actually advance safety. The moment for that is now. Seize it with haste.
By which I mean the broad AI safety ecosystem – funders, researchers, policy organisations that primarily focus on substantial and catastrophic risks from very advanced AI systems. It pains me to say that this polemic provides a good overview.
Some safety advocates will say this was them making the best of a bad situation – that the AI Act would have happened anyways, and they sought to improve it. I doubt this, both because safety advocates played a substantial role in passing it and because their opposition would surely have been counterfactually influential – but it matters little for the forward-looking evaluation.
Yes, I know that some safety advocates believe the reduction of existential risk is also the primary national priority of any smaller government. But even if that’s true, no one’s gotten very far in selling it.