Homeostatic AI Progress
Very fast AI progress faces a self-regulating dynamic: Greater speed leads to more organizational strains, compute trade-offs and societal backlash.
This post is a ‘dispatch’, intended to be shorter & more directly related to current events than my essays. Do let me know if you have thoughts on how to prioritise between these!
AI progress is notoriously difficult to predict — but that doesn’t stop anyone from trying. Recent contributions, especially the in-depth forecast AI 2027, have revitalised interest in debating AI timelines. A big part of these discussions relates to technical considerations, to which I have very little to contribute. However, an increasingly larger part relates to the role of frictions – both economic and societal – to the deployment of advanced AI. Understanding these frictions becomes ever more important as AI diffusion picks up, and I believe current debate misses an important nuance in assessing the force of these frictions.
The faster AI capabilities increase and the faster advanced AI capabilities are deployed throughout society, the greater societal and economic frictions become. These deployment frictions then act on development speeds as well, lengthening iteration phases and reducing economic and political incentive for further AI progress. As a result, AI progress might be homeostatic: Breakout development speeds would be reined in by escalating deployment frictions.
There are three core factors contributing to this effect: Changes in the dynamic of AI developers, compute trade-offs between development & deployment, and societal backlash. These factors matter on two levels: Because they might explain how predictions could be off even if they got the technical elements right; or because they might serve action-guiding to those trying to steer the effect of frictions on AI progress.
AI Developer Overextensions
The first relevant friction is coming from inside the labs. If deployment goes very fast, AI developers will have to accommodate the strains that come with it. So far, it seems like developers will bear the brunt of the cost and overhead of deploying AI systems. For all the facetious discussion around ‘GPUs melting’, a business reality underlies these dynamics: If a company like OpenAI wants to capitalise on rapid subscriber growth from new viral releases like GPT-4o’s image generation feature, it needs to allocate a substantial amount of its resources. And it needs to capitalise on just these moments to sustain interest, funding and support for its future growth.
One particularly pervasive effect of that incentive relates to organisational make-up. Providing a very popular software product requires a very different kind of organisation than researching new technology – a larger organisation, with a lot of people working on front-end, reliability, marketing, legal issues, and ultimately on administering all the people working on all these issues. Both leadership focus and company culture are ultimately affected by these trends. Ask anyone who has seen both the culture of 2020s Google and of a highly effective Silicon Valley start-up from the inside, and they’ll concur that product-oriented corporate structures can have paralysing effect on innovation. The faster these organisational shifts go, the less likely they are to go smoothly: There’s a big difference between marginally upscaling your workforce or customer base and multiplying it over short timeframes. It’s hard to overstate just how high the organisational strain is that timelines like AI 2027 would place on developers: It’d have a company like OpenAI grow from a remarkable start-up to a world-defining enterprise in less than 2 years.
Amidst that growth, developers would need to retain their innovative magic: The current steep trajectory of AI is the consequence of truly impressive technical innovation driven by a select few organisations that caught lightning in a bottle, and its breakneck speed has only been sustained by continued output of major breakthroughs. Even if you think the path to human-level AI mostly runs through scaling up compute, leveraging that compute still requires innovation – just take early last year’s problems with GPT-4.5-tier pre-training scaling that got bailed out by inference scaling taking off. Neither Bell Labs nor RAND’s glory days lasted – despite many great innovations. The time of hyperinnovative AI labs, too, could pass.
For a while, it seemed like this effect could be staved off: through either disruption of incumbents, nationalisation, or division of labor. I think all are unlikely:
As AI developers structures ossify under the pressures of driving deployment, it still remains unlikely that they’ll be dethroned quickly by development-driven smaller actors. A number of trends entrenches the current market leaders in their position: Privileged, contractually guaranteed access to the very limited global supply of premium compute; integrations into major software platforms; and soon, privileged consideration by the US government in its pursuit to securitise AI development. So even despite the structural slow-downs that labs might experience, they would still be the ones driving at least the domestic frontier.
Division of labor is unlikely, too: A popular theory circa 2023 held that the ultimate deployment of advanced AI might mostly run through separate businesses (often derisively called wrappers). But leading AI developers are working to counteract that trend: they are all positioning not just as model developers, but as providers of integrated systems, thereby creating more durable moats. A side effect of that decision is that the very same organisations that develop top AI models will continue to be burdened with the tricky tradeoffs and organisational overhead from being deployers of AI-powered systems.
Lastly, greater government oversight up to nationalisation is unlikely to remedy the effect: Yes, national governments could decide to come in and force heavy focus on innovation again. I’ll comment on nationalisation generally below, but I’ll say here that that would probably be the first time government involvement made a process more culturally friendly to innovation. If driving AI progress remains an issue of genuine innovation, government intervention might not be too helpful.
As a result, an increase in deployment and consumer uptake of AI systems directly results in costs to the development speeds at the frontier. How large is that effect? It’s hard to say. I believe this is the least of the three effects I list, but it could still be meaningful: Genuine innovation is difficult to force, and often dependent on fickle sets of cultural factors. The faster AI progress goes, the greater the danger to that culture.
Development/Deployment-Tradeoffs
The second escalating friction comes from hard compute limits that are determined years before the bottleneck occurs. That forces a trade-off: The more impressive AI progress, the more of that compute will be required for running AI systems. My core underlying assumption here is that the next years will continue to see demand outpacing supply for compute both in running top models and developing at the frontier – I write more on others’ thoughts that AI progress might break free of compute constraints further below.
As long as compute constraints remain, I believe capacity is unlikely to catch up – both due to strong empirical work on that topic, and by more general principle: I believe the actors involved in widening the bottlenecks to compute supply are still much more skeptical of the future demand than is appropriate, specifically those involved in the decisions around licensing power supply and scaling up semiconductor fabrication. And even if they weren’t, upscaling would take time – between the volatility of the current overreliance on TSMC’s Taiwan production and the time it took for the Arizona plant to get down to just-below-SOTA fabrication levels, supply doesn’t seem set to keep up with demand.
In that compute-constrained world, deployment and development trade off directly. That starts as a lab-internal effect, where deployment and product improvements eat into (compute) resources otherwise earmarked for development: Past media reports have already hinted at the discontent other allocation conflicts have caused among research groups at major developers. But it goes beyond lab interna: If AI development arrives at economically or strategically highly valuable capabilities fast, there will be a lot of demand for compute to deploy these capabilities throughout the economy. As a result, developers will be forced to assign their internal compute budgets to inference, prices for renting top-tier chips will generally go up – making it harder to secure more compute for whatever development project you might be pursuing, whether that’s scaling up your current training run or founding a new developer.
Put a bit more simply: Because expansion takes time, the amount of available compute in 2027 is mostly determined today. If AI progress goes very fast until then, that means a lot of this compute will be needed to run valuable AI systems – so there’s less compute to further push ahead development instead.
Is this effect offset by increasing national involvement in frontier development– e.g. because the US government, caught in a race with China, will deprioritise deployment and mandate instead that 2027’s compute is always predominantly spent on development? This is what I take AI 2027 to imply, but I think it’s unlikely: the USG is also subject to this deployment-versus-development dilemma. If powerful AI systems by 2027 (or, analogously, at any other compute-constrained point in time) are really promising and effective enough to motivate meaningful government involvement to push them further and further, they are also effective enough to warrant widespread deployment. I do not think the USG shares the ‘Wunderwaffe’ aspirations of some technologists: If the system works to make the military more effective, even a maximally AI-enthusiastic USG will want to spend meaningful compute to deploy it throughout – if only because of the threat of China pivoting from development to deployment and using the resulting window of having a supercharged force while the US is still prioritising development. So even the central coordination function of government involvement does not solve the homeostatic incentive to allocate more compute to deployment.
Addendum: Or might efficiency gains offset this effect? I have two reasons to ultimately be skeptical: First, because efficiency gains have so far occurred below the frontier. o3-mini is more efficient than o1-pro, but o1-pro is better and still highly demanding on compute. Wherever AI is deployed for critical applications, whether that’s military or central economic roles, there will be strong incentives to use whatever’s best – lest your model loses to others that spend big to do so. And second, because even efficiency gains along the lines of 3x price reduction per year only do so much to offset radical growth effects. Naively, an agentic system improving to be coherent twice as long and being used twice as much quadruples compute – and many predictions see much greater deployment expansions within a given year.
Fear Of The Reaper
People will desperately want rapid AI progress to slow down once it hits them hard. I’ve written extensively about labor market impacts before, but that’s not all there is on backlash: A lot of the deployment avenues for tomorrow’s frontier systems will feel fundamentally alien and scary to people. They’ll see automated systems doing things they thought squarely in the realm of humans – art, genuine interaction, their own chosen trades and passions. They’ll find themselves interacting with AI systems more and more, and they might perceive that interaction as zero-sum: as if someone had taken an important thing from them. Estrangement, fear and protest will be a natural reaction. Even more, these feelings will be fertile breeding ground for politicisation: Any populist message yearns for that kind of emotional turmoil, and there will be plenty of politics to be made in rallying people around an anti-AI message.
This effect is much stronger if timelines are shorter and development speed is faster. There is a frog-boiling effect that stops working at a certain pace of radical change. Just compare the automation of weaving to the automation of agriculture: The former came a lot more sudden, and the shock and backlash was enormous; labor displacements, but also elite craftsmen driven by the devaluation of their work, led to violent uprisings and lasting political repercussions; while the latter was slow-rolled to an extent that allowed for institutional adaptation, rural-urban migration and reemployment and ultimately cushioned backlash. So the faster your development speeds look, the less you should expect institutions to be able to keep up – and the more you should expect violent backlash that leads to crackdowns and slowdowns, in turn leading to less deployment and slower development.
AI 2027 might pose a contention: That major governments will be in a tight race that leads them to ignore this sort of effect. Some of this recurs to the other aspects I’ve responded to above; it’s easier to insulate development progress if it became more and more divorced from private companies and deployment incentives, which I argued is unlikely. But for some further response: AI 2027 argues that by the release of a cheap remote worker, the US President will have bought into AI progress so much as to ignore political pleas to regulate. I think that mischaracterizes the incentives that act on leading politicians – especially the current administration. As I’ve written elsewhere, the administration’s message of job-focused economic populism, its commitment to previous idiosyncratic policy positions, and the general drive of political incentives make it extremely likely that they’d crack down. And if they did not, all the more political power would be up for grabs to whomever decided to advance AI regulation instead; whether that’s 2028’s Democratic candidate or one of many State governments. I think it’s very unlikely that politics would just let this disruption happen.
Ways Out?
There are some conceivable ways to short-circuit the feedback loops I discuss, but I don’t think any of them are particularly likely trajectories.
One is AI-enabled acceleration of development speeds. This seems to be the ‘way out’ that some leading AI developers and some of the more aggressive predictions favour: They put a lot of emphasis on the possibility of AI-assisted research leading to meaningful further acceleration. I’m skeptical: First, it seems not obvious to me that easily scalable human-level research engineer ability is currently a major constraint – there are just too many groups of above average software engineers, including at major tier-2 developers, that haven’t made any major strides in the last years. And second, even past breakthroughs remained somewhat in line with the predictions of the Bitter Lesson, introducing another axis of compute scaling just as pre-training scaling was not hitting a wall, but still facing some real barriers on its way to robust agent capabilities. Many short-circuits that could arise from that ability might still face real-world bottlenecks like compute, and so ultimately face the same frictions I describe above. Nevertheless, there are some well-argued reasons to believe in software-driven intelligence explosions; and if you agree with them, you might not find my model particularly persuasive.
Another is AI-powered widening of bottlenecks, e.g. dramatically improved chip manufacturing or AI-supported institutions managing transitions better. This strikes me as more plausible than getting rid of compute constraints altogether in many ways; the world has seen massive increases in manufacturing output – such as during the first phases of automation – and massive increases in state capacity – such as during the rise of the administrative state or of telecommunications – before. It seems in principle quite possible that AI progress could have effects of the same magnitude, thereby alleviating compute bottlenecks or guiding people through the transition in a way that massively reduces political upheaval. There is good normative work on these matters, but I have read few convincing contributions on how likely and how soon AI-supported bottleneck widening might happen, leaving me at ‘this might be possible, but it would be really hard and it could take a while’ – so I’ll leave my assessment of this as a hypothetical: If I turn out to be wrong, I think this AI-powered widening would be a likely reason why.
The third is development divorcing from deployment: AI developers and their government overlords might cease to roll out capabilities, foregoing much public reaction and interaction, and just ceaselessly push the frontier. But fast, successful deployment will be absolutely necessary to maintain the level of investment into compute, talent and energy that the current trajectory has required and will continue to require: In the world of markets and politics, too, extraordinary claims require extraordinary evidence. Right now, big tech companies and governments alike are taking a gamble on AI predictions – the former by investing, the latter by lending regulatory support. Progress toward superintelligence will require these investments to grow and grow, further and further increasing the burden of justification. Soon, AI developers will be asking for investments that make up sizable parts of major countries’ GDPs. For that money, you’ll have to prove AI’s real-world effects — through deployment at scale. I don’t think marginal contributions to secret cyberwarfare programs or military coordination projects alone will suffice.
Finally, a frequently underrated mechanism is proliferation pathways bypassing usual channels – like when graphic designers are cut out the conversation and image generation happens via a direct consumer technology. It’s a well taken point, but I believe it reaches its limits when more central parts of society are affected. As soon as higher stakes are involved – real money, big decisions, health outcomes –, institutional trust enjoys a much higher premium. That’s the case for many big-ticket AI impacts: You want AI outcomes and contributions to be considered reliable and trustworthy, and that might either happen through a long build-up of trust in new pathways – like when early conceptions of the internet as a wild west were supplanted by considering it a host of real information – or through deployment running through slow-moving present institutions, from companies to governments. Both of these take time, and the latter come with a great deal of friction and resistance.
Directional Updates
AI progress past a certain point might be homeostatic: Whenever the pace of development and deployment increases, so do frictions to deployment that ultimately rein the pace back in. I think this effect is robust across a fairly broad range of characterisations of future AI progress; and applies specifically where predictions allege large jumps in capabilities or require rapid AI diffusion. It’s not intended as a response to any one prediction, but as one factor I wish many more of these predictions priced in. It also has quite a lot of bearing on policy: It means that even the most fervent accelerationist has an interest in moderating the pace of the AI revolution, if only to smooth over some of the most egregious potential sources of backlash that could derail its promise altogether.
All in all, my argument is not to say that you should ‘lengthen your timelines’ by a fixed amount of time; but that you should keep Amontons’ First Law in mind: The more load your predicted AI capabilities place on the fabric of our current society, the greater will be the frictions that slow their further growth.