Thanks for the essay! Great to have clarity on where we agree and disagree.
What should we track in order to know if we're getting close to Phase 2? I agree with your point that we need more fine-grained automation data, but even if we had full visibility into what's happening, what exactly are the relevant indicators?
IMO we should specifically look at deployments of relatively general-purpose AI systems operating with minimal human supervision to handle tasks with high costs of errors. All three factors are important; I don't think we can generalize from e.g. low-stakes to high-stakes deployments. https://substack.com/@aisnakeoil/note/c-133518692
Currently there seem to be ~0 such deployments, and my prediction is that in 2030 it will still be ~0 (compared to the size of the economy).
First, just to clarify, I'm not sure if I'd want you to read the view I describe as 'normal automation' as meaning the same thing as your notion of 'normal technology'; I feel like there are plenty of deployments beyond the scope of normal automation that still fit your normal tech framework.
Second, how do we know? I feel like that depends a little on which necessary attributes we maximise first. One model might be something like: Agents turn out to be generally coherent and error-free at some point, but aren't good enough at their tasks yet; and so phase 2 happens as we build RL environments or similar for each domain and then they get good enough. In that world, we'd just see local automation pressures in one sector, then another, then another.
The other model, which seems a bit more likely to me right now, is: Agents get really good at things pretty fast, but their horizons vary, and their reliability stays spotty. You might think we face some kind of 'capability overhang' in that setting, where marginal breakthroughs in agent infrastructure/scaffolding/reliability suddenly push agents into automation-worthy viability. I'm much less sure what's a good way to predict that, because I agree with you that generalising from low-stakes to high-stakes deployments is fraught, and we might overrate how informative low-stakes deployments are.
So far for the frictionless models. I think real-world frictions make things easier to predict: Different economic sectors will be more or less ready for change, more or less entrenched, get more or less political protection from displacement, etc etc. So even if we get agents that outperform human workers, I feel like real-world diffusion will still take a long time, which makes my 'automation timelines' much longer than my somewhat optimistic (and less well-informed) view of purely technical tractability would imply. My sense is that you're right that there currently are no such deployments, and we probably won't be deep into phase 2 in 2030 either, though I'm a little bit more optimistic that some narrow, very learnable tasks in sectors with risk-happy leadership and little regulatory constraints would already be very agent-driven; SWE comes to mind as an example.
This friction-rich automation would definitely show up in automation data as it happens - less junior hiring, sector-specific high unemployment, growing general unemployment, wage depression for some task profiles. Ideally, we'd also look at what you describe, i.e. deployments of autonomous systems in high-stakes cases, but I suspect we'll never quite get to a point where self-reports and transparency are good enough to assess this in real time despite labs' incentives to overreport. Figuring out how exactly to determine relevant indicators feels like a tricky triangulation between what data will be available and what part of it actually seems useful. Sorry to say I don't have a satisfactory answer to that yet, but getting all the data we can seems like a robust first step.
Mosty I do agree, but FWIW "wages grow during the initial period but then collapse" is the standard trajectory in econ models of the situation (eg Korinek and Suh (2024). Scenarios for the Transition to AGI) and is also what we mention in Gradual Disempowerment. In this sense the "two competing perspectives" "AI Snake Oil" vs. "Intelligence Curse" seems like a more nuanced existing understanding was partially replaced by oversimplified "x vs. y" takes.
Thanks. I do agree that a lot of the scholarly work on this, and some of the more sophisticated scenario-building, does well to reflect that nuance. I think policy advocacy and political communication around this is not at that level yet, which is mostly what I'm referring to - but I'll make sure to clarify that!
Thanks for this! The two phase framing is logical and clarifying.
Absent redistributive policy, it seems like the human welfare implications of phase 2 are sensitive to how rich society is. At some threshold, if there is enough wealth being generated, the ultra-rich might donate enough money to support displaced human workers. In that case, the loss of market income might not severely reduce living standards. For example, Bill Gates would willingly sacrifice a lot of his consumption to take care of the rest of humanity.
That might seem absurdly optimistic and maybe it is. But by phase 2, I would think we would have far greater wealth/income than we currently do and far more concentration that we currently do, such that it's hard to think about without a model. In that scenario, welfare depends on:
Thanks for the essay! Great to have clarity on where we agree and disagree.
What should we track in order to know if we're getting close to Phase 2? I agree with your point that we need more fine-grained automation data, but even if we had full visibility into what's happening, what exactly are the relevant indicators?
IMO we should specifically look at deployments of relatively general-purpose AI systems operating with minimal human supervision to handle tasks with high costs of errors. All three factors are important; I don't think we can generalize from e.g. low-stakes to high-stakes deployments. https://substack.com/@aisnakeoil/note/c-133518692
Currently there seem to be ~0 such deployments, and my prediction is that in 2030 it will still be ~0 (compared to the size of the economy).
Thank you! Some thoughts:
First, just to clarify, I'm not sure if I'd want you to read the view I describe as 'normal automation' as meaning the same thing as your notion of 'normal technology'; I feel like there are plenty of deployments beyond the scope of normal automation that still fit your normal tech framework.
Second, how do we know? I feel like that depends a little on which necessary attributes we maximise first. One model might be something like: Agents turn out to be generally coherent and error-free at some point, but aren't good enough at their tasks yet; and so phase 2 happens as we build RL environments or similar for each domain and then they get good enough. In that world, we'd just see local automation pressures in one sector, then another, then another.
The other model, which seems a bit more likely to me right now, is: Agents get really good at things pretty fast, but their horizons vary, and their reliability stays spotty. You might think we face some kind of 'capability overhang' in that setting, where marginal breakthroughs in agent infrastructure/scaffolding/reliability suddenly push agents into automation-worthy viability. I'm much less sure what's a good way to predict that, because I agree with you that generalising from low-stakes to high-stakes deployments is fraught, and we might overrate how informative low-stakes deployments are.
So far for the frictionless models. I think real-world frictions make things easier to predict: Different economic sectors will be more or less ready for change, more or less entrenched, get more or less political protection from displacement, etc etc. So even if we get agents that outperform human workers, I feel like real-world diffusion will still take a long time, which makes my 'automation timelines' much longer than my somewhat optimistic (and less well-informed) view of purely technical tractability would imply. My sense is that you're right that there currently are no such deployments, and we probably won't be deep into phase 2 in 2030 either, though I'm a little bit more optimistic that some narrow, very learnable tasks in sectors with risk-happy leadership and little regulatory constraints would already be very agent-driven; SWE comes to mind as an example.
This friction-rich automation would definitely show up in automation data as it happens - less junior hiring, sector-specific high unemployment, growing general unemployment, wage depression for some task profiles. Ideally, we'd also look at what you describe, i.e. deployments of autonomous systems in high-stakes cases, but I suspect we'll never quite get to a point where self-reports and transparency are good enough to assess this in real time despite labs' incentives to overreport. Figuring out how exactly to determine relevant indicators feels like a tricky triangulation between what data will be available and what part of it actually seems useful. Sorry to say I don't have a satisfactory answer to that yet, but getting all the data we can seems like a robust first step.
Mosty I do agree, but FWIW "wages grow during the initial period but then collapse" is the standard trajectory in econ models of the situation (eg Korinek and Suh (2024). Scenarios for the Transition to AGI) and is also what we mention in Gradual Disempowerment. In this sense the "two competing perspectives" "AI Snake Oil" vs. "Intelligence Curse" seems like a more nuanced existing understanding was partially replaced by oversimplified "x vs. y" takes.
Thanks. I do agree that a lot of the scholarly work on this, and some of the more sophisticated scenario-building, does well to reflect that nuance. I think policy advocacy and political communication around this is not at that level yet, which is mostly what I'm referring to - but I'll make sure to clarify that!
Thanks for this! The two phase framing is logical and clarifying.
Absent redistributive policy, it seems like the human welfare implications of phase 2 are sensitive to how rich society is. At some threshold, if there is enough wealth being generated, the ultra-rich might donate enough money to support displaced human workers. In that case, the loss of market income might not severely reduce living standards. For example, Bill Gates would willingly sacrifice a lot of his consumption to take care of the rest of humanity.
That might seem absurdly optimistic and maybe it is. But by phase 2, I would think we would have far greater wealth/income than we currently do and far more concentration that we currently do, such that it's hard to think about without a model. In that scenario, welfare depends on:
1. The amount of redistribution (more is better)
2. The amount of resources (more is better)
3. The amount of concentration (ambiguous?)
4. The generousity of the rich (more is better)