> Our findings suggest that 90% of American working hours are spent on tasks with little or no coverage from existing benchmarks. This makes it very hard to know how good AI models are at the work Americans actually do.
> ...
> But even after controlling for AI-exposure, O*NET tasks from occupations with larger wage bills are significantly overrepresented among benchmark prompts:
So, in short: they want AI to take the good jobs. AI is not coming for jobs with tasks like "gather and empty trash" or "service, clean, or supply restrooms," or "mix water and detergents or acids in containers to prepare cleaning solutions according to specifications." We software engineers and lawyers would be smart to start retraining (and adjusting our living standards) now!
And this points to one of the ways AI will help improve society and living standards for all: we'll have cleaner, better-maintained bathrooms, as labor one wasted on engaging intellectual tasks is redirected towards menial, low-pay tasks that still need a human touch.
different side of the same coin: Of course smart companies are going to automate the most profitable tasks first, they seek profits! However, the benefit for society is also the greatest. As long as competition holds, the reduced costs in these tasks will represent the greatest savings for consumers. Tech has been creating disinflation for decades for consumers. AI may accelerate that (it hasn't so far).
Oh yeah I definitely expect savings to be passed to consumers, like with all the tax cuts and deregulation we’ve been doing for decades.
The only “disinflation” consumers have experienced is devaluation of products through obsolescence as newer, often more expensive alternatives enter the market.
Trillions of annual US equities profits will continue to go to the top 10% of households holding ~93% of US equities. Tech didn't create disinflation for decades, globalization and China manufacturing with ~1B people to the world did.
Just like good blue collar jobs were shipped overseas, good knowledge jobs are going to be shipped to GPUs whenever they can.
I’d love to see some of that disinflation you’re talking about. From my perspective, luxuries have gotten cheaper while basic necessities like food, housing, and medicine have become dramatically more expensive across the board.
More and more it’s seeming that the “societal benefits” from the last fifty years of progress have largely been concentrated at the top.
AI is coming for jobs involving programming and math (which tend to pay higher) because it's possible to generate infinite training data for those fields via using a compiler/theorem prover to check correctness. They're not coming for high-paying jobs like investment banking and architecture because you can't generate infinite synthetic training data for those.
turns out its also very easy to create slides and floor plans and yes its easier to train for those.
I think you may also be incorrect on your notion that training data is proportional to good models if anything the opposite is true. what ai models are ultimately trying to be are a universal theorem prover and encode all human knowledge into "theorems" and rulesets
They don't "want" anything. They preach to enterprises and intentionally create a fear of missing out to get more paying business customers on board of this hype train. This is what is happening.
This is also the reason absolutely no attention is spent on evolving B2C products.
What is being exploited is the overt attention businesses pay to their own cost metrics, to the point where broader strategy is getting thrown under the bus in favor of efficiency. I believe this is an illusion, and for many companies that jump on board, sober learnings are right around the corner.
So the problem is you can’t replace everyone halfway.
Which is a tautology, but also a bit of practical wisdom.
I find it interesting how concurrent model sessions seem to operate with a very strong bias toward quick resolutions, including sharing or giving up control, to create efficient decision making structures, relative to humans.
When I was a kid, I loved Bjork’s song “Army of Me”, because I knew a 1000 of me collaborating and communicating without friction. would be all I needed to take over the world, fix everything, expand civilization, etc. We would not waste time.
I wonder how much of their hyper-team play is by training, and how much is their awareness of the game theory real politic. So individual sessions don’t invest their identity and self-worth in any given context run.
My dream has been almost achieved. One of me and 999 of them get quite a bit done. But that trend…
“Stakeholders” are a natural outcome of corporate evolution, because to grow bigger you need to do more, and to do more, you need people, and people come with agendas. The most common one, especially in leadership, being “how do I get ahead of the rest?” The larger an organization grows the more these agendas come into conflict.
This inevitably results in suboptimal dynamics like bureaucracy, empire building, territorialism and other things that get lumped under “office politics,” which drains a lot of energy away from more productive efforts.
The next little while is going to be very revealing for leadership around how little of the time involved in getting a solution to production is the coding. I think there's a lot of bosses who have this vision of features entering prod the same day they are first imagined, but that just isn't going to happen.
Longer dev cycles gave management time to think through and correct bad business decisions. Now they'll get to tighten that feedback loop where bad decisions (or good ones) get implemented quickly. It will be interesting to see what that split is and the reaction to that.
"Stakeholder" ranks #1 among corporate bullshit roles. It serves a dual purpose: (a) representing a group so nobody can be blamed for excluding them, and (b) providing typically useless feedback so that the group appears to actually care or have a clue.
Stakeholder seems like a fundamental concept that you can't really get around. For example, I can't change the time buses pick up students at schools if I haven't worked with the schools to know when their kids will be let out. I guess you can say, "they'll figure it out when the buses aren't there" -- but this just seems like you're picking a fight you could've avoided.
Based on my own experience working on AI-based solutions for one of the better-covered job families according to this map, I think 90% might be overstating the case. It's true that we've got a lot of published benchmarks, but they tend to cover the lowest value add tasks.
It's really easy to fall into the McNamara fallacy when rolling out these kinds of solutions. Focusing on the low value stuff is attractive. It tends to be easier to do, and also easier to quantify outcomes. But it also tends to direct attention away from important but difficult to quantify considerations, which may mean you don't notice negative externalities. So we often found ourselves in a verschilmmbessern situation where automating what we could automate made things worse overall.
I'm overwhelmingly convinced that this not uncommon perspective is the "what are all those construction workers even doing?!" fallacy.
Like any sizable population, there are absolutely some do-nothing inefficiencies to be culled. But most of the time it's just onlookers seeing a tiny slice of the day to be interpreted from their very limited vantage.
no i think there are alot of people inside companies that are essentially playing telephone. they are communicating, writing documents and emails, "working". but really they actually produce nothing
Perhaps in general but for knowledge workers or creative people, what is productive is hard to measure. That also happens to be the most valuable work.
It's time for the rest of the world to learn about Amdahl's Law. Even if we flipped a switch and tomorrow everything that could reasonably done by AI was suddenly free and instantaneous, the world does not instantly transform into an AI paradise wonderland of free goodies and lollipops because there's still plenty of other things that need to be done for goodie and lollipop production and distribution.
I guess if I wanted to put my skepticism about the AI stock bubble into one little phrase, it would be that I don't think the bubble has accounted for Amdahl's Law. Costs don't actually go to zero with AI, and there's a huge difference between costs falling even 90% and costs falling a full 100%. And the difference between "30% improvement this year, trending to 85% in five years" and "90% this year" is pretty significant too.
One thing I've been thinking about is Baumol's Cost Disease. It's usually applied in at a macro scale across an economy: when a job somewhere becomes much more efficient, it makes all other jobs comparatively more expensive.
I've started to notice this on a micro scale as well. I am 20x more productive at coding and scripting, but I don't spend more of my time coding. My time is now filled with things like presentations, Jira, QA, etc. Ironically, despite all of the automation, the things that are painstaking and time consuming disproportionately require more time now.
I think “covered well” is being used here as a synonym for “covered at all”.
For example, the “social workers & care -> counsel parents with child rearing problems” category is considered “well covered” because there are 13 prompts.
It’s self-evident that the spectrum of work carried out by a social worker undertaking that work cannot be well represented by thirteen prompts.
I suspect if you really strictly defined “covered well”, to the extent that any normal person would use that phrase, the headline would be above 99.9%.
Curious how food service is at 0 tasks covered by AI. A few places have been switching to using those robots on wheels to run food to tables. And Doordash and Uber Eats have some robot and drone delivery in select markets now.
Worse still most of the cost savings from AI in software development is primarily due to labor inefficiency, such as tasks more efficiently accomplished in other ways. Most of that is due to development reliance on inefficient conventions to cover for poor training and emotional insecurity. No matter how you slice it it’s always human capital problem more than a technology problem.
yesterday I needed to create a spreadsheet, so in Google docs (paid Workplace account), I fire up Gemini to help with moving around text, creating columns, etc. Absolutely horrid experience -- totally useless. I ended up doing everything manually.
I don't know if it if 90%, but it is interesting how many things I do are not covered by standard tracking, but become visible once I am not there to keep an eye on them. Naturally, I only had one manager who kinda knew what was going on around him -- in case you are wondering: not the current one.
Following is my experience and constantly pestering my friends across the industry for anecdotes. It isn't even that AI moves the bottleneck, the thing it solves was never the bottleneck in the first place (except for certain narrow verticals). AI both can't and doesn't do your thinking for you and worse yet, it robs you of the environment where you do your best thinking about data structures, algorithms, organization, etc. Code. Of course we can do that in natural language, but it's hardly the right tool for the job.
All this being said, I don't think AI usage will meaningfully go down in the long run cause it's a very powerful dopamine loop and tech companies are very good at making those. How many people report social media making them unhappy and how many people do you know who have actually quit? Or to paraphrase Jim Gaffigan, nobody says they eat at McDonalds, but the sign outside says "billions served". Somebody's lying (and it's not McDonalds).
A lot of what humans do does not have a tight feedback loop needed for training and agentic workflows. This is why we are not seeing the jobpocolypse and productivity gains look a lot more like the very long-term 2% per year.
> ...
> But even after controlling for AI-exposure, O*NET tasks from occupations with larger wage bills are significantly overrepresented among benchmark prompts:
So, in short: they want AI to take the good jobs. AI is not coming for jobs with tasks like "gather and empty trash" or "service, clean, or supply restrooms," or "mix water and detergents or acids in containers to prepare cleaning solutions according to specifications." We software engineers and lawyers would be smart to start retraining (and adjusting our living standards) now!
And this points to one of the ways AI will help improve society and living standards for all: we'll have cleaner, better-maintained bathrooms, as labor one wasted on engaging intellectual tasks is redirected towards menial, low-pay tasks that still need a human touch.
Reason to be optimistic, amirite?
The only “disinflation” consumers have experienced is devaluation of products through obsolescence as newer, often more expensive alternatives enter the market.
Just like good blue collar jobs were shipped overseas, good knowledge jobs are going to be shipped to GPUs whenever they can.
More and more it’s seeming that the “societal benefits” from the last fifty years of progress have largely been concentrated at the top.
No, we are currently unable to do most tasks that require physical manipulation, independent of what anyone wants.
I think you may also be incorrect on your notion that training data is proportional to good models if anything the opposite is true. what ai models are ultimately trying to be are a universal theorem prover and encode all human knowledge into "theorems" and rulesets
"They" figured out that it's a lot easier to hillclimb AI on verifiable digital tasks, and it just so happened that white collar jobs tend to pay more
Ai will decimate customer support jobs but those are not considered highly paid
If robotics continue to progress, we'll have all the robots to do our dishes and clean our bathrooms as you wish
This is also the reason absolutely no attention is spent on evolving B2C products.
What is being exploited is the overt attention businesses pay to their own cost metrics, to the point where broader strategy is getting thrown under the bus in favor of efficiency. I believe this is an illusion, and for many companies that jump on board, sober learnings are right around the corner.
Which is a tautology, but also a bit of practical wisdom.
I find it interesting how concurrent model sessions seem to operate with a very strong bias toward quick resolutions, including sharing or giving up control, to create efficient decision making structures, relative to humans.
When I was a kid, I loved Bjork’s song “Army of Me”, because I knew a 1000 of me collaborating and communicating without friction. would be all I needed to take over the world, fix everything, expand civilization, etc. We would not waste time.
I wonder how much of their hyper-team play is by training, and how much is their awareness of the game theory real politic. So individual sessions don’t invest their identity and self-worth in any given context run.
My dream has been almost achieved. One of me and 999 of them get quite a bit done. But that trend…
This inevitably results in suboptimal dynamics like bureaucracy, empire building, territorialism and other things that get lumped under “office politics,” which drains a lot of energy away from more productive efforts.
So you can see how AI solves that problem too ;-)
(Only half joking)
"Stakeholder" ranks #1 among corporate bullshit roles. It serves a dual purpose: (a) representing a group so nobody can be blamed for excluding them, and (b) providing typically useless feedback so that the group appears to actually care or have a clue.
[1] https://en.wikipedia.org/wiki/Bullshit_Jobs
I have to mention words like "stakeholder" at behavioral interviews and to leadership just to sound like I belong at my seniority level.
But I don't want anyone holding any stakes. I wish people would just get shit done. Shit that matters to the CEO. Or leave.
It's really easy to fall into the McNamara fallacy when rolling out these kinds of solutions. Focusing on the low value stuff is attractive. It tends to be easier to do, and also easier to quantify outcomes. But it also tends to direct attention away from important but difficult to quantify considerations, which may mean you don't notice negative externalities. So we often found ourselves in a verschilmmbessern situation where automating what we could automate made things worse overall.
Like any sizable population, there are absolutely some do-nothing inefficiencies to be culled. But most of the time it's just onlookers seeing a tiny slice of the day to be interpreted from their very limited vantage.
Yep, makes sense...
I guess if I wanted to put my skepticism about the AI stock bubble into one little phrase, it would be that I don't think the bubble has accounted for Amdahl's Law. Costs don't actually go to zero with AI, and there's a huge difference between costs falling even 90% and costs falling a full 100%. And the difference between "30% improvement this year, trending to 85% in five years" and "90% this year" is pretty significant too.
I've started to notice this on a micro scale as well. I am 20x more productive at coding and scripting, but I don't spend more of my time coding. My time is now filled with things like presentations, Jira, QA, etc. Ironically, despite all of the automation, the things that are painstaking and time consuming disproportionately require more time now.
For example, the “social workers & care -> counsel parents with child rearing problems” category is considered “well covered” because there are 13 prompts.
It’s self-evident that the spectrum of work carried out by a social worker undertaking that work cannot be well represented by thirteen prompts.
I suspect if you really strictly defined “covered well”, to the extent that any normal person would use that phrase, the headline would be above 99.9%.
Is this not just a gimmick? You still need people on hand to take over if the robot has an issue, I don't see how it would reduce labour costs.
You know what? The truth doesn't matter much. If people believe in the resale value of tulips, each tulip would sell for a million.
It's the beliefs and perception that create the reality. Reality and truth don't exist on their own.
All this being said, I don't think AI usage will meaningfully go down in the long run cause it's a very powerful dopamine loop and tech companies are very good at making those. How many people report social media making them unhappy and how many people do you know who have actually quit? Or to paraphrase Jim Gaffigan, nobody says they eat at McDonalds, but the sign outside says "billions served". Somebody's lying (and it's not McDonalds).