The author uses a strange definition of teleoperation.
Following GPS directions is no different than assembling an Ikea furniture by following the instructions. So I guess Ikea has been teleoperating millions of humans for decades?
It's something that you'd pause and think about if you were trying to carefully craft a sensible argument. But the online attention economy now runs on AI op-eds. In this world of overabundance, the winning strategy is to produce outrageous takes and do it as quickly as possible.
I respect the author, but he succumbed to an... unnecessary temptation. There will be a dozen other "here's what AI means for humanity" posts on HN before the end of the week. For the human-written ones, I doubt that any of the authors will be proud of their analyses five years from now.
"So, teleoperation is probably very economically productive. Is it a good thing? I think mostly no, for several reasons." followed by several paragraphs discussing the negative consequences. This is celebrating?
Secondary reason: Oh, it will also make most people unable to have a living wage. We should probably do something about that, if it doesn't strain the economy too much.
The dehumanization of everyone who will still have a job? Nah, not a problem.
Though I think this dismisses the potential of people to organize, especially in jobs like those. This is why I'm skeptical that future will happen as easily as he is warning here.
Human extinction, mass starvation, and terrible jobs are all outcomes I think we should try hard to avoid. But I do think I have them in the right order!
One of the upshots of "pacing the frontier" is that we are all going to have to accept that compute will be a regulated, restricted resource. This includes both "how-much" restrictions (high-performance computers just won't be available to the public) and "what-kind" restrictions (software and hardware features will enforce policies on what software you may run, much like a game console or iPhone).
Back in the 2000s, one of my favorite programmers from the Before Times—Fabrice Bellard—came up with an algorithm for computing digits of pi that was so much better than the state of the art, he beat with a $3000 PC the record for speed of pi calculation, which had been established with a supercomputer.
Some bright spark—maybe they'll have Claude assist, maybe it'll trigger Claude's safety locks, I dunno—is going to derive a way to do transformer or other frontier AI inference in a really efficient manner on consumer hardware, making local models practical to run on the PCs people have today and nearly, if not actually, competitive with today's frontier models. "There is another theory which states that this has already happened"—perhaps several times already, and we just need a few more goes of the cycle. I find that for supposedly a crowd of the smartest dudes in the room, people in the Valley like to do things the stupid way around, by sheer fucking brute force: scale up massively, burn billions and trillions of VCbux on capex, aggressively conserve developer effort at the expense of everybody's CPU time and storage capacity—yours, your cloud provider's, your customers'... So I suspect, but at this time cannot prove, that there's a whole lot of headroom for optimization of AI inference, which has the potential to put some truly powerful models in the hands of ordinary people.
And that's very dangerous, because so much for pacing the frontier! If Joe Schmoe can build a mini-AI lab in his mancave, he can then gain access to a model that gives him all the big scary things Dario and Sam are worried about: instructions on how to build bombs or bioweapons, child porn generation, human sacrifice, cats living with dogs, mass hysteria! If you're concerned about an AI-mediated workplace panopticon, Joe Schmoe could, in principle, build one of those for his auto body shop with a local model. The only way to stop Joe from doing so is to stop Joe from acquiring the equipment it would take to run such a model.
Pacing the frontier is at least as much about pacing what you can compute as it is about what goes on in billion-dollar data centers. Any argument for limiting AI growth and development—and admittedly, there are some strong ones—is an argument against general purpose computing. So if we're really going to regulate AI, bend over for the laws that mandate OS/firmware/hardware-enforced restrictions on what binaries your computer may run. And so long open source. Oh well, those laws were coming anyway; that's what the age verification thing is all about. Society has determined that people can't be trusted with computers.
>Which is why I'm heartened to see a lot of support, including from the CEOs of Anthropic and OpenAI, for managing the pace at which these systems become increasingly capable.
This poor guy actually believes the PR, that the frontier model CEOs want to slow things down (and not just setup regulatory-enforced moats around their empires).
https://web.archive.org/web/20260725174355/https://marshallb...
https://news.ycombinator.com/item?id=41743688
Following GPS directions is no different than assembling an Ikea furniture by following the instructions. So I guess Ikea has been teleoperating millions of humans for decades?
I respect the author, but he succumbed to an... unnecessary temptation. There will be a dozen other "here's what AI means for humanity" posts on HN before the end of the week. For the human-written ones, I doubt that any of the authors will be proud of their analyses five years from now.
FYI, I don't think this is desirable. See the section of the post starting with "Is it a good thing? I think mostly no..."
> This technology allows employees to be easily slotted in and out of different jobs with very little friction, which destroys their bargaining power.
> AI permanently blacklists employees from every minimum-wage job just for being late a few times
> This can easily lead to a race to the bottom where companies no longer have to providing fulfilling, dignified jobs.
If I were stuck in this hell, I would gladly assist any superintelligent AI taking a swipe at the humans who put me in it.
...and of course he is a Rationalist/EA guy. Why am I not surprised?
I think the order of concerns is also telling.
Primary reason why it's bad: Increase risk of runaway superintellicence, XRisk, etc etc
Secondary reason: Oh, it will also make most people unable to have a living wage. We should probably do something about that, if it doesn't strain the economy too much.
The dehumanization of everyone who will still have a job? Nah, not a problem.
Though I think this dismisses the potential of people to organize, especially in jobs like those. This is why I'm skeptical that future will happen as easily as he is warning here.
Human extinction, mass starvation, and terrible jobs are all outcomes I think we should try hard to avoid. But I do think I have them in the right order!
This completely confused and misses the point of teleoperation vs following instructions/rules/protocols. Who set the GPS navigation destination?
Any sort of rule following is teleoperation by this definition then ⇒ the whole world is teleoperated.
Hospitals introduced intubation checklists to make sure protocols are followed and patients don't die. How about coding guidelines?
Back in the 2000s, one of my favorite programmers from the Before Times—Fabrice Bellard—came up with an algorithm for computing digits of pi that was so much better than the state of the art, he beat with a $3000 PC the record for speed of pi calculation, which had been established with a supercomputer.
Some bright spark—maybe they'll have Claude assist, maybe it'll trigger Claude's safety locks, I dunno—is going to derive a way to do transformer or other frontier AI inference in a really efficient manner on consumer hardware, making local models practical to run on the PCs people have today and nearly, if not actually, competitive with today's frontier models. "There is another theory which states that this has already happened"—perhaps several times already, and we just need a few more goes of the cycle. I find that for supposedly a crowd of the smartest dudes in the room, people in the Valley like to do things the stupid way around, by sheer fucking brute force: scale up massively, burn billions and trillions of VCbux on capex, aggressively conserve developer effort at the expense of everybody's CPU time and storage capacity—yours, your cloud provider's, your customers'... So I suspect, but at this time cannot prove, that there's a whole lot of headroom for optimization of AI inference, which has the potential to put some truly powerful models in the hands of ordinary people.
And that's very dangerous, because so much for pacing the frontier! If Joe Schmoe can build a mini-AI lab in his mancave, he can then gain access to a model that gives him all the big scary things Dario and Sam are worried about: instructions on how to build bombs or bioweapons, child porn generation, human sacrifice, cats living with dogs, mass hysteria! If you're concerned about an AI-mediated workplace panopticon, Joe Schmoe could, in principle, build one of those for his auto body shop with a local model. The only way to stop Joe from doing so is to stop Joe from acquiring the equipment it would take to run such a model.
Pacing the frontier is at least as much about pacing what you can compute as it is about what goes on in billion-dollar data centers. Any argument for limiting AI growth and development—and admittedly, there are some strong ones—is an argument against general purpose computing. So if we're really going to regulate AI, bend over for the laws that mandate OS/firmware/hardware-enforced restrictions on what binaries your computer may run. And so long open source. Oh well, those laws were coming anyway; that's what the age verification thing is all about. Society has determined that people can't be trusted with computers.
>Which is why I'm heartened to see a lot of support, including from the CEOs of Anthropic and OpenAI, for managing the pace at which these systems become increasingly capable.
This poor guy actually believes the PR, that the frontier model CEOs want to slow things down (and not just setup regulatory-enforced moats around their empires).